[Fri Jan 25 11:42:58 2019][667739.579843] LNet: HW NUMA nodes: 4, HW CPU cores: 48, npartitions: 4 [Fri Jan 25 11:42:58 2019][667739.587457] alg: No test for adler32 (adler32-zlib) [Fri Jan 25 11:42:59 2019][667740.389262] Lustre: Lustre: Build Version: 2.12.0 [Fri Jan 25 11:42:59 2019][667740.508252] LNet: Using FastReg for registration [Fri Jan 25 11:42:59 2019][667740.524815] LNet: Added LNI 10.0.10.52@o2ib7 [8/256/0/180] [Fri Jan 25 11:43:00 2019][667741.741106] LDISKFS-fs (dm-1): file extents enabled, maximum tree depth=5 [Fri Jan 25 11:43:00 2019][667741.761082] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [Fri Jan 25 11:43:00 2019][667742.007431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [Fri Jan 25 11:43:00 2019][667742.031444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [Fri Jan 25 11:43:01 2019][667742.486860] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Jan 25 11:43:01 2019][667742.524409] LustreError: 11-0: fir-OST0012-osc-MDT0003: operation ost_connect to node 10.0.10.103@o2ib7 failed: rc = -16 [Fri Jan 25 11:43:01 2019][667742.566756] Lustre: fir-MDT0003: Imperative Recovery not enabled, recovery window 300-900 [Fri Jan 25 11:43:01 2019][667742.576694] Lustre: fir-MDD0003: changelog on [Fri Jan 25 11:43:01 2019][667742.581848] Lustre: fir-MDT0003: in recovery but waiting for the first client to connect [Fri Jan 25 11:43:01 2019][667742.836917] Lustre: fir-MDT0003: Will be in recovery for at least 5:00, or until 10 clients reconnect [Fri Jan 25 11:43:01 2019][667742.837152] Lustre: fir-MDT0001: Connection restored to 0efd883e-065d-0727-44f7-75a6230140a1 (at 10.9.0.2@o2ib4) [Fri Jan 25 11:43:01 2019][667742.856488] Lustre: Skipped 1 previous similar message [Fri Jan 25 11:43:02 2019][667743.851160] Lustre: fir-MDT0003: Connection restored to c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) [Fri Jan 25 11:43:02 2019][667743.851161] Lustre: fir-MDT0001: Connection restored to c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) [Fri Jan 25 11:43:02 2019][667743.851164] Lustre: Skipped 1 previous similar message [Fri Jan 25 11:43:05 2019][667746.635504] Lustre: fir-MDT0001: Connection restored to 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) [Fri Jan 25 11:43:05 2019][667746.645764] Lustre: Skipped 3 previous similar messages [Fri Jan 25 11:43:12 2019][667753.624895] Lustre: fir-MDT0001: Connection restored to 9d0731b3-b468-e234-aad6-42e89a84a6d4 (at 10.9.101.60@o2ib4) [Fri Jan 25 11:43:12 2019][667753.635416] Lustre: Skipped 5 previous similar messages [Fri Jan 25 11:43:15 2019][667757.123890] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.103@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Jan 25 11:43:21 2019][667763.168425] Lustre: fir-MDT0003: Connection restored to 10.0.10.106@o2ib7 (at 10.0.10.106@o2ib7) [Fri Jan 25 11:43:21 2019][667763.177313] Lustre: Skipped 16 previous similar messages [Fri Jan 25 11:43:21 2019][667763.177586] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.0.10.106@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Jan 25 11:43:21 2019][667763.177588] LustreError: Skipped 28 previous similar messages [Fri Jan 25 11:43:25 2019][667766.805615] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.0.10.51@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Jan 25 11:43:25 2019][667766.822993] LustreError: Skipped 31 previous similar messages [Fri Jan 25 11:43:26 2019][667767.729424] LustreError: 11-0: fir-OST000e-osc-MDT0003: operation ost_connect to node 10.0.10.103@o2ib7 failed: rc = -16 [Fri Jan 25 11:43:26 2019][667767.740383] LustreError: Skipped 84 previous similar messages [Fri Jan 25 11:43:40 2019][667782.199320] Lustre: fir-MDT0003: Connection restored to 10.0.10.108@o2ib7 (at 10.0.10.108@o2ib7) [Fri Jan 25 11:43:40 2019][667782.208194] Lustre: Skipped 16 previous similar messages [Fri Jan 25 11:43:51 2019][667792.817730] LustreError: 11-0: fir-OST0012-osc-MDT0003: operation ost_connect to node 10.0.10.103@o2ib7 failed: rc = -16 [Fri Jan 25 11:43:51 2019][667792.828686] LustreError: Skipped 39 previous similar messages [Fri Jan 25 11:43:52 2019][667793.577505] Lustre: fir-MDT0003: Recovery over after 0:51, of 10 clients 10 recovered and 0 were evicted. [Fri Jan 25 15:37:51 2019][681832.893947] Lustre: fir-MDT0001: Connection restored to 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6) [Fri Jan 25 15:37:51 2019][681832.904299] Lustre: Skipped 57 previous similar messages [Fri Jan 25 16:06:40 2019][683562.121657] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 16:21:05 2019][684426.504092] Lustre: 127214:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:03:13 2019][686955.128962] Lustre: 128084:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:04:13 2019][687015.052625] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:10:09 2019][687371.376728] perf: interrupt took too long (2503 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 [Fri Jan 25 17:11:28 2019][687449.672108] Lustre: 126625:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:11:28 2019][687449.683938] Lustre: 126625:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Fri Jan 25 17:11:44 2019][687465.555182] Lustre: 127214:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:11:44 2019][687465.567040] Lustre: 127214:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Fri Jan 25 17:12:07 2019][687488.914879] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:12:07 2019][687488.926705] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Fri Jan 25 17:14:24 2019][687626.354169] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:14:58 2019][687659.957219] Lustre: 127247:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:17:35 2019][687817.192283] Lustre: 126623:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:24:36 2019][688238.001376] Lustre: 127247:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:24:36 2019][688238.013199] Lustre: 127247:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Fri Jan 25 17:28:56 2019][688498.381639] Lustre: 127247:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:34:26 2019][688827.697193] Lustre: 129280:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:44:22 2019][689424.415557] Lustre: 127134:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:44:22 2019][689424.427390] Lustre: 127134:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages [Fri Jan 25 17:57:12 2019][690194.304558] Lustre: 126625:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Fri Jan 25 19:25:24 2019][695485.869840] Lustre: 129794:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Fri Jan 25 19:25:24 2019][695485.881688] Lustre: 129794:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Fri Jan 25 19:35:00 2019][696061.807006] Lustre: 129262:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Fri Jan 25 19:35:00 2019][696061.818829] Lustre: 129262:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Fri Jan 25 19:41:23 2019][696444.737623] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Fri Jan 25 19:41:23 2019][696444.749452] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Fri Jan 25 19:51:28 2019][697050.572164] Lustre: 127247:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Fri Jan 25 19:51:28 2019][697050.583988] Lustre: 127247:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Fri Jan 25 20:11:07 2019][698229.060417] Lustre: 127209:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Fri Jan 25 20:11:07 2019][698229.072241] Lustre: 127209:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Fri Jan 25 20:30:14 2019][699376.151388] Lustre: 129262:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Fri Jan 25 20:30:14 2019][699376.163212] Lustre: 129262:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Fri Jan 25 20:40:15 2019][699976.898373] Lustre: 129262:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Fri Jan 25 20:40:15 2019][699976.910199] Lustre: 129262:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Fri Jan 25 21:12:36 2019][701917.980859] perf: interrupt took too long (3132 > 3128), lowering kernel.perf_event_max_sample_rate to 63000 [Fri Jan 25 21:14:50 2019][702052.037340] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 21:14:50 2019][702052.049163] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Fri Jan 25 21:23:14 2019][702555.867095] Lustre: 128085:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Fri Jan 25 21:26:48 2019][702770.392655] Lustre: fir-MDT0003: Client 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) reconnecting [Fri Jan 25 21:26:48 2019][702770.402779] Lustre: fir-MDT0003: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Fri Jan 25 21:26:48 2019][702770.413121] Lustre: Skipped 6 previous similar messages [Fri Jan 25 21:26:49 2019][702770.790386] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Fri Jan 25 21:26:49 2019][702770.800731] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.201@o2ib7 (6): c: 6, oc: 0, rc: 8 [Fri Jan 25 21:26:49 2019][702770.813420] LustreError: 126419:0:(events.c:305:request_in_callback()) event type 2, status -103, service mdt_io [Fri Jan 25 21:26:49 2019][702771.187394] Lustre: 126479:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1548480574/real 0] req@ffffa08a7bad6900 x1623663728744736/t0(0) o13->fir-OST0013-osc-MDT0001@10.0.10.104@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1548480581 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Jan 25 21:26:49 2019][702771.214905] Lustre: fir-OST0013-osc-MDT0001: Connection to fir-OST0013 (at 10.0.10.104@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Fri Jan 25 21:26:50 2019][702771.885026] Lustre: fir-MDT0003: Client 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6) reconnecting [Fri Jan 25 21:26:50 2019][702771.895111] Lustre: Skipped 1 previous similar message [Fri Jan 25 21:26:50 2019][702771.900372] Lustre: fir-MDT0003: Connection restored to 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6) [Fri Jan 25 21:26:50 2019][702771.910714] Lustre: Skipped 1 previous similar message [Fri Jan 25 21:26:50 2019][702771.947407] Lustre: 126477:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1548480574/real 0] req@ffffa08a6f6f4e00 x1623663728745120/t0(0) o13->fir-OST002e-osc-MDT0001@10.0.10.107@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1548480581 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Jan 25 21:26:50 2019][702771.974922] Lustre: fir-OST002e-osc-MDT0001: Connection to fir-OST002e (at 10.0.10.107@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Fri Jan 25 21:26:51 2019][702772.934681] Lustre: fir-MDT0001: Client 9daa8e10-c154-ca96-d85d-3f6355b9ec3f (at 10.8.2.5@o2ib6) reconnecting [Fri Jan 25 21:26:51 2019][702772.944693] Lustre: fir-MDT0001: Connection restored to 9daa8e10-c154-ca96-d85d-3f6355b9ec3f (at 10.8.2.5@o2ib6) [Fri Jan 25 21:26:51 2019][702772.990411] Lustre: 126488:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1548480574/real 0] req@ffffa08a7c629200 x1623663728744448/t0(0) o13->fir-OST0011-osc-MDT0003@10.0.10.104@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1548480581 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Jan 25 21:26:51 2019][702772.990415] Lustre: fir-OST0011-osc-MDT0001: Connection to fir-OST0011 (at 10.0.10.104@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Fri Jan 25 21:26:51 2019][702772.990417] Lustre: Skipped 3 previous similar messages [Fri Jan 25 21:26:51 2019][702773.039386] Lustre: 126488:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [Fri Jan 25 21:26:54 2019][702775.790427] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Fri Jan 25 21:26:54 2019][702775.800766] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 3 previous similar messages [Fri Jan 25 21:26:54 2019][702775.811109] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.209@o2ib7 (21): c: 7, oc: 0, rc: 8 [Fri Jan 25 21:26:54 2019][702775.823352] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 3 previous similar messages [Fri Jan 25 21:26:54 2019][702776.067435] Lustre: 126485:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1548480578/real 0] req@ffffa08a6f6f3600 x1623663728746048/t0(0) o13->fir-OST002c-osc-MDT0001@10.0.10.107@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1548480585 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Jan 25 21:26:54 2019][702776.067437] Lustre: 126486:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1548480578/real 0] req@ffffa08a7c62c500 x1623663728746304/t0(0) o13->fir-OST002a-osc-MDT0003@10.0.10.107@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1548480585 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Jan 25 21:26:54 2019][702776.067443] Lustre: fir-OST002a-osc-MDT0003: Connection to fir-OST002a (at 10.0.10.107@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Fri Jan 25 21:26:54 2019][702776.067444] Lustre: Skipped 6 previous similar messages [Fri Jan 25 21:26:54 2019][702776.143903] Lustre: 126485:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 12 previous similar messages [Fri Jan 25 21:26:55 2019][702777.006973] Lustre: fir-MDT0001: Client f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6) reconnecting [Fri Jan 25 21:26:55 2019][702777.017079] Lustre: fir-MDT0001: Connection restored to f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6) [Fri Jan 25 21:26:56 2019][702777.790447] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Fri Jan 25 21:26:56 2019][702777.800794] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.101@o2ib7 (8): c: 0, oc: 0, rc: 8 [Fri Jan 25 21:26:56 2019][702777.918255] sched: RT throttling activated [Fri Jan 25 21:26:56 2019][702777.918266] LustreError: 127710:0:(pack_generic.c:590:__lustre_unpack_msg()) message length 0 too small for magic/version check [Fri Jan 25 21:26:56 2019][702777.918272] Lustre: mdt_io: This server is not able to keep up with request traffic (cpu-bound). [Fri Jan 25 21:26:56 2019][702777.918276] Lustre: 129301:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=6 reqQ=0 recA=14, svcEst=31, delay=13022 [Fri Jan 25 21:26:56 2019][702777.918290] Lustre: 129301:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-7s), not sending early reply. Consider increasing at_early_margin (5)? req@ffffa06b7a665450 x1623566371635104/t0(0) o4->9b3a0481-8c2f-a606-18ad-7a06b856368e@10.8.3.11@o2ib6:495/0 lens 624/448 e 0 to 0 dl 1548480580 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Jan 25 21:26:56 2019][702777.918300] Lustre: 127696:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:7s); client may timeout. req@ffffa06382221850 x1623566371635040/t8664402556(0) o4->9b3a0481-8c2f-a606-18ad-7a06b856368e@10.8.3.11@o2ib6:495/0 lens 800/416 e 0 to 0 dl 1548480580 ref 2 fl Complete:/0/0 rc 0/0 [Fri Jan 25 21:26:56 2019][702777.918304] LustreError: 127702:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffffa05b447e9450 x1623566371635328/t0(0) o4->9b3a0481-8c2f-a606-18ad-7a06b856368e@10.8.3.11@o2ib6:520/0 lens 488/448 e 1 to 0 dl 1548480605 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jan 25 21:26:56 2019][702777.918323] Lustre: fir-MDT0003: Bulk IO write error with 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6), client will retry: rc = -107 [Fri Jan 25 21:26:56 2019][702777.918337] Lustre: 127701:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 9s req@ffffa05b447e4c50 x1623621103335152/t0(0) o4->85c45b71-10aa-be9f-071b-9a9c50548db2@10.8.15.7@o2ib6:0/0 lens 1856/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 [Fri Jan 25 21:26:56 2019][702777.989462] LustreError: 127535:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffffa07b416a2450 x1623669142371696/t0(0) o4->f9afcafe-b7fa-076a-0d87-fc0f78fd02d0@10.8.6.20@o2ib6:521/0 lens 488/448 e 1 to 0 dl 1548480606 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jan 25 21:26:56 2019][702778.100239] LustreError: 127710:0:(sec.c:2068:sptlrpc_svc_unwrap_request()) error unpacking request from 12345-10.8.3.11@o2ib6 x1623566371635056 [Fri Jan 25 21:27:02 2019][702783.790494] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Fri Jan 25 21:27:02 2019][702783.800840] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.204@o2ib7 (6): c: 7, oc: 0, rc: 8 [Fri Jan 25 21:27:02 2019][702783.860514] LustreError: 126673:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffffa07b416a4c50 x1623566371635136/t0(0) o4->9b3a0481-8c2f-a606-18ad-7a06b856368e@10.8.3.11@o2ib6:520/0 lens 488/448 e 1 to 0 dl 1548480605 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jan 25 21:27:02 2019][702783.884664] Lustre: fir-MDT0001: Bulk IO write error with 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6), client will retry: rc = -110 [Fri Jan 25 21:27:02 2019][702783.897782] Lustre: Skipped 3 previous similar messages [Fri Jan 25 21:27:04 2019][702786.179518] Lustre: 126483:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1548480587/real 1548480587] req@ffffa08a7bad4800 x1623663728749232/t0(0) o13->fir-OST0021-osc-MDT0003@10.0.10.106@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1548480594 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Jan 25 21:27:04 2019][702786.207806] Lustre: 126483:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 16 previous similar messages [Fri Jan 25 21:27:04 2019][702786.217810] Lustre: fir-OST0021-osc-MDT0003: Connection to fir-OST0021 (at 10.0.10.106@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Fri Jan 25 21:27:04 2019][702786.233962] Lustre: Skipped 14 previous similar messages [Fri Jan 25 21:27:04 2019][702786.248675] Lustre: fir-OST0021-osc-MDT0003: Connection restored to 10.0.10.106@o2ib7 (at 10.0.10.106@o2ib7) [Fri Jan 25 21:27:04 2019][702786.258587] Lustre: Skipped 35 previous similar messages [Fri Jan 25 21:27:07 2019][702788.790535] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Fri Jan 25 21:27:07 2019][702788.800875] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 2 previous similar messages [Fri Jan 25 21:27:07 2019][702788.811216] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.102@o2ib7 (9): c: 0, oc: 0, rc: 8 [Fri Jan 25 21:27:07 2019][702788.823374] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 2 previous similar messages [Fri Jan 25 21:27:13 2019][702794.931589] Lustre: fir-OST0000-osc-MDT0003: Connection to fir-OST0000 (at 10.0.10.101@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Fri Jan 25 21:27:13 2019][702794.947740] Lustre: Skipped 51 previous similar messages [Fri Jan 25 21:27:17 2019][702798.790613] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds [Fri Jan 25 21:27:17 2019][702798.800954] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 4 previous similar messages [Fri Jan 25 21:27:17 2019][702798.811294] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.106@o2ib7 (7): c: 6, oc: 0, rc: 8 [Fri Jan 25 21:27:17 2019][702798.823451] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 4 previous similar messages [Fri Jan 25 21:27:17 2019][702798.947624] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [Fri Jan 25 21:27:18 2019][702800.331666] LNetError: 126624:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.3.11@o2ib6 from 10.0.10.52@o2ib7 [Fri Jan 25 21:27:19 2019][702801.099678] LNetError: 129246:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.3.11@o2ib6 from 10.0.10.52@o2ib7 [Fri Jan 25 21:27:19 2019][702801.111494] LNetError: 129246:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 1 previous similar message [Fri Jan 25 21:27:20 2019][702802.713636] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [mdt_io01_009:127261] [Fri Jan 25 21:27:20 2019][702802.721634] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [mdt_io02_022:129309] [Fri Jan 25 21:27:20 2019][702802.721679] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:21 2019][702802.721698] CPU: 2 PID: 129309 Comm: mdt_io02_022 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:21 2019][702802.721699] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:21 2019][702802.721700] task: ffffa08b6dbb0000 ti: ffffa08b55a1c000 task.ti: ffffa08b55a1c000 [Fri Jan 25 21:27:21 2019][702802.721707] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Fri Jan 25 21:27:21 2019][702802.721708] RSP: 0018:ffffa08b55a1f798 EFLAGS: 00000246 [Fri Jan 25 21:27:21 2019][702802.721709] RAX: 0000000000000000 RBX: ffffa052e74194c8 RCX: 0000000000110000 [Fri Jan 25 21:27:21 2019][702802.721709] RDX: ffffa07b7f81b780 RSI: 0000000001110101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:21 2019][702802.721710] RBP: ffffa08b55a1f798 R08: ffffa07b7f61b780 R09: 0000000000000000 [Fri Jan 25 21:27:21 2019][702802.721711] R10: ffffa07b7f61f140 R11: fffff7f77df2c400 R12: ffffa08b55a1f738 [Fri Jan 25 21:27:21 2019][702802.721712] R13: ffffa052e74195b0 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:21 2019][702802.721713] FS: 00007efe99c66700(0000) GS:ffffa07b7f600000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:21 2019][702802.721714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:21 2019][702802.721715] CR2: 00007efe9c7c6000 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:21 2019][702802.721716] Call Trace: [Fri Jan 25 21:27:21 2019][702802.721721] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:21 2019][702802.721724] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:21 2019][702802.721742] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:21 2019][702802.721755] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:21 2019][702802.721765] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Fri Jan 25 21:27:21 2019][702802.721770] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Fri Jan 25 21:27:21 2019][702802.721773] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:21 2019][702802.721784] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Fri Jan 25 21:27:21 2019][702802.721793] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:21 2019][702802.721802] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Fri Jan 25 21:27:21 2019][702802.721815] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:21 2019][702802.721825] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Fri Jan 25 21:27:21 2019][702802.721835] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Fri Jan 25 21:27:21 2019][702802.721857] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Fri Jan 25 21:27:21 2019][702802.721923] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.721966] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.722004] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.722040] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.722043] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:21 2019][702802.722044] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:21 2019][702802.722076] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.722118] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.722157] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.722168] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:21 2019][702802.722205] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.722242] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.722244] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:21 2019][702802.722246] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:21 2019][702802.722283] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.722319] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.722322] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:21 2019][702802.722324] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:21 2019][702802.722326] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:21 2019][702802.722327] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:21 2019][702802.722346] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Fri Jan 25 21:27:21 2019][702802.739636] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [mdt_io00_007:127251] [Fri Jan 25 21:27:21 2019][702802.739684] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:21 2019][702802.739705] CPU: 4 PID: 127251 Comm: mdt_io00_007 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:21 2019][702802.739706] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:21 2019][702802.739707] task: ffffa08b7061b0c0 ti: ffffa08b5f38c000 task.ti: ffffa08b5f38c000 [Fri Jan 25 21:27:21 2019][702802.739715] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:21 2019][702802.739717] RSP: 0018:ffffa08b5f38f808 EFLAGS: 00000246 [Fri Jan 25 21:27:21 2019][702802.739718] RAX: 0000000000000000 RBX: ffffa052e74d9d18 RCX: 0000000000210000 [Fri Jan 25 21:27:21 2019][702802.739719] RDX: ffffa07b7f85b780 RSI: 0000000001310101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:21 2019][702802.739719] RBP: ffffa08b5f38f808 R08: ffffa05b7ee5b780 R09: 0000000000000000 [Fri Jan 25 21:27:21 2019][702802.739720] R10: ffffa05b7ee5f140 R11: fffff7f6ba549e00 R12: ffffa08b5f38f7a8 [Fri Jan 25 21:27:21 2019][702802.739721] R13: ffffa052e74d9e00 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:21 2019][702802.739722] FS: 00007fc83477e880(0000) GS:ffffa05b7ee40000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:21 2019][702802.739723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:21 2019][702802.739724] CR2: 00007f06ea509000 CR3: 00000010306d8000 CR4: 00000000003407e0 [Fri Jan 25 21:27:21 2019][702802.739725] Call Trace: [Fri Jan 25 21:27:21 2019][702802.739731] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:21 2019][702802.739735] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:21 2019][702802.739752] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:21 2019][702802.739765] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:21 2019][702802.739768] [] ? ktime_get+0x52/0xe0 [Fri Jan 25 21:27:21 2019][702802.739778] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Jan 25 21:27:21 2019][702802.739788] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:21 2019][702802.739789] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:21 2019][702802.739803] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:21 2019][702802.739814] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:21 2019][702802.739837] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:21 2019][702802.739910] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.739951] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.739989] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.739992] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:21 2019][702802.739995] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:21 2019][702802.739997] [] ? __enqueue_entity+0x78/0x80 [Fri Jan 25 21:27:21 2019][702802.740041] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.740085] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.740127] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.740137] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:21 2019][702802.740178] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.740217] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.740220] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:21 2019][702802.740223] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:21 2019][702802.740263] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.740302] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:21 2019][702802.740305] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:21 2019][702802.740306] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:21 2019][702802.740309] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:21 2019][702802.740310] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:21 2019][702802.740330] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:21 2019][702802.748636] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 24s! [mdt_io01_008:127235] [Fri Jan 25 21:27:21 2019][702802.748682] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:21 2019][702802.748700] CPU: 5 PID: 127235 Comm: mdt_io01_008 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:22 2019][702802.748701] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:22 2019][702802.748702] task: ffffa08b78069040 ti: ffffa08b5eb94000 task.ti: ffffa08b5eb94000 [Fri Jan 25 21:27:22 2019][702802.748710] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Fri Jan 25 21:27:22 2019][702802.748711] RSP: 0018:ffffa08b5eb97798 EFLAGS: 00000246 [Fri Jan 25 21:27:22 2019][702802.748712] RAX: 0000000000000000 RBX: ffffa073694667e8 RCX: 0000000000290000 [Fri Jan 25 21:27:22 2019][702802.748713] RDX: ffffa05b7ee9b780 RSI: 0000000000410101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:22 2019][702802.748714] RBP: ffffa08b5eb97798 R08: ffffa06b7f65b780 R09: 0000000000000000 [Fri Jan 25 21:27:22 2019][702802.748715] R10: ffffa06b7f65f140 R11: fffff7f6ff742c00 R12: ffffa08b5eb97738 [Fri Jan 25 21:27:22 2019][702802.748715] R13: ffffa073694668d0 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:22 2019][702802.748717] FS: 00007ff1c7e68840(0000) GS:ffffa06b7f640000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:22 2019][702802.748717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:22 2019][702802.748718] CR2: 00005651892bd300 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:22 2019][702802.748719] Call Trace: [Fri Jan 25 21:27:22 2019][702802.748725] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:22 2019][702802.748729] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:22 2019][702802.748745] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:22 2019][702802.748758] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:22 2019][702802.748767] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Fri Jan 25 21:27:22 2019][702802.748773] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Fri Jan 25 21:27:22 2019][702802.748777] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:22 2019][702802.748790] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Fri Jan 25 21:27:22 2019][702802.748799] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:22 2019][702802.748807] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Fri Jan 25 21:27:22 2019][702802.748820] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:22 2019][702802.748831] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Fri Jan 25 21:27:22 2019][702802.748841] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Fri Jan 25 21:27:22 2019][702802.748864] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Fri Jan 25 21:27:22 2019][702802.748930] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.748972] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.748976] [] ? __slab_free+0x81/0x2f0 [Fri Jan 25 21:27:22 2019][702802.748977] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:22 2019][702802.748979] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:22 2019][702802.749011] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.749053] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.749092] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.749102] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:22 2019][702802.749140] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.749177] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.749180] [] ? wake_up_state+0x20/0x20 [Fri Jan 25 21:27:22 2019][702802.749217] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.749254] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.749256] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:22 2019][702802.749258] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:22 2019][702802.749261] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:22 2019][702802.749263] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:22 2019][702802.749281] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Fri Jan 25 21:27:22 2019][702802.773635] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [mdt_io00_011:127365] [Fri Jan 25 21:27:22 2019][702802.773676] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:22 2019][702802.773694] CPU: 8 PID: 127365 Comm: mdt_io00_011 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:22 2019][702802.773695] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:22 2019][702802.773696] task: ffffa08b61125140 ti: ffffa08b6eb8c000 task.ti: ffffa08b6eb8c000 [Fri Jan 25 21:27:22 2019][702802.773704] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:22 2019][702802.773706] RSP: 0018:ffffa08b6eb8f798 EFLAGS: 00000246 [Fri Jan 25 21:27:22 2019][702802.773707] RAX: 0000000000000000 RBX: ffffa05bed7d2990 RCX: 0000000000410000 [Fri Jan 25 21:27:22 2019][702802.773708] RDX: ffffa06b7f61b780 RSI: 0000000000090101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:22 2019][702802.773708] RBP: ffffa08b6eb8f798 R08: ffffa05b7ee9b780 R09: 0000000000000000 [Fri Jan 25 21:27:22 2019][702802.773709] R10: ffffa05b7ee9f140 R11: fffff7f77e9caa00 R12: ffffa08b6eb8f738 [Fri Jan 25 21:27:22 2019][702802.773709] R13: ffffa05bed7d2a78 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:22 2019][702802.773711] FS: 00007fcf13a0f700(0000) GS:ffffa05b7ee80000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:22 2019][702802.773712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:22 2019][702802.773712] CR2: 00007fcf185aa000 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:22 2019][702802.773713] Call Trace: [Fri Jan 25 21:27:22 2019][702802.773719] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:22 2019][702802.773723] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:22 2019][702802.773738] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:22 2019][702802.773749] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:22 2019][702802.773757] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Fri Jan 25 21:27:22 2019][702802.773761] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Fri Jan 25 21:27:22 2019][702802.773765] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:22 2019][702802.773773] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Fri Jan 25 21:27:22 2019][702802.773781] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:22 2019][702802.773788] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Fri Jan 25 21:27:22 2019][702802.773799] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:22 2019][702802.773808] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Fri Jan 25 21:27:22 2019][702802.773816] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Fri Jan 25 21:27:22 2019][702802.773833] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Fri Jan 25 21:27:22 2019][702802.773891] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.773926] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.773958] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.773960] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:22 2019][702802.773962] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:22 2019][702802.773988] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.774023] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.774056] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.774065] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:22 2019][702802.774097] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.774128] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.774130] [] ? wake_up_state+0x20/0x20 [Fri Jan 25 21:27:22 2019][702802.774161] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.774191] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:22 2019][702802.774195] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:22 2019][702802.774197] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:22 2019][702802.774199] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:22 2019][702802.774201] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:22 2019][702802.774219] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:22 2019][702802.782634] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [mdt_io01_013:127657] [Fri Jan 25 21:27:22 2019][702802.782682] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:22 2019][702802.782700] CPU: 9 PID: 127657 Comm: mdt_io01_013 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:22 2019][702802.782701] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:22 2019][702802.782702] task: ffffa08b76cca080 ti: ffffa08b72e5c000 task.ti: ffffa08b72e5c000 [Fri Jan 25 21:27:22 2019][702802.782710] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:22 2019][702802.782712] RSP: 0018:ffffa08b72e5f808 EFLAGS: 00000246 [Fri Jan 25 21:27:22 2019][702802.782713] RAX: 0000000000000000 RBX: ffffa05bebce1d18 RCX: 0000000000490000 [Fri Jan 25 21:27:22 2019][702802.782713] RDX: ffffa05b7ee5b780 RSI: 0000000000210101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:22 2019][702802.782714] RBP: ffffa08b72e5f808 R08: ffffa06b7f69b780 R09: 0000000000000000 [Fri Jan 25 21:27:22 2019][702802.782715] R10: ffffa06b7f69f140 R11: fffff7f6ef4a3200 R12: ffffa08b72e5f7a8 [Fri Jan 25 21:27:22 2019][702802.782715] R13: ffffa05bebce1e00 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:22 2019][702802.782717] FS: 00007f6b107bf700(0000) GS:ffffa06b7f680000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:22 2019][702802.782718] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:22 2019][702802.782719] CR2: 00007f7385faa000 CR3: 000000407fe34000 CR4: 00000000003407e0 [Fri Jan 25 21:27:22 2019][702802.782720] Call Trace: [Fri Jan 25 21:27:22 2019][702802.782725] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:22 2019][702802.782729] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:22 2019][702802.782745] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:22 2019][702802.782758] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:22 2019][702802.782761] [] ? ktime_get+0x52/0xe0 [Fri Jan 25 21:27:22 2019][702802.782770] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Jan 25 21:27:23 2019][702802.782779] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:23 2019][702802.782781] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:23 2019][702802.782794] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:23 2019][702802.782804] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:23 2019][702802.782827] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:23 2019][702802.782894] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.782932] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.782936] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:23 2019][702802.782938] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:23 2019][702802.782980] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.783022] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.783061] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.783071] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:23 2019][702802.783109] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.783146] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.783149] [] ? wake_up_state+0x20/0x20 [Fri Jan 25 21:27:23 2019][702802.783185] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.783222] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.783224] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:23 2019][702802.783226] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:23 2019][702802.783229] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:23 2019][702802.783230] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:23 2019][702802.783249] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:23 2019][702802.790636] NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [mdt_io02_025:130204] [Fri Jan 25 21:27:23 2019][702802.790663] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:23 2019][702802.790675] CPU: 10 PID: 130204 Comm: mdt_io02_025 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:23 2019][702802.790676] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:23 2019][702802.790677] task: ffffa07b2cf89040 ti: ffffa08acde68000 task.ti: ffffa08acde68000 [Fri Jan 25 21:27:23 2019][702802.790680] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:23 2019][702802.790681] RSP: 0018:ffffa08acde6b798 EFLAGS: 00000246 [Fri Jan 25 21:27:23 2019][702802.790682] RAX: 0000000000000000 RBX: ffffa052e7400c78 RCX: 0000000000510000 [Fri Jan 25 21:27:23 2019][702802.790683] RDX: ffffa07b7f6db780 RSI: 0000000000710101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:23 2019][702802.790684] RBP: ffffa08acde6b798 R08: ffffa07b7f69b780 R09: 0000000000000000 [Fri Jan 25 21:27:23 2019][702802.790685] R10: ffffa07b7f69f140 R11: fffff7f77f589000 R12: ffffa08acde6b738 [Fri Jan 25 21:27:23 2019][702802.790685] R13: ffffa052e7400d60 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:23 2019][702802.790687] FS: 00007efe99c66700(0000) GS:ffffa07b7f680000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:23 2019][702802.790688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:23 2019][702802.790688] CR2: 00007efe9c7c7000 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:23 2019][702802.790689] Call Trace: [Fri Jan 25 21:27:23 2019][702802.790692] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:23 2019][702802.790694] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:23 2019][702802.790707] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:23 2019][702802.790719] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:23 2019][702802.790721] [] ? load_balance+0x178/0x9a0 [Fri Jan 25 21:27:23 2019][702802.790728] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Fri Jan 25 21:27:23 2019][702802.790737] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:23 2019][702802.790745] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Fri Jan 25 21:27:23 2019][702802.790755] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:23 2019][702802.790766] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Fri Jan 25 21:27:23 2019][702802.790776] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Fri Jan 25 21:27:23 2019][702802.790791] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Fri Jan 25 21:27:23 2019][702802.790834] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.790876] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.790890] [] ? LNetMEAttach+0x7d/0x2d0 [lnet] [Fri Jan 25 21:27:23 2019][702802.790892] [] ? __mutex_lock_slowpath+0x1b8/0x1d0 [Fri Jan 25 21:27:23 2019][702802.790924] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.790964] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.791004] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.791010] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:23 2019][702802.791048] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.791085] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.791087] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:23 2019][702802.791088] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:23 2019][702802.791125] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.791161] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.791163] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:23 2019][702802.791165] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:23 2019][702802.791166] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:23 2019][702802.791168] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:23 2019][702802.791187] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:23 2019][702802.816635] NMI watchdog: BUG: soft lockup - CPU#13 stuck for 24s! [mdt_io01_028:129301] [Fri Jan 25 21:27:23 2019][702802.816663] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:23 2019][702802.816675] CPU: 13 PID: 129301 Comm: mdt_io01_028 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:23 2019][702802.816676] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:23 2019][702802.816677] task: ffffa08b76725140 ti: ffffa08b60e98000 task.ti: ffffa08b60e98000 [Fri Jan 25 21:27:23 2019][702802.816681] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:23 2019][702802.816681] RSP: 0018:ffffa08b60e9b798 EFLAGS: 00000246 [Fri Jan 25 21:27:23 2019][702802.816682] RAX: 0000000000000000 RBX: ffffa05bebcc0428 RCX: 0000000000690000 [Fri Jan 25 21:27:23 2019][702802.816683] RDX: ffffa07b7f79b780 RSI: 0000000000d10101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:23 2019][702802.816684] RBP: ffffa08b60e9b798 R08: ffffa06b7f6db780 R09: 0000000000000000 [Fri Jan 25 21:27:23 2019][702802.816685] R10: ffffa06b7f6df140 R11: fffff7f6ff119a00 R12: ffffa08b60e9b738 [Fri Jan 25 21:27:23 2019][702802.816685] R13: ffffa05bebcc0510 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:23 2019][702802.816687] FS: 00007fbd6eec6740(0000) GS:ffffa06b7f6c0000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:23 2019][702802.816688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:23 2019][702802.816688] CR2: 00007fbd6dc291b0 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:23 2019][702802.816689] Call Trace: [Fri Jan 25 21:27:23 2019][702802.816692] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:23 2019][702802.816694] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:23 2019][702802.816706] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:23 2019][702802.816718] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:23 2019][702802.816724] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Fri Jan 25 21:27:23 2019][702802.816729] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Fri Jan 25 21:27:23 2019][702802.816731] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:23 2019][702802.816739] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Fri Jan 25 21:27:23 2019][702802.816747] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:23 2019][702802.816755] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Fri Jan 25 21:27:23 2019][702802.816766] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:23 2019][702802.816776] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Fri Jan 25 21:27:23 2019][702802.816786] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Fri Jan 25 21:27:23 2019][702802.816801] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Fri Jan 25 21:27:23 2019][702802.816844] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.816886] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.816888] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:23 2019][702802.816890] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:23 2019][702802.816922] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.816962] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.817001] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:23 2019][702802.817008] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:23 2019][702802.817045] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.817082] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.817084] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:24 2019][702802.817087] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:24 2019][702802.817123] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.817160] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.817161] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:24 2019][702802.817163] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:24 2019][702802.817165] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:24 2019][702802.817166] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:24 2019][702802.817185] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:24 2019][702802.824635] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 24s! [mdt_io02_016:127693] [Fri Jan 25 21:27:24 2019][702802.824677] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:24 2019][702802.824695] CPU: 14 PID: 127693 Comm: mdt_io02_016 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:24 2019][702802.824695] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:24 2019][702802.824696] task: ffffa06b78802080 ti: ffffa08b7a66c000 task.ti: ffffa08b7a66c000 [Fri Jan 25 21:27:24 2019][702802.824702] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:24 2019][702802.824703] RSP: 0018:ffffa08b7a66f808 EFLAGS: 00000246 [Fri Jan 25 21:27:24 2019][702802.824704] RAX: 0000000000000000 RBX: ffffa052e9398428 RCX: 0000000000710000 [Fri Jan 25 21:27:24 2019][702802.824705] RDX: ffffa07b7f89b780 RSI: 0000000001510101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:24 2019][702802.824706] RBP: ffffa08b7a66f808 R08: ffffa07b7f6db780 R09: 0000000000000000 [Fri Jan 25 21:27:24 2019][702802.824706] R10: ffffa07b7f6df140 R11: fffff7f73faff600 R12: ffffa08b7a66f7a8 [Fri Jan 25 21:27:24 2019][702802.824707] R13: ffffa052e9398510 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:24 2019][702802.824708] FS: 00007fcf13a0f700(0000) GS:ffffa07b7f6c0000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:24 2019][702802.824709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:24 2019][702802.824710] CR2: 00007fcf040ca0d8 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:24 2019][702802.824711] Call Trace: [Fri Jan 25 21:27:24 2019][702802.824716] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:24 2019][702802.824719] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:24 2019][702802.824735] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:24 2019][702802.824748] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:24 2019][702802.824757] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:24 2019][702802.824760] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:24 2019][702802.824773] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:24 2019][702802.824784] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:24 2019][702802.824807] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:24 2019][702802.824873] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.824876] [] ? __schedule+0x3ff/0x890 [Fri Jan 25 21:27:24 2019][702802.824879] [] ? __slab_free+0x81/0x2f0 [Fri Jan 25 21:27:24 2019][702802.824894] [] ? lnet_ptl_attach_md+0x37e/0x530 [lnet] [Fri Jan 25 21:27:24 2019][702802.824936] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.824978] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.825017] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.825027] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:24 2019][702802.825065] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.825102] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.825139] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.825176] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.825178] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:24 2019][702802.825180] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:24 2019][702802.825182] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:24 2019][702802.825184] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:24 2019][702802.825202] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:24 2019][702802.850639] NMI watchdog: BUG: soft lockup - CPU#17 stuck for 23s! [mdt_io01_029:129974] [Fri Jan 25 21:27:24 2019][702802.850678] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:24 2019][702802.850694] CPU: 17 PID: 129974 Comm: mdt_io01_029 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:24 2019][702802.850694] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:24 2019][702802.850695] task: ffffa06b76152080 ti: ffffa08b0efec000 task.ti: ffffa08b0efec000 [Fri Jan 25 21:27:24 2019][702802.850700] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:24 2019][702802.850701] RSP: 0018:ffffa08b0efef808 EFLAGS: 00000246 [Fri Jan 25 21:27:24 2019][702802.850702] RAX: 0000000000000000 RBX: ffffa05bebdd3a30 RCX: 0000000000890000 [Fri Jan 25 21:27:24 2019][702802.850703] RDX: ffffa05b7f0db780 RSI: 0000000001610101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:24 2019][702802.850704] RBP: ffffa08b0efef808 R08: ffffa06b7f71b780 R09: 0000000000000000 [Fri Jan 25 21:27:24 2019][702802.850704] R10: ffffa06b7f71f140 R11: fffff7f70046cc00 R12: ffffa08b0efef7a8 [Fri Jan 25 21:27:24 2019][702802.850705] R13: ffffa05bebdd3b18 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:24 2019][702802.850706] FS: 00007ff1c7e68840(0000) GS:ffffa06b7f700000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:24 2019][702802.850707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:24 2019][702802.850708] CR2: 00007ff1c73ef1a0 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:24 2019][702802.850709] Call Trace: [Fri Jan 25 21:27:24 2019][702802.850712] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:24 2019][702802.850716] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:24 2019][702802.850730] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:24 2019][702802.850742] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:24 2019][702802.850746] [] ? ___slab_alloc+0x209/0x4f0 [Fri Jan 25 21:27:24 2019][702802.850754] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Jan 25 21:27:24 2019][702802.850762] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:24 2019][702802.850765] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:24 2019][702802.850777] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:24 2019][702802.850787] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:24 2019][702802.850806] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:24 2019][702802.850855] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.850857] [] ? __slab_free+0x81/0x2f0 [Fri Jan 25 21:27:24 2019][702802.850894] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.850897] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:24 2019][702802.850899] [] ? __enqueue_entity+0x78/0x80 [Fri Jan 25 21:27:24 2019][702802.850940] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.850981] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.851019] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.851029] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:24 2019][702802.851066] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.851103] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.851104] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:24 2019][702802.851107] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:24 2019][702802.851143] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.851179] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:24 2019][702802.851181] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:24 2019][702802.851183] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:24 2019][702802.851185] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:24 2019][702802.851186] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:24 2019][702802.851204] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:24 2019][702802.858635] NMI watchdog: BUG: soft lockup - CPU#18 stuck for 23s! [mdt_io02_020:129292] [Fri Jan 25 21:27:24 2019][702802.858662] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:24 2019][702802.858674] CPU: 18 PID: 129292 Comm: mdt_io02_020 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:25 2019][702802.858674] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:25 2019][702802.858675] task: ffffa08b76722080 ti: ffffa08b57f4c000 task.ti: ffffa08b57f4c000 [Fri Jan 25 21:27:25 2019][702802.858679] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:25 2019][702802.858680] RSP: 0018:ffffa08b57f4f808 EFLAGS: 00000246 [Fri Jan 25 21:27:25 2019][702802.858681] RAX: 0000000000000000 RBX: ffffa05bed7f2140 RCX: 0000000000910000 [Fri Jan 25 21:27:25 2019][702802.858682] RDX: ffffa07b7f8db780 RSI: 0000000001710101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:25 2019][702802.858682] RBP: ffffa08b57f4f808 R08: ffffa07b7f71b780 R09: 0000000000000000 [Fri Jan 25 21:27:25 2019][702802.858683] R10: ffffa07b7f71f140 R11: fffff7f7402ba200 R12: ffffa08b57f4f7a8 [Fri Jan 25 21:27:25 2019][702802.858684] R13: ffffa05bed7f2228 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:25 2019][702802.858685] FS: 00007efe9a467700(0000) GS:ffffa07b7f700000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:25 2019][702802.858686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:25 2019][702802.858687] CR2: 00007efe9c7c6000 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:25 2019][702802.858688] Call Trace: [Fri Jan 25 21:27:25 2019][702802.858690] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:25 2019][702802.858693] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:25 2019][702802.858705] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:25 2019][702802.858717] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:25 2019][702802.858720] [] ? ktime_get+0x52/0xe0 [Fri Jan 25 21:27:25 2019][702802.858725] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Jan 25 21:27:25 2019][702802.858734] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:25 2019][702802.858736] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:25 2019][702802.858746] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:25 2019][702802.858757] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:25 2019][702802.858772] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:25 2019][702802.858815] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.858818] [] ? __slab_free+0x81/0x2f0 [Fri Jan 25 21:27:25 2019][702802.858855] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.858857] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:25 2019][702802.858859] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:25 2019][702802.858860] [] ? __enqueue_entity+0x78/0x80 [Fri Jan 25 21:27:25 2019][702802.858902] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.858943] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.858982] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.858988] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:25 2019][702802.859026] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.859062] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.859064] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:25 2019][702802.859066] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:25 2019][702802.859102] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.859138] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.859140] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:25 2019][702802.859141] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:25 2019][702802.859143] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:25 2019][702802.859145] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:25 2019][702802.859164] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:25 2019][702802.876637] NMI watchdog: BUG: soft lockup - CPU#20 stuck for 23s! [mdt_io00_004:127225] [Fri Jan 25 21:27:25 2019][702802.876665] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:25 2019][702802.876677] CPU: 20 PID: 127225 Comm: mdt_io00_004 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:25 2019][702802.876678] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:25 2019][702802.876679] task: ffffa07b74625140 ti: ffffa08b6bb20000 task.ti: ffffa08b6bb20000 [Fri Jan 25 21:27:25 2019][702802.876682] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:25 2019][702802.876683] RSP: 0018:ffffa08b6bb23808 EFLAGS: 00000246 [Fri Jan 25 21:27:25 2019][702802.876684] RAX: 0000000000000000 RBX: ffffa05bed7b3e58 RCX: 0000000000a10000 [Fri Jan 25 21:27:25 2019][702802.876685] RDX: ffffa07b7f61b780 RSI: 0000000000110101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:25 2019][702802.876686] RBP: ffffa08b6bb23808 R08: ffffa05b7ef5b780 R09: 0000000000000000 [Fri Jan 25 21:27:25 2019][702802.876686] R10: ffffa05b7ef5f140 R11: fffff7f6b91eca00 R12: ffffa08b6bb237a8 [Fri Jan 25 21:27:25 2019][702802.876687] R13: ffffa05bed7b3f40 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:25 2019][702802.876688] FS: 00007fc83477e880(0000) GS:ffffa05b7ef40000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:25 2019][702802.876689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:25 2019][702802.876690] CR2: 00007ffd4e567948 CR3: 00000010306d8000 CR4: 00000000003407e0 [Fri Jan 25 21:27:25 2019][702802.876691] Call Trace: [Fri Jan 25 21:27:25 2019][702802.876694] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:25 2019][702802.876696] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:25 2019][702802.876708] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:25 2019][702802.876720] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:25 2019][702802.876722] [] ? ktime_get+0x52/0xe0 [Fri Jan 25 21:27:25 2019][702802.876728] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Jan 25 21:27:25 2019][702802.876736] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:25 2019][702802.876738] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:25 2019][702802.876749] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:25 2019][702802.876759] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:25 2019][702802.876774] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:25 2019][702802.876817] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.876821] [] ? __slab_free+0x81/0x2f0 [Fri Jan 25 21:27:25 2019][702802.876857] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.876859] [] ? __slab_free+0x81/0x2f0 [Fri Jan 25 21:27:25 2019][702802.876901] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.876942] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.876981] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.876987] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:25 2019][702802.877025] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.877062] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.877063] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:25 2019][702802.877065] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:25 2019][702802.877102] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.877138] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:25 2019][702802.877140] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:25 2019][702802.877141] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:25 2019][702802.877143] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:25 2019][702802.877145] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:25 2019][702802.877163] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:25 2019][702802.884636] NMI watchdog: BUG: soft lockup - CPU#21 stuck for 23s! [mdt01_032:129374] [Fri Jan 25 21:27:25 2019][702802.884663] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:25 2019][702802.884674] CPU: 21 PID: 129374 Comm: mdt01_032 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:25 2019][702802.884675] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:25 2019][702802.884676] task: ffffa08b6dbb5140 ti: ffffa08b65e64000 task.ti: ffffa08b65e64000 [Fri Jan 25 21:27:25 2019][702802.884680] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:25 2019][702802.884681] RSP: 0018:ffffa08b65e67508 EFLAGS: 00000246 [Fri Jan 25 21:27:25 2019][702802.884682] RAX: 0000000000000000 RBX: ffffa05be78298f0 RCX: 0000000000a90000 [Fri Jan 25 21:27:25 2019][702802.884683] RDX: ffffa06b7f71b780 RSI: 0000000000890101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:25 2019][702802.884683] RBP: ffffa08b65e67508 R08: ffffa06b7f75b780 R09: 0000000000000000 [Fri Jan 25 21:27:25 2019][702802.884684] R10: ffffa06b7f75f140 R11: fffff7f6c0604c00 R12: ffffa08b65e674a8 [Fri Jan 25 21:27:25 2019][702802.884685] R13: ffffa05be78299d8 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:25 2019][702802.884686] FS: 00007f61a4229740(0000) GS:ffffa06b7f740000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:25 2019][702802.884687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:25 2019][702802.884688] CR2: 0000000001fcee78 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:25 2019][702802.884689] Call Trace: [Fri Jan 25 21:27:25 2019][702802.884691] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:25 2019][702802.884693] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:25 2019][702802.884706] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:25 2019][702802.884718] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:25 2019][702802.884752] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Jan 25 21:27:25 2019][702802.884764] [] ? dynlock_unlock+0x194/0x1e0 [osd_ldiskfs] [Fri Jan 25 21:27:26 2019][702802.884768] [] ? __brelse+0x3d/0x50 [Fri Jan 25 21:27:26 2019][702802.884777] [] ? iam_path_release+0x42/0x60 [osd_ldiskfs] [Fri Jan 25 21:27:26 2019][702802.884785] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:26 2019][702802.884794] [] ldiskfs_getblk+0x65/0x200 [ldiskfs] [Fri Jan 25 21:27:26 2019][702802.884801] [] ldiskfs_bread+0x27/0xc0 [ldiskfs] [Fri Jan 25 21:27:26 2019][702802.884808] [] ldiskfs_append+0x81/0x150 [ldiskfs] [Fri Jan 25 21:27:26 2019][702802.884816] [] ldiskfs_init_new_dir+0xcf/0x230 [ldiskfs] [Fri Jan 25 21:27:26 2019][702802.884817] [] ? __brelse+0x3d/0x50 [Fri Jan 25 21:27:26 2019][702802.884824] [] ldiskfs_add_dot_dotdot+0x4e/0x90 [ldiskfs] [Fri Jan 25 21:27:26 2019][702802.884832] [] osd_add_dot_dotdot_internal.isra.77+0x5f/0x80 [osd_ldiskfs] [Fri Jan 25 21:27:26 2019][702802.884841] [] osd_index_ea_insert+0xb1a/0x1240 [osd_ldiskfs] [Fri Jan 25 21:27:26 2019][702802.884857] [] lod_sub_insert+0x1c1/0x340 [lod] [Fri Jan 25 21:27:26 2019][702802.884866] [] lod_insert+0x24/0x30 [lod] [Fri Jan 25 21:27:26 2019][702802.884876] [] __mdd_index_insert_only+0x1cc/0x280 [mdd] [Fri Jan 25 21:27:26 2019][702802.884883] [] mdd_create_object+0x6c8/0x820 [mdd] [Fri Jan 25 21:27:26 2019][702802.884891] [] mdd_create+0xd80/0x1440 [mdd] [Fri Jan 25 21:27:26 2019][702802.884904] [] mdt_create+0xb54/0x1090 [mdt] [Fri Jan 25 21:27:26 2019][702802.884932] [] ? lprocfs_job_stats_log+0xd1/0x640 [obdclass] [Fri Jan 25 21:27:26 2019][702802.884945] [] mdt_reint_create+0x16b/0x360 [mdt] [Fri Jan 25 21:27:26 2019][702802.884958] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Jan 25 21:27:26 2019][702802.884970] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Jan 25 21:27:26 2019][702802.884981] [] mdt_reint+0x67/0x140 [mdt] [Fri Jan 25 21:27:26 2019][702802.885025] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.885065] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.885071] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:26 2019][702802.885109] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.885146] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.885148] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:26 2019][702802.885149] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:26 2019][702802.885186] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.885223] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.885224] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:26 2019][702802.885226] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:26 2019][702802.885228] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:26 2019][702802.885229] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:26 2019][702802.885248] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:26 2019][702802.919635] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! [mdt_io01_004:127221] [Fri Jan 25 21:27:26 2019][702802.919672] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:26 2019][702802.919688] CPU: 25 PID: 127221 Comm: mdt_io01_004 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:26 2019][702802.919689] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:26 2019][702802.919690] task: ffffa07b74621040 ti: ffffa08b5d070000 task.ti: ffffa08b5d070000 [Fri Jan 25 21:27:26 2019][702802.919695] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:26 2019][702802.919696] RSP: 0018:ffffa08b5d073808 EFLAGS: 00000246 [Fri Jan 25 21:27:26 2019][702802.919697] RAX: 0000000000000000 RBX: ffffa07369443608 RCX: 0000000000c90000 [Fri Jan 25 21:27:26 2019][702802.919697] RDX: ffffa05b7ef5b780 RSI: 0000000000a10101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:26 2019][702802.919698] RBP: ffffa08b5d073808 R08: ffffa06b7f79b780 R09: 0000000000000000 [Fri Jan 25 21:27:26 2019][702802.919699] R10: ffffa06b7f79f140 R11: fffff7f700d86000 R12: ffffa08b5d0737a8 [Fri Jan 25 21:27:26 2019][702802.919699] R13: ffffa073694436f0 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:26 2019][702802.919700] FS: 00007ff1c7e68840(0000) GS:ffffa06b7f780000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:26 2019][702802.919701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:26 2019][702802.919702] CR2: 00007ff1c73ef1a0 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:26 2019][702802.919703] Call Trace: [Fri Jan 25 21:27:26 2019][702802.919706] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:26 2019][702802.919710] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:26 2019][702802.919724] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:26 2019][702802.919736] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:26 2019][702802.919739] [] ? ktime_get+0x52/0xe0 [Fri Jan 25 21:27:26 2019][702802.919747] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Jan 25 21:27:26 2019][702802.919755] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:26 2019][702802.919757] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:26 2019][702802.919769] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:26 2019][702802.919779] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:26 2019][702802.919798] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:26 2019][702802.919846] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.919884] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.919887] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:26 2019][702802.919889] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:26 2019][702802.919929] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.919969] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.920007] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.920017] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:26 2019][702802.920054] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.920090] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.920092] [] ? wake_up_state+0x20/0x20 [Fri Jan 25 21:27:26 2019][702802.920128] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.920164] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.920166] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:26 2019][702802.920167] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:26 2019][702802.920169] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:26 2019][702802.920171] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:26 2019][702802.920189] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:26 2019][702802.928635] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 23s! [mdt_io02_013:127669] [Fri Jan 25 21:27:26 2019][702802.928662] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:26 2019][702802.928673] CPU: 26 PID: 127669 Comm: mdt_io02_013 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:26 2019][702802.928674] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:26 2019][702802.928674] task: ffffa08b76cce180 ti: ffffa08b7aad4000 task.ti: ffffa08b7aad4000 [Fri Jan 25 21:27:26 2019][702802.928677] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:26 2019][702802.928678] RSP: 0018:ffffa08b7aad7798 EFLAGS: 00000246 [Fri Jan 25 21:27:26 2019][702802.928679] RAX: 0000000000000000 RBX: ffffa05bebd15f98 RCX: 0000000000d10000 [Fri Jan 25 21:27:26 2019][702802.928679] RDX: ffffa06b7f79b780 RSI: 0000000000c90101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:26 2019][702802.928680] RBP: ffffa08b7aad7798 R08: ffffa07b7f79b780 R09: 0000000000000000 [Fri Jan 25 21:27:26 2019][702802.928681] R10: ffffa07b7f79f140 R11: fffff7f700c4a400 R12: ffffa08b7aad7738 [Fri Jan 25 21:27:26 2019][702802.928681] R13: ffffa05bebd16080 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:26 2019][702802.928683] FS: 00007efe99465700(0000) GS:ffffa07b7f780000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:26 2019][702802.928683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:26 2019][702802.928684] CR2: 00007efe9c7c6000 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:26 2019][702802.928685] Call Trace: [Fri Jan 25 21:27:26 2019][702802.928687] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:26 2019][702802.928688] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:26 2019][702802.928699] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:26 2019][702802.928710] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:26 2019][702802.928711] [] ? load_balance+0x178/0x9a0 [Fri Jan 25 21:27:26 2019][702802.928713] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:26 2019][702802.928721] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Fri Jan 25 21:27:26 2019][702802.928728] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:26 2019][702802.928736] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Fri Jan 25 21:27:26 2019][702802.928746] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:26 2019][702802.928756] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Fri Jan 25 21:27:26 2019][702802.928765] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Fri Jan 25 21:27:26 2019][702802.928780] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Fri Jan 25 21:27:26 2019][702802.928820] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.928860] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.928896] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:26 2019][702802.928898] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:26 2019][702802.928899] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:27 2019][702802.928930] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.928970] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.929008] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.929014] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:27 2019][702802.929050] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.929086] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.929087] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:27 2019][702802.929089] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:27 2019][702802.929124] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.929160] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.929161] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:27 2019][702802.929163] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:27 2019][702802.929165] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:27 2019][702802.929166] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:27 2019][702802.929184] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:27 2019][702802.945636] NMI watchdog: BUG: soft lockup - CPU#28 stuck for 24s! [mdt_io00_003:127220] [Fri Jan 25 21:27:27 2019][702802.945662] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:27 2019][702802.945673] CPU: 28 PID: 127220 Comm: mdt_io00_003 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:27 2019][702802.945674] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:27 2019][702802.945674] task: ffffa06b7897e180 ti: ffffa08b63514000 task.ti: ffffa08b63514000 [Fri Jan 25 21:27:27 2019][702802.945677] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x1d0/0x200 [Fri Jan 25 21:27:27 2019][702802.945678] RSP: 0018:ffffa08b63517808 EFLAGS: 00000202 [Fri Jan 25 21:27:27 2019][702802.945679] RAX: 0000000000610101 RBX: ffffa073694914c8 RCX: 0000000000000001 [Fri Jan 25 21:27:27 2019][702802.945680] RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:27 2019][702802.945680] RBP: ffffa08b63517808 R08: 0000000000000101 R09: ffffffffc11d0d1a [Fri Jan 25 21:27:27 2019][702802.945681] R10: ffffa05b7efdf140 R11: fffff7f780dcc800 R12: ffffa08b635177a8 [Fri Jan 25 21:27:27 2019][702802.945682] R13: ffffa073694915b0 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:27 2019][702802.945683] FS: 00007f711ec23740(0000) GS:ffffa05b7efc0000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:27 2019][702802.945684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:27 2019][702802.945684] CR2: 00007f711e67bd10 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:27 2019][702802.945685] Call Trace: [Fri Jan 25 21:27:27 2019][702802.945687] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:27 2019][702802.945689] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:27 2019][702802.945699] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:27 2019][702802.945710] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:27 2019][702802.945712] [] ? ___slab_alloc+0x209/0x4f0 [Fri Jan 25 21:27:27 2019][702802.945717] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Jan 25 21:27:27 2019][702802.945725] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:27 2019][702802.945727] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:27 2019][702802.945737] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:27 2019][702802.945747] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:27 2019][702802.945762] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:27 2019][702802.945802] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.945805] [] ? __slab_free+0x81/0x2f0 [Fri Jan 25 21:27:27 2019][702802.945806] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:27 2019][702802.945808] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:27 2019][702802.945810] [] ? __enqueue_entity+0x78/0x80 [Fri Jan 25 21:27:27 2019][702802.945849] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.945888] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.945926] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.945932] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:27 2019][702802.945969] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.946004] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.946006] [] ? wake_up_state+0x20/0x20 [Fri Jan 25 21:27:27 2019][702802.946041] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.946076] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.946078] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:27 2019][702802.946080] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:27 2019][702802.946081] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:27 2019][702802.946083] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:27 2019][702802.946101] Code: fe 00 01 00 00 74 f4 e9 93 fe ff ff 0f 1f 80 00 00 00 00 83 fa 01 75 11 0f 1f 00 e9 68 fe ff ff 0f 1f 00 85 c0 74 0c f3 90 8b 07 <0f> b6 c0 83 f8 03 75 f0 b8 01 00 00 00 66 89 07 5d c3 66 0f 1f [Fri Jan 25 21:27:27 2019][702802.953636] NMI watchdog: BUG: soft lockup - CPU#29 stuck for 24s! [mdt_io01_021:129240] [Fri Jan 25 21:27:27 2019][702802.953662] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:27 2019][702802.953673] CPU: 29 PID: 129240 Comm: mdt_io01_021 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:27 2019][702802.953674] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:27 2019][702802.953675] task: ffffa06b77cde180 ti: ffffa08b614f0000 task.ti: ffffa08b614f0000 [Fri Jan 25 21:27:27 2019][702802.953677] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Fri Jan 25 21:27:27 2019][702802.953678] RSP: 0018:ffffa08b614f3808 EFLAGS: 00000246 [Fri Jan 25 21:27:27 2019][702802.953679] RAX: 0000000000000000 RBX: ffffa052e7415f98 RCX: 0000000000e90000 [Fri Jan 25 21:27:27 2019][702802.953680] RDX: ffffa07b7f7db780 RSI: 0000000000f10101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:27 2019][702802.953680] RBP: ffffa08b614f3808 R08: ffffa06b7f7db780 R09: 0000000000000000 [Fri Jan 25 21:27:27 2019][702802.953681] R10: ffffa06b7f7df140 R11: fffff7f73a2a6a00 R12: ffffa08b614f37a8 [Fri Jan 25 21:27:27 2019][702802.953681] R13: ffffa052e7416080 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:27 2019][702802.953683] FS: 00007f1758524740(0000) GS:ffffa06b7f7c0000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:27 2019][702802.953683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:27 2019][702802.953684] CR2: 00007f1757da71a0 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:27 2019][702802.953685] Call Trace: [Fri Jan 25 21:27:27 2019][702802.953687] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:27 2019][702802.953689] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:27 2019][702802.953699] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:27 2019][702802.953710] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:27 2019][702802.953713] [] ? ktime_get+0x52/0xe0 [Fri Jan 25 21:27:27 2019][702802.953718] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Jan 25 21:27:27 2019][702802.953726] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:27 2019][702802.953727] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:27 2019][702802.953738] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:27 2019][702802.953748] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:27 2019][702802.953762] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:27 2019][702802.953803] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.953840] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.953875] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.953877] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:27 2019][702802.953879] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:27 2019][702802.953881] [] ? __mutex_unlock_slowpath+0x70/0x90 [Fri Jan 25 21:27:27 2019][702802.953921] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.953962] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.954000] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.954006] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:27 2019][702802.954043] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.954078] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.954080] [] ? wake_up_state+0x20/0x20 [Fri Jan 25 21:27:27 2019][702802.954115] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.954151] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:27 2019][702802.954153] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:27 2019][702802.954154] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:27 2019][702802.954156] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:27 2019][702802.954158] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:27 2019][702802.954176] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Fri Jan 25 21:27:27 2019][702802.955688] LNetError: 126666:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.6.20@o2ib6 from 10.0.10.52@o2ib7 [Fri Jan 25 21:27:27 2019][702802.962636] NMI watchdog: BUG: soft lockup - CPU#30 stuck for 24s! [mdt_io02_004:127224] [Fri Jan 25 21:27:27 2019][702802.962661] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:28 2019][702802.962672] CPU: 30 PID: 127224 Comm: mdt_io02_004 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:28 2019][702802.962672] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:28 2019][702802.962673] task: ffffa07b74620000 ti: ffffa08b5def8000 task.ti: ffffa08b5def8000 [Fri Jan 25 21:27:28 2019][702802.962677] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x156/0x200 [Fri Jan 25 21:27:28 2019][702802.962678] RSP: 0018:ffffa08b5defb808 EFLAGS: 00000202 [Fri Jan 25 21:27:28 2019][702802.962678] RAX: 0000000000000101 RBX: ffffa052e740b1e0 RCX: 0000000000f10000 [Fri Jan 25 21:27:28 2019][702802.962679] RDX: 0000000000610101 RSI: 0000000000000101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:28 2019][702802.962680] RBP: ffffa08b5defb808 R08: ffffa07b7f7db780 R09: 0000000000000000 [Fri Jan 25 21:27:28 2019][702802.962680] R10: ffffa07b7f7df140 R11: fffff7f780fa8400 R12: ffffa08b5defb7a8 [Fri Jan 25 21:27:28 2019][702802.962681] R13: ffffa052e740b2c8 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:28 2019][702802.962682] FS: 00007efe9a467700(0000) GS:ffffa07b7f7c0000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:28 2019][702802.962683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:28 2019][702802.962684] CR2: 00007efe9c7c6000 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:28 2019][702802.962684] Call Trace: [Fri Jan 25 21:27:28 2019][702802.962687] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:28 2019][702802.962689] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:28 2019][702802.962700] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:28 2019][702802.962710] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:28 2019][702802.962712] [] ? ___slab_alloc+0x209/0x4f0 [Fri Jan 25 21:27:28 2019][702802.962721] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Jan 25 21:27:28 2019][702802.962728] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:28 2019][702802.962730] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:28 2019][702802.962739] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:28 2019][702802.962748] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:28 2019][702802.962760] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:28 2019][702802.962797] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.962800] [] ? __enqueue_entity+0x78/0x80 [Fri Jan 25 21:27:28 2019][702802.962835] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.962870] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.962903] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.962908] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:28 2019][702802.962940] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.962971] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.962974] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:28 2019][702802.962976] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:28 2019][702802.963006] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.963037] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.963038] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:28 2019][702802.963040] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:28 2019][702802.963042] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:28 2019][702802.963043] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:28 2019][702802.963060] Code: 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 <8b> 17 0f b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84 [Fri Jan 25 21:27:28 2019][702802.988636] NMI watchdog: BUG: soft lockup - CPU#33 stuck for 23s! [mdt_io01_007:127228] [Fri Jan 25 21:27:28 2019][702802.988672] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:28 2019][702802.988687] CPU: 33 PID: 127228 Comm: mdt_io01_007 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:28 2019][702802.988687] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:28 2019][702802.988688] task: ffffa08b76cc1040 ti: ffffa08b6093c000 task.ti: ffffa08b6093c000 [Fri Jan 25 21:27:28 2019][702802.988692] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Fri Jan 25 21:27:28 2019][702802.988693] RSP: 0018:ffffa08b6093f808 EFLAGS: 00000246 [Fri Jan 25 21:27:28 2019][702802.988694] RAX: 0000000000000000 RBX: ffffa05bebd23e58 RCX: 0000000001090000 [Fri Jan 25 21:27:28 2019][702802.988694] RDX: ffffa06b7f7db780 RSI: 0000000000e90101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:28 2019][702802.988695] RBP: ffffa08b6093f808 R08: ffffa06b7f81b780 R09: 0000000000000000 [Fri Jan 25 21:27:28 2019][702802.988696] R10: ffffa06b7f81f140 R11: fffff7f6f2429200 R12: ffffa08b6093f7a8 [Fri Jan 25 21:27:28 2019][702802.988696] R13: ffffa05bebd23f40 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:28 2019][702802.988697] FS: 00007f9ae59e9740(0000) GS:ffffa06b7f800000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:28 2019][702802.988698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:28 2019][702802.988699] CR2: 00007f9ae54072e0 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:28 2019][702802.988700] Call Trace: [Fri Jan 25 21:27:28 2019][702802.988702] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:28 2019][702802.988705] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:28 2019][702802.988718] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:28 2019][702802.988729] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:28 2019][702802.988732] [] ? ktime_get+0x52/0xe0 [Fri Jan 25 21:27:28 2019][702802.988739] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Jan 25 21:27:28 2019][702802.988746] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:28 2019][702802.988748] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:28 2019][702802.988760] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:28 2019][702802.988770] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:28 2019][702802.988789] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:28 2019][702802.988833] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.988869] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.988905] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.988907] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:28 2019][702802.988910] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:28 2019][702802.988949] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.988989] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.989027] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.989035] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:28 2019][702802.989071] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.989107] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.989109] [] ? wake_up_state+0x20/0x20 [Fri Jan 25 21:27:28 2019][702802.989144] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.989180] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:28 2019][702802.989181] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:28 2019][702802.989183] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:28 2019][702802.989185] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:28 2019][702802.989186] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:28 2019][702802.989204] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Fri Jan 25 21:27:28 2019][702802.997636] NMI watchdog: BUG: soft lockup - CPU#34 stuck for 23s! [mdt_io02_015:127692] [Fri Jan 25 21:27:28 2019][702802.997662] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:28 2019][702802.997674] CPU: 34 PID: 127692 Comm: mdt_io02_015 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:28 2019][702802.997674] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:28 2019][702802.997675] task: ffffa06b788030c0 ti: ffffa08b70f78000 task.ti: ffffa08b70f78000 [Fri Jan 25 21:27:28 2019][702802.997678] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Fri Jan 25 21:27:28 2019][702802.997679] RSP: 0018:ffffa08b70f7b798 EFLAGS: 00000246 [Fri Jan 25 21:27:28 2019][702802.997679] RAX: 0000000000000000 RBX: ffffa0736afe7460 RCX: 0000000001110000 [Fri Jan 25 21:27:28 2019][702802.997680] RDX: ffffa06b7f65b780 RSI: 0000000000290101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:28 2019][702802.997681] RBP: ffffa08b70f7b798 R08: ffffa07b7f81b780 R09: 0000000000000000 [Fri Jan 25 21:27:28 2019][702802.997681] R10: ffffa07b7f81f140 R11: fffff7f78080ee00 R12: ffffa08b70f7b738 [Fri Jan 25 21:27:29 2019][702802.997682] R13: ffffa0736afe7548 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:29 2019][702802.997683] FS: 00007efe99c66700(0000) GS:ffffa07b7f800000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:29 2019][702802.997684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:29 2019][702802.997685] CR2: 00007f9ae472f80d CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:29 2019][702802.997685] Call Trace: [Fri Jan 25 21:27:29 2019][702802.997687] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:29 2019][702802.997689] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:29 2019][702802.997700] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:29 2019][702802.997710] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:29 2019][702802.997716] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Fri Jan 25 21:27:29 2019][702802.997721] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Fri Jan 25 21:27:29 2019][702802.997723] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:29 2019][702802.997730] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Fri Jan 25 21:27:29 2019][702802.997738] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:29 2019][702802.997745] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Fri Jan 25 21:27:29 2019][702802.997755] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:29 2019][702802.997765] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Fri Jan 25 21:27:29 2019][702802.997775] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Fri Jan 25 21:27:29 2019][702802.997789] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Fri Jan 25 21:27:29 2019][702802.997829] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Fri Jan 25 21:27:29 2019][702802.997869] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Fri Jan 25 21:27:29 2019][702802.997905] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:29 2019][702802.997907] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:29 2019][702802.997909] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:29 2019][702802.997940] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Fri Jan 25 21:27:29 2019][702802.997979] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:29 2019][702802.998017] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:29 2019][702802.998023] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:29 2019][702802.998060] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:29 2019][702802.998095] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:29 2019][702802.998097] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:29 2019][702802.998098] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:29 2019][702802.998134] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:29 2019][702802.998169] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:29 2019][702802.998171] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:29 2019][702802.998172] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:29 2019][702802.998174] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:29 2019][702802.998176] [] ? ret_from_fork+0x48/0x56 [Fri Jan 25 21:27:29 2019][702802.998177] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:29 2019][702802.998195] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Fri Jan 25 21:27:29 2019][702803.014637] NMI watchdog: BUG: soft lockup - CPU#36 stuck for 23s! [mdt_io00_025:1416] [Fri Jan 25 21:27:29 2019][702803.014664] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:29 2019][702803.014676] CPU: 36 PID: 1416 Comm: mdt_io00_025 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:29 2019][702803.014677] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:29 2019][702803.014678] task: ffffa07b3dd7a080 ti: ffffa08a5d6c8000 task.ti: ffffa08a5d6c8000 [Fri Jan 25 21:27:29 2019][702803.014682] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:29 2019][702803.014683] RSP: 0018:ffffa08a5d6cb798 EFLAGS: 00000246 [Fri Jan 25 21:27:29 2019][702803.014684] RAX: 0000000000000000 RBX: ffffa073694498f0 RCX: 0000000001210000 [Fri Jan 25 21:27:29 2019][702803.014684] RDX: ffffa06b7f75b780 RSI: 0000000000a90101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:29 2019][702803.014685] RBP: ffffa08a5d6cb798 R08: ffffa05b7f05b780 R09: 0000000000000000 [Fri Jan 25 21:27:29 2019][702803.014686] R10: ffffa05b7f05f140 R11: fffff7f780a7bc00 R12: ffffa08a5d6cb738 [Fri Jan 25 21:27:29 2019][702803.014686] R13: ffffa073694499d8 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:29 2019][702803.014688] FS: 00007f3205d71740(0000) GS:ffffa05b7f040000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:29 2019][702803.014689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:29 2019][702803.014689] CR2: 0000000001f9c7d8 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:29 2019][702803.014690] Call Trace: [Fri Jan 25 21:27:29 2019][702803.014693] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:29 2019][702803.014695] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:29 2019][702803.014707] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:29 2019][702803.014719] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:29 2019][702803.014729] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Fri Jan 25 21:27:29 2019][702803.014737] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:29 2019][702803.014746] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Fri Jan 25 21:27:29 2019][702803.014756] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:29 2019][702803.014767] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Fri Jan 25 21:27:29 2019][702803.014777] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Fri Jan 25 21:27:29 2019][702803.014791] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Fri Jan 25 21:27:29 2019][702803.014835] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Fri Jan 25 21:27:29 2019][702803.014876] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Fri Jan 25 21:27:29 2019][702803.014878] [] ? get_partial_node.isra.49+0xfe/0x230 [Fri Jan 25 21:27:29 2019][702803.014910] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Fri Jan 25 21:27:29 2019][702803.014951] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:29 2019][702803.014990] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:29 2019][702803.014996] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:29 2019][702803.015034] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:29 2019][702803.015071] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:29 2019][702803.015073] [] ? kmalloc_order_trace+0x2e/0xa0 [Fri Jan 25 21:27:29 2019][702803.015109] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:29 2019][702803.015146] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:29 2019][702803.015147] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:29 2019][702803.015149] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:29 2019][702803.015151] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:29 2019][702803.015152] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:29 2019][702803.015171] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:29 2019][702803.022636] NMI watchdog: BUG: soft lockup - CPU#37 stuck for 23s! [mdt_io01_003:127219] [Fri Jan 25 21:27:29 2019][702803.022662] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:29 2019][702803.022673] CPU: 37 PID: 127219 Comm: mdt_io01_003 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:29 2019][702803.022674] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:29 2019][702803.022675] task: ffffa06b78978000 ti: ffffa08b7867c000 task.ti: ffffa08b7867c000 [Fri Jan 25 21:27:29 2019][702803.022677] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:29 2019][702803.022678] RSP: 0018:ffffa08b7867f808 EFLAGS: 00000246 [Fri Jan 25 21:27:29 2019][702803.022679] RAX: 0000000000000000 RBX: ffffa0736944cef8 RCX: 0000000001290000 [Fri Jan 25 21:27:29 2019][702803.022680] RDX: ffffa06b7f8db780 RSI: 0000000001690101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:29 2019][702803.022680] RBP: ffffa08b7867f808 R08: ffffa06b7f85b780 R09: 0000000000000000 [Fri Jan 25 21:27:29 2019][702803.022681] R10: ffffa06b7f85f140 R11: fffff7f77d5cf800 R12: ffffa08b7867f7a8 [Fri Jan 25 21:27:29 2019][702803.022682] R13: ffffa0736944cfe0 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:29 2019][702803.022683] FS: 00007f3bdbac6740(0000) GS:ffffa06b7f840000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:29 2019][702803.022684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:29 2019][702803.022684] CR2: 00000000011a1108 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:29 2019][702803.022685] Call Trace: [Fri Jan 25 21:27:29 2019][702803.022687] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:29 2019][702803.022689] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:29 2019][702803.022699] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:29 2019][702803.022710] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:29 2019][702803.022712] [] ? ktime_get+0x52/0xe0 [Fri Jan 25 21:27:29 2019][702803.022717] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Jan 25 21:27:29 2019][702803.022725] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:29 2019][702803.022726] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:29 2019][702803.022737] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:29 2019][702803.022746] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:29 2019][702803.022761] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:29 2019][702803.022802] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:29 2019][702803.022804] [] ? __slab_free+0x81/0x2f0 [Fri Jan 25 21:27:30 2019][702803.022840] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.022841] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:30 2019][702803.022843] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:30 2019][702803.022883] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.022922] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.022960] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.022966] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:30 2019][702803.023002] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.023038] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.023039] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:30 2019][702803.023041] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:30 2019][702803.023076] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.023112] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.023113] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:30 2019][702803.023115] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:30 2019][702803.023117] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:30 2019][702803.023118] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:30 2019][702803.023136] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:30 2019][702803.031637] NMI watchdog: BUG: soft lockup - CPU#38 stuck for 23s! [mdt_io02_000:126671] [Fri Jan 25 21:27:30 2019][702803.031663] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:30 2019][702803.031674] CPU: 38 PID: 126671 Comm: mdt_io02_000 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:30 2019][702803.031674] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:30 2019][702803.031675] task: ffffa08b5f2c1040 ti: ffffa08b7977c000 task.ti: ffffa08b7977c000 [Fri Jan 25 21:27:30 2019][702803.031678] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:30 2019][702803.031678] RSP: 0018:ffffa08b7977f798 EFLAGS: 00000246 [Fri Jan 25 21:27:30 2019][702803.031679] RAX: 0000000000000000 RBX: ffffa052e7454ef8 RCX: 0000000001310000 [Fri Jan 25 21:27:30 2019][702803.031680] RDX: ffffa06b7f6db780 RSI: 0000000000690101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:30 2019][702803.031680] RBP: ffffa08b7977f798 R08: ffffa07b7f85b780 R09: 0000000000000000 [Fri Jan 25 21:27:30 2019][702803.031681] R10: ffffa07b7f85f140 R11: fffff7f735130200 R12: ffffa08b7977f738 [Fri Jan 25 21:27:30 2019][702803.031682] R13: ffffa052e7454fe0 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:30 2019][702803.031683] FS: 00007f9d4bd7f740(0000) GS:ffffa07b7f840000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:30 2019][702803.031684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:30 2019][702803.031685] CR2: 0000000002516aa8 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:30 2019][702803.031685] Call Trace: [Fri Jan 25 21:27:30 2019][702803.031687] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:30 2019][702803.031689] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:30 2019][702803.031699] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:30 2019][702803.031710] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:30 2019][702803.031716] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Fri Jan 25 21:27:30 2019][702803.031721] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Fri Jan 25 21:27:30 2019][702803.031723] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:30 2019][702803.031734] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Fri Jan 25 21:27:30 2019][702803.031742] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:30 2019][702803.031750] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Fri Jan 25 21:27:30 2019][702803.031760] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:30 2019][702803.031770] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Fri Jan 25 21:27:30 2019][702803.031779] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Fri Jan 25 21:27:30 2019][702803.031794] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Fri Jan 25 21:27:30 2019][702803.031835] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.031875] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.031913] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.031949] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.031950] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:30 2019][702803.031952] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:30 2019][702803.031983] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.032022] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.032060] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.032066] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:30 2019][702803.032103] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.032139] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.032140] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:30 2019][702803.032142] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:30 2019][702803.032177] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.032213] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.032214] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:30 2019][702803.032216] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:30 2019][702803.032217] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:30 2019][702803.032219] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:30 2019][702803.032237] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:30 2019][702803.056642] NMI watchdog: BUG: soft lockup - CPU#41 stuck for 24s! [mdt01_011:128319] [Fri Jan 25 21:27:30 2019][702803.056677] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:30 2019][702803.056692] CPU: 41 PID: 128319 Comm: mdt01_011 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:30 2019][702803.056693] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:30 2019][702803.056694] task: ffffa07b74acc100 ti: ffffa08b7bb24000 task.ti: ffffa08b7bb24000 [Fri Jan 25 21:27:30 2019][702803.056708] RIP: 0010:[] [] ldiskfs_inode_touch_time_cmp+0xd/0x90 [ldiskfs] [Fri Jan 25 21:27:30 2019][702803.056709] RSP: 0018:ffffa08b7bb27310 EFLAGS: 00000286 [Fri Jan 25 21:27:30 2019][702803.056710] RAX: 8000040400080000 RBX: ffffffffa7419f22 RCX: 00000001288e6be2 [Fri Jan 25 21:27:30 2019][702803.056710] RDX: ffffa05d3ff83560 RSI: ffffa079f7459848 RDI: 0000000000000000 [Fri Jan 25 21:27:30 2019][702803.056711] RBP: ffffa08b7bb27360 R08: ffffa08b7bb27490 R09: 0000000000ce8740 [Fri Jan 25 21:27:30 2019][702803.056712] R10: 000000001fca3d01 R11: fffff7f6ff7f28c0 R12: 0000000000000000 [Fri Jan 25 21:27:30 2019][702803.056712] R13: ffffa04ca9076c80 R14: ffffa06b7823c400 R15: 0000000000000000 [Fri Jan 25 21:27:30 2019][702803.056714] FS: 00007f96a17e0740(0000) GS:ffffa06b7f880000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:30 2019][702803.056714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:30 2019][702803.056715] CR2: 00007f96a11fe2e0 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:30 2019][702803.056716] Call Trace: [Fri Jan 25 21:27:30 2019][702803.056720] [] ? merge+0x62/0xc0 [Fri Jan 25 21:27:30 2019][702803.056731] [] ? ldiskfs_init_inode_table+0x410/0x410 [ldiskfs] [Fri Jan 25 21:27:30 2019][702803.056733] [] list_sort+0x9b/0x250 [Fri Jan 25 21:27:30 2019][702803.056743] [] __ldiskfs_es_shrink+0x1ce/0x2a0 [ldiskfs] [Fri Jan 25 21:27:30 2019][702803.056752] [] ldiskfs_es_shrink+0xb4/0x130 [ldiskfs] [Fri Jan 25 21:27:30 2019][702803.056756] [] shrink_slab+0x175/0x340 [Fri Jan 25 21:27:30 2019][702803.056758] [] ? zone_watermark_ok+0x1f/0x30 [Fri Jan 25 21:27:30 2019][702803.056761] [] ? compaction_suitable+0xa3/0xb0 [Fri Jan 25 21:27:30 2019][702803.056763] [] zone_reclaim+0x1d1/0x2f0 [Fri Jan 25 21:27:30 2019][702803.056765] [] get_page_from_freelist+0x87b/0xa70 [Fri Jan 25 21:27:30 2019][702803.056768] [] __alloc_pages_nodemask+0x176/0x420 [Fri Jan 25 21:27:30 2019][702803.056770] [] alloc_pages_current+0x98/0x110 [Fri Jan 25 21:27:30 2019][702803.056772] [] __get_free_pages+0xe/0x40 [Fri Jan 25 21:27:30 2019][702803.056774] [] kmalloc_order_trace+0x2e/0xa0 [Fri Jan 25 21:27:30 2019][702803.056777] [] __kmalloc+0x211/0x230 [Fri Jan 25 21:27:30 2019][702803.056819] [] null_alloc_rs+0x16d/0x340 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.056857] [] sptlrpc_svc_alloc_rs+0x66/0x350 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.056873] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Jan 25 21:27:30 2019][702803.056910] [] lustre_pack_reply_v2+0x8a/0x280 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.056946] [] lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.056982] [] lustre_pack_reply+0x11/0x20 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.057019] [] req_capsule_server_pack+0x43/0xf0 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.057034] [] mdt_getxattr+0x71c/0x12e0 [mdt] [Fri Jan 25 21:27:30 2019][702803.057046] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Jan 25 21:27:30 2019][702803.057081] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.057093] [] mdt_intent_getxattr+0xc5/0x270 [mdt] [Fri Jan 25 21:27:30 2019][702803.057105] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Jan 25 21:27:30 2019][702803.057117] [] ? mdt_intent_getattr+0x480/0x480 [mdt] [Fri Jan 25 21:27:30 2019][702803.057147] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Jan 25 21:27:30 2019][702803.057158] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Jan 25 21:27:31 2019][702803.057166] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Jan 25 21:27:31 2019][702803.057198] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.057234] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.057275] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.057315] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.057353] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.057359] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:31 2019][702803.057395] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.057430] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.057433] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:31 2019][702803.057435] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:31 2019][702803.057471] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.057506] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.057508] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:31 2019][702803.057510] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:31 2019][702803.057512] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:31 2019][702803.057513] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:31 2019][702803.057531] Code: ff 8d 4a 01 89 d0 f0 0f b1 0f 39 d0 0f 84 fb fd ff ff 89 c2 eb e2 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 86 e8 fc ff ff <48> 89 e5 48 c1 e8 2b a8 01 74 15 48 8b 8a e8 fc ff ff b8 01 00 [Fri Jan 25 21:27:31 2019][702803.065637] NMI watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [mdt_io02_014:127690] [Fri Jan 25 21:27:31 2019][702803.065662] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:31 2019][702803.065673] CPU: 42 PID: 127690 Comm: mdt_io02_014 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:31 2019][702803.065673] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:31 2019][702803.065674] task: ffffa06b78806180 ti: ffffa08b7bf58000 task.ti: ffffa08b7bf58000 [Fri Jan 25 21:27:31 2019][702803.065677] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:31 2019][702803.065678] RSP: 0018:ffffa08b7bf5b808 EFLAGS: 00000246 [Fri Jan 25 21:27:31 2019][702803.065679] RAX: 0000000000000000 RBX: ffffa052e93d2140 RCX: 0000000001510000 [Fri Jan 25 21:27:31 2019][702803.065679] RDX: ffffa07b7f71b780 RSI: 0000000000910101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:31 2019][702803.065680] RBP: ffffa08b7bf5b808 R08: ffffa07b7f89b780 R09: 0000000000000000 [Fri Jan 25 21:27:31 2019][702803.065680] R10: ffffa07b7f89f140 R11: fffff7f700ee0600 R12: ffffa08b7bf5b7a8 [Fri Jan 25 21:27:31 2019][702803.065681] R13: ffffa052e93d2228 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:31 2019][702803.065682] FS: 00007f06ea4f9880(0000) GS:ffffa07b7f880000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:31 2019][702803.065683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:31 2019][702803.065684] CR2: 00007f06e1f73470 CR3: 000000103ba82000 CR4: 00000000003407e0 [Fri Jan 25 21:27:31 2019][702803.065684] Call Trace: [Fri Jan 25 21:27:31 2019][702803.065686] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:31 2019][702803.065688] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:31 2019][702803.065699] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:31 2019][702803.065710] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:31 2019][702803.065712] [] ? ktime_get+0x52/0xe0 [Fri Jan 25 21:27:31 2019][702803.065717] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Jan 25 21:27:31 2019][702803.065725] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:31 2019][702803.065727] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:31 2019][702803.065737] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:31 2019][702803.065747] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:31 2019][702803.065761] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:31 2019][702803.065802] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.065804] [] ? __slab_free+0x81/0x2f0 [Fri Jan 25 21:27:31 2019][702803.065839] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.065841] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:31 2019][702803.065843] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:31 2019][702803.065882] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.065922] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.065959] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.065966] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:31 2019][702803.066002] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.066038] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.066039] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:31 2019][702803.066041] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:31 2019][702803.066076] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.066112] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.066113] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:31 2019][702803.066115] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:31 2019][702803.066117] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:31 2019][702803.066118] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:31 2019][702803.066136] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:31 2019][702803.082639] NMI watchdog: BUG: soft lockup - CPU#44 stuck for 22s! [mdt_io00_015:127702] [Fri Jan 25 21:27:31 2019][702803.082664] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:31 2019][702803.082675] CPU: 44 PID: 127702 Comm: mdt_io00_015 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:31 2019][702803.082675] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:31 2019][702803.082676] task: ffffa07b3baba080 ti: ffffa08b7be50000 task.ti: ffffa08b7be50000 [Fri Jan 25 21:27:31 2019][702803.082679] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Fri Jan 25 21:27:31 2019][702803.082680] RSP: 0018:ffffa08b7be53808 EFLAGS: 00000246 [Fri Jan 25 21:27:31 2019][702803.082680] RAX: 0000000000000000 RBX: ffffa05bebebe3c0 RCX: 0000000001610000 [Fri Jan 25 21:27:31 2019][702803.082681] RDX: ffffa06b7f69b780 RSI: 0000000000490101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:31 2019][702803.082682] RBP: ffffa08b7be53808 R08: ffffa05b7f0db780 R09: 0000000000000000 [Fri Jan 25 21:27:31 2019][702803.082682] R10: ffffa05b7f0df140 R11: fffff7f780863800 R12: ffffa08b7be537a8 [Fri Jan 25 21:27:31 2019][702803.082683] R13: ffffa05bebebe4a8 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:31 2019][702803.082684] FS: 00007fc83477e880(0000) GS:ffffa05b7f0c0000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:31 2019][702803.082685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:31 2019][702803.082686] CR2: 00007f06e16ddeb8 CR3: 00000010306d8000 CR4: 00000000003407e0 [Fri Jan 25 21:27:31 2019][702803.082686] Call Trace: [Fri Jan 25 21:27:31 2019][702803.082688] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:31 2019][702803.082690] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:31 2019][702803.082701] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:31 2019][702803.082712] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:31 2019][702803.082713] [] ? ktime_get+0x52/0xe0 [Fri Jan 25 21:27:31 2019][702803.082719] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Jan 25 21:27:31 2019][702803.082727] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:31 2019][702803.082729] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:31 2019][702803.082738] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:31 2019][702803.082748] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:31 2019][702803.082763] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:31 2019][702803.082803] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.082805] [] ? __slab_free+0x81/0x2f0 [Fri Jan 25 21:27:31 2019][702803.082841] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.082843] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:31 2019][702803.082845] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:31 2019][702803.082884] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.082923] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.082961] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.082967] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:31 2019][702803.083004] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.083039] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.083041] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:31 2019][702803.083042] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:31 2019][702803.083078] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.083114] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:31 2019][702803.083115] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:31 2019][702803.083116] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:31 2019][702803.083118] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:31 2019][702803.083120] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:31 2019][702803.083138] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Fri Jan 25 21:27:31 2019][702803.091637] NMI watchdog: BUG: soft lockup - CPU#45 stuck for 22s! [mdt_io01_010:127262] [Fri Jan 25 21:27:31 2019][702803.091663] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:32 2019][702803.091675] CPU: 45 PID: 127262 Comm: mdt_io01_010 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:32 2019][702803.091675] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:32 2019][702803.091676] task: ffffa08b6ce21040 ti: ffffa08b78178000 task.ti: ffffa08b78178000 [Fri Jan 25 21:27:32 2019][702803.091679] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Fri Jan 25 21:27:32 2019][702803.091680] RSP: 0018:ffffa08b7817b808 EFLAGS: 00000246 [Fri Jan 25 21:27:32 2019][702803.091680] RAX: 0000000000000000 RBX: ffffa0736affa140 RCX: 0000000001690000 [Fri Jan 25 21:27:32 2019][702803.091681] RDX: ffffa05b7f05b780 RSI: 0000000001210101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:32 2019][702803.091682] RBP: ffffa08b7817b808 R08: ffffa06b7f8db780 R09: 0000000000000000 [Fri Jan 25 21:27:32 2019][702803.091682] R10: ffffa06b7f8df140 R11: fffff7f73ff04c00 R12: ffffa08b7817b7a8 [Fri Jan 25 21:27:32 2019][702803.091683] R13: ffffa0736affa228 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:32 2019][702803.091684] FS: 00007f8530c3c740(0000) GS:ffffa06b7f8c0000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:32 2019][702803.091685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:32 2019][702803.091686] CR2: 00007f853074f1e0 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:32 2019][702803.091686] Call Trace: [Fri Jan 25 21:27:32 2019][702803.091688] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:32 2019][702803.091690] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:32 2019][702803.091700] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:32 2019][702803.091711] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:32 2019][702803.091713] [] ? ktime_get+0x52/0xe0 [Fri Jan 25 21:27:32 2019][702803.091718] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Jan 25 21:27:32 2019][702803.091726] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:32 2019][702803.091727] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:32 2019][702803.091737] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:32 2019][702803.091747] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:32 2019][702803.091762] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:32 2019][702803.091803] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.091805] [] ? __slab_free+0x81/0x2f0 [Fri Jan 25 21:27:32 2019][702803.091841] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.091843] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:32 2019][702803.091845] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:32 2019][702803.091885] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.091924] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.091962] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.091969] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:32 2019][702803.092005] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.092007] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:32 2019][702803.092008] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:32 2019][702803.092044] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.092080] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.092081] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:32 2019][702803.092082] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:32 2019][702803.092084] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:32 2019][702803.092086] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:32 2019][702803.092104] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Fri Jan 25 21:27:32 2019][702803.100637] NMI watchdog: BUG: soft lockup - CPU#46 stuck for 24s! [mdt_io02_008:127265] [Fri Jan 25 21:27:32 2019][702803.100661] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:32 2019][702803.100672] CPU: 46 PID: 127265 Comm: mdt_io02_008 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:32 2019][702803.100672] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:32 2019][702803.100673] task: ffffa08b6ce24100 ti: ffffa08b7c334000 task.ti: ffffa08b7c334000 [Fri Jan 25 21:27:32 2019][702803.100677] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:32 2019][702803.100678] RSP: 0018:ffffa08b7c337808 EFLAGS: 00000246 [Fri Jan 25 21:27:32 2019][702803.100679] RAX: 0000000000000000 RBX: ffffa052e9390000 RCX: 0000000001710000 [Fri Jan 25 21:27:32 2019][702803.100679] RDX: ffffa06b7f81b780 RSI: 0000000001090101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:32 2019][702803.100680] RBP: ffffa08b7c337808 R08: ffffa07b7f8db780 R09: 0000000000000000 [Fri Jan 25 21:27:32 2019][702803.100681] R10: ffffa07b7f8df140 R11: fffff7f740644e00 R12: ffffa08b7c3377a8 [Fri Jan 25 21:27:32 2019][702803.100681] R13: ffffa052e93900e8 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:32 2019][702803.100682] FS: 00007efe98463700(0000) GS:ffffa07b7f8c0000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:32 2019][702803.100683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:32 2019][702803.100684] CR2: 00007efe9c7c6000 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Fri Jan 25 21:27:32 2019][702803.100685] Call Trace: [Fri Jan 25 21:27:32 2019][702803.100687] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:32 2019][702803.100689] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:32 2019][702803.100700] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:32 2019][702803.100711] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:32 2019][702803.100713] [] ? ktime_get+0x52/0xe0 [Fri Jan 25 21:27:32 2019][702803.100717] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Jan 25 21:27:32 2019][702803.100725] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:32 2019][702803.100726] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:32 2019][702803.100736] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:32 2019][702803.100745] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:32 2019][702803.100757] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:32 2019][702803.100793] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.100825] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.100855] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.100857] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:32 2019][702803.100859] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:32 2019][702803.100860] [] ? __enqueue_entity+0x78/0x80 [Fri Jan 25 21:27:32 2019][702803.100895] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.100929] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.100962] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.100968] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:32 2019][702803.100999] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.101030] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.101032] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:32 2019][702803.101033] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:32 2019][702803.101064] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.101094] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:32 2019][702803.101096] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:32 2019][702803.101097] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:32 2019][702803.101099] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:32 2019][702803.101100] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:32 2019][702803.101117] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:32 2019][702805.723880] Lustre: mdt: This server is not able to keep up with request traffic (cpu-bound). [Fri Jan 25 21:27:32 2019][702805.723883] Lustre: 126668:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=17 reqQ=0 recA=17, svcEst=31, delay=26799 [Fri Jan 25 21:27:32 2019][702805.723884] Lustre: Skipped 3 previous similar messages [Fri Jan 25 21:27:32 2019][702805.723889] Lustre: 126469:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1548480587/real 1548480587] req@ffffa0736fba7500 x1623663728749056/t0(0) o13->fir-OST001b-osc-MDT0003@10.0.10.106@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1548480594 ref 1 fl Rpc:RX/0/ffffffff rc 0/-1 [Fri Jan 25 21:27:32 2019][702805.723892] Lustre: 126668:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 2 previous similar messages [Fri Jan 25 21:27:32 2019][702805.723894] Lustre: 126469:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 70 previous similar messages [Fri Jan 25 21:27:32 2019][702805.724015] LNetError: 126432:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) [Fri Jan 25 21:27:32 2019][702805.724110] Lustre: 126668:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-12s), not sending early reply. Consider increasing at_early_margin (5)? req@ffffa06382223050 x1623670114895760/t0(0) o4->9daa8e10-c154-ca96-d85d-3f6355b9ec3f@10.8.2.5@o2ib6:518/0 lens 624/448 e 1 to 0 dl 1548480603 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Jan 25 21:27:32 2019][702805.724112] Lustre: 126668:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 6 previous similar messages [Fri Jan 25 21:27:32 2019][702805.724169] Lustre: fir-MDT0003: Client fir-MDT0003-lwp-OST0002_UUID (at 10.0.10.101@o2ib7) reconnecting [Fri Jan 25 21:27:32 2019][702805.724171] Lustre: Skipped 8 previous similar messages [Fri Jan 25 21:27:32 2019][702805.724182] LustreError: 127695:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.6.33@o2ib6: deadline 6:22s ago [Fri Jan 25 21:27:32 2019][702805.724182] req@ffffa06b77373850 x1623661016630768/t0(0) o4->cc9aeebd-5eb1-fd0a-28e5-4a7c51fd2782@10.8.6.33@o2ib6:508/0 lens 488/0 e 0 to 0 dl 1548480593 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [Fri Jan 25 21:27:33 2019][702805.724191] Lustre: fir-MDT0001: Connection restored to 10.0.10.101@o2ib7 (at 10.0.10.101@o2ib7) [Fri Jan 25 21:27:33 2019][702805.724192] Lustre: fir-MDT0003: Connection restored to 10.0.10.101@o2ib7 (at 10.0.10.101@o2ib7) [Fri Jan 25 21:27:33 2019][702805.724196] Lustre: 127695:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:22s); client may timeout. req@ffffa06b77373850 x1623661016630768/t0(0) o4->cc9aeebd-5eb1-fd0a-28e5-4a7c51fd2782@10.8.6.33@o2ib6:508/0 lens 488/0 e 0 to 0 dl 1548480593 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [Fri Jan 25 21:27:33 2019][702805.724198] Lustre: 127695:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [Fri Jan 25 21:27:33 2019][702806.807667] NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [mdt_io00_018:129263] [Fri Jan 25 21:27:33 2019][702806.807694] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:33 2019][702806.807706] CPU: 12 PID: 129263 Comm: mdt_io00_018 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:33 2019][702806.807706] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:33 2019][702806.807707] task: ffffa07b7a276180 ti: ffffa08b647e4000 task.ti: ffffa08b647e4000 [Fri Jan 25 21:27:33 2019][702806.807711] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Fri Jan 25 21:27:33 2019][702806.807711] RSP: 0018:ffffa08b647e78f0 EFLAGS: 00000246 [Fri Jan 25 21:27:33 2019][702806.807712] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000610000 [Fri Jan 25 21:27:33 2019][702806.807713] RDX: ffffa06b7f85b780 RSI: 0000000001290101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:33 2019][702806.807714] RBP: ffffa08b647e78f0 R08: ffffa05b7eedb780 R09: 0000000000000000 [Fri Jan 25 21:27:33 2019][702806.807715] R10: 000007e9777de000 R11: ffffa06b077fb0a8 R12: ffffa07b4a2a0378 [Fri Jan 25 21:27:33 2019][702806.807715] R13: ffffa08b647e7888 R14: ffffa05b7eedac00 R15: ffffa07b7a2761e8 [Fri Jan 25 21:27:33 2019][702806.807717] FS: 00007fc83477e880(0000) GS:ffffa05b7eec0000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:33 2019][702806.807717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:33 2019][702806.807718] CR2: 00000000027449c8 CR3: 00000010306d8000 CR4: 00000000003407e0 [Fri Jan 25 21:27:33 2019][702806.807719] Call Trace: [Fri Jan 25 21:27:33 2019][702806.807721] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:33 2019][702806.807723] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:33 2019][702806.807733] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:33 2019][702806.807742] [] ldiskfs_map_blocks+0x210/0x700 [ldiskfs] [Fri Jan 25 21:27:33 2019][702806.807743] [] ? ktime_get_ts64+0x52/0xf0 [Fri Jan 25 21:27:33 2019][702806.807754] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:33 2019][702806.807764] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Fri Jan 25 21:27:33 2019][702806.807779] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Fri Jan 25 21:27:33 2019][702806.807821] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Fri Jan 25 21:27:33 2019][702806.807823] [] ? __slab_free+0x81/0x2f0 [Fri Jan 25 21:27:33 2019][702806.807859] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:33 2019][702806.807862] [] ? __mutex_lock_slowpath+0x1b8/0x1d0 [Fri Jan 25 21:27:33 2019][702806.807864] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:33 2019][702806.807904] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Jan 25 21:27:33 2019][702806.807943] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:33 2019][702806.807981] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:33 2019][702806.807987] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:33 2019][702806.808023] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:33 2019][702806.808059] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:33 2019][702806.808061] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:33 2019][702806.808062] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:33 2019][702806.808098] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:33 2019][702806.808134] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:33 2019][702806.808135] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:33 2019][702806.808137] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:33 2019][702806.808139] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:33 2019][702806.808140] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:33 2019][702806.808158] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Fri Jan 25 21:27:33 2019][702813.971747] Lustre: fir-OST001e-osc-MDT0001: Connection to fir-OST001e (at 10.0.10.105@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Fri Jan 25 21:27:33 2019][702813.971749] Lustre: Skipped 23 previous similar messages [Fri Jan 25 21:27:33 2019][702815.256132] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Fri Jan 25 21:27:33 2019][702815.361288] CPU: 1 PID: 127261 Comm: mdt_io01_009 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Jan 25 21:27:33 2019][702815.373879] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Fri Jan 25 21:27:33 2019][702815.381531] task: ffffa08b6ce20000 ti: ffffa08b5d6c0000 task.ti: ffffa08b5d6c0000 [Fri Jan 25 21:27:33 2019][702815.389099] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Fri Jan 25 21:27:33 2019][702815.399117] RSP: 0018:ffffa08b5d6c3798 EFLAGS: 00000246 [Fri Jan 25 21:27:33 2019][702815.404515] RAX: 0000000000000000 RBX: ffffa05bebd28428 RCX: 0000000000090000 [Fri Jan 25 21:27:33 2019][702815.411736] RDX: ffffa07b7f69b780 RSI: 0000000000510101 RDI: ffffa08b758ff480 [Fri Jan 25 21:27:33 2019][702815.418954] RBP: ffffa08b5d6c3798 R08: ffffa06b7f61b780 R09: 0000000000000000 [Fri Jan 25 21:27:33 2019][702815.426175] R10: ffffa06b7f61f140 R11: fffff7f77d5eb200 R12: ffffa08b5d6c3738 [Fri Jan 25 21:27:33 2019][702815.433393] R13: ffffa05bebd28510 R14: 0000000000000000 R15: 1fffffffffffffff [Fri Jan 25 21:27:33 2019][702815.440614] FS: 00007f6b107d0700(0000) GS:ffffa06b7f600000(0000) knlGS:0000000000000000 [Fri Jan 25 21:27:33 2019][702815.448785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 21:27:33 2019][702815.454618] CR2: 00007fe52ba6b080 CR3: 000000407fe34000 CR4: 00000000003407e0 [Fri Jan 25 21:27:33 2019][702815.461839] Call Trace: [Fri Jan 25 21:27:33 2019][702815.464379] [] queued_spin_lock_slowpath+0xb/0xf [Fri Jan 25 21:27:33 2019][702815.470735] [] _raw_spin_lock+0x20/0x30 [Fri Jan 25 21:27:33 2019][702815.476317] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Fri Jan 25 21:27:33 2019][702815.483108] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Fri Jan 25 21:27:33 2019][702815.490422] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Fri Jan 25 21:27:33 2019][702815.497211] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Fri Jan 25 21:27:33 2019][702815.504170] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:33 2019][702815.510619] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Fri Jan 25 21:27:33 2019][702815.517061] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Fri Jan 25 21:27:33 2019][702815.523942] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Fri Jan 25 21:27:33 2019][702815.531695] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Fri Jan 25 21:27:33 2019][702815.539787] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Fri Jan 25 21:27:33 2019][702815.546927] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Fri Jan 25 21:27:33 2019][702815.554602] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Fri Jan 25 21:27:33 2019][702815.561172] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Fri Jan 25 21:27:33 2019][702815.567473] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Fri Jan 25 21:27:33 2019][702815.574116] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Jan 25 21:27:33 2019][702815.581111] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Jan 25 21:27:33 2019][702815.587639] [] ? update_curr+0x14c/0x1e0 [Fri Jan 25 21:27:33 2019][702815.593296] [] ? account_entity_dequeue+0xae/0xd0 [Fri Jan 25 21:27:33 2019][702815.599767] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Fri Jan 25 21:27:33 2019][702815.607116] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Jan 25 21:27:33 2019][702815.614116] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Jan 25 21:27:33 2019][702815.621775] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Jan 25 21:27:33 2019][702815.628944] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Jan 25 21:27:33 2019][702815.636716] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Jan 25 21:27:33 2019][702815.643592] [] ? default_wake_function+0x12/0x20 [Fri Jan 25 21:27:33 2019][702815.649943] [] ? __wake_up_common+0x5b/0x90 [Fri Jan 25 21:27:33 2019][702815.655897] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Jan 25 21:27:33 2019][702815.662283] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Jan 25 21:27:33 2019][702815.669764] [] kthread+0xd1/0xe0 [Fri Jan 25 21:27:33 2019][702815.674730] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:33 2019][702815.680910] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Jan 25 21:27:33 2019][702815.687434] [] ? insert_kthread_work+0x40/0x40 [Fri Jan 25 21:27:33 2019][702815.693614] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Fri Jan 25 21:27:33 2019][702815.714388] LustreError: 126421:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffffa07957a06000 [Fri Jan 25 21:27:33 2019][702815.714448] LustreError: 126669:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffffa05e74c77850 x1623670115008192/t0(0) o4->9daa8e10-c154-ca96-d85d-3f6355b9ec3f@10.8.2.5@o2ib6:508/0 lens 488/448 e 0 to 0 dl 1548480593 ref 1 fl Interpret:/2/0 rc 0/0 [Fri Jan 25 21:27:33 2019][702815.714454] LustreError: 126423:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffffa08b13b7b400 [Fri Jan 25 21:27:34 2019][702815.714466] LustreError: 126425:0:(events.c:305:request_in_callback()) event type 2, status -103, service mdt_io [Fri Jan 25 21:27:34 2019][702815.714475] LustreError: 130240:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE req@ffffa05b447e7c50 x1623670115008432/t0(0) o4->9daa8e10-c154-ca96-d85d-3f6355b9ec3f@10.8.2.5@o2ib6:548/0 lens 488/448 e 0 to 0 dl 1548480633 ref 1 fl Interpret:/2/0 rc 0/0 [Fri Jan 25 21:27:34 2019][702815.714477] Lustre: fir-MDT0001: Bulk IO write error with 9daa8e10-c154-ca96-d85d-3f6355b9ec3f (at 10.8.2.5@o2ib6), client will retry: rc = -110 [Fri Jan 25 21:27:34 2019][702815.714484] LustreError: 129285:0:(pack_generic.c:590:__lustre_unpack_msg()) message length 0 too small for magic/version check [Fri Jan 25 21:27:34 2019][702815.714489] LustreError: 129285:0:(sec.c:2068:sptlrpc_svc_unwrap_request()) error unpacking request from 12345-10.8.6.20@o2ib6 x1623669142372256 [Fri Jan 25 21:27:34 2019][702815.714491] Lustre: 126669:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:32s); client may timeout. req@ffffa05e74c77850 x1623670115008192/t0(0) o4->9daa8e10-c154-ca96-d85d-3f6355b9ec3f@10.8.2.5@o2ib6:508/0 lens 488/448 e 0 to 0 dl 1548480593 ref 1 fl Complete:/2/ffffffff rc -110/-1 [Fri Jan 25 21:27:34 2019][702815.714505] Lustre: fir-MDT0001: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Fri Jan 25 21:27:34 2019][702815.714506] Lustre: Skipped 3 previous similar messages [Fri Jan 25 21:27:34 2019][702815.714509] LustreError: 129285:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.6.20@o2ib6: deadline 6:32s ago [Fri Jan 25 21:27:34 2019][702815.714509] req@ffffa06382225450 x1623669142371680/t0(0) o4->f9afcafe-b7fa-076a-0d87-fc0f78fd02d0@10.8.6.20@o2ib6:508/0 lens 488/0 e 0 to 0 dl 1548480593 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [Fri Jan 25 21:27:34 2019][702815.714818] LNetError: 126427:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) [Fri Jan 25 21:27:34 2019][702815.714821] LNetError: 126427:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 7 previous similar messages [Fri Jan 25 21:27:34 2019][702815.717765] LustreError: 126670:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffffa05e74f61050 x1623661016630960/t0(0) o4->cc9aeebd-5eb1-fd0a-28e5-4a7c51fd2782@10.8.6.33@o2ib6:545/0 lens 488/448 e 0 to 0 dl 1548480630 ref 1 fl Interpret:/2/0 rc 0/0 [Fri Jan 25 21:27:34 2019][702815.717768] LustreError: 126670:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 2 previous similar messages [Fri Jan 25 21:27:41 2019][702823.504379] Lustre: fir-MDT0003: Connection restored to 10.0.10.107@o2ib7 (at 10.0.10.107@o2ib7) [Fri Jan 25 21:27:41 2019][702823.513284] Lustre: Skipped 36 previous similar messages [Fri Jan 25 21:27:49 2019][702831.721028] Lustre: fir-MDT0001: Client fir-MDT0001-lwp-OST000a_UUID (at 10.0.10.101@o2ib7) reconnecting [Fri Jan 25 21:27:49 2019][702831.730594] Lustre: Skipped 38 previous similar messages [Fri Jan 25 21:27:57 2019][702838.988144] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Jan 25 21:27:57 2019][702839.004478] LustreError: Skipped 1 previous similar message [Fri Jan 25 21:28:05 2019][702847.640517] Lustre: fir-MDT0001: Received new LWP connection from 10.0.10.51@o2ib7, removing former export from same NID [Fri Jan 25 21:28:05 2019][702847.651479] Lustre: Skipped 3 previous similar messages [Fri Jan 25 21:28:15 2019][702856.825496] Lustre: fir-MDT0001: Connection restored to 10.0.10.101@o2ib7 (at 10.0.10.101@o2ib7) [Fri Jan 25 21:28:15 2019][702856.834377] Lustre: Skipped 66 previous similar messages [Fri Jan 25 21:28:24 2019][702865.965889] Lustre: fir-MDT0003: Client 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6) reconnecting [Fri Jan 25 21:28:24 2019][702865.975974] Lustre: Skipped 47 previous similar messages [Fri Jan 25 21:29:24 2019][702926.159432] Lustre: fir-MDT0003: Connection restored to 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) [Fri Jan 25 21:29:24 2019][702926.169698] Lustre: Skipped 66 previous similar messages [Fri Jan 25 21:29:29 2019][702931.155922] Lustre: 129267:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 21:29:29 2019][702931.167754] Lustre: 129267:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Fri Jan 25 21:29:58 2019][702959.914894] Lustre: fir-MDT0001: Client 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) reconnecting [Fri Jan 25 21:29:58 2019][702959.924898] Lustre: Skipped 1 previous similar message [Fri Jan 25 21:35:03 2019][703264.996434] Lustre: 127063:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 21:52:14 2019][704295.930397] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 21:52:14 2019][704295.942225] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Fri Jan 25 22:32:41 2019][706723.002705] Lustre: 129280:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Fri Jan 25 22:32:41 2019][706723.014532] Lustre: 129280:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Fri Jan 25 22:36:24 2019][706945.802777] Lustre: 127209:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 22:45:59 2019][707521.552716] Lustre: 126623:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 22:58:38 2019][708280.583343] Lustre: 129794:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Fri Jan 25 22:58:38 2019][708280.595166] Lustre: 129794:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Fri Jan 25 23:38:06 2019][710648.432870] Lustre: 128085:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 23:38:06 2019][710648.444695] Lustre: 128085:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Fri Jan 25 23:42:50 2019][710932.251641] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Fri Jan 25 23:42:50 2019][710932.263472] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 00:39:45 2019][714347.751174] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 00:39:45 2019][714347.763001] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 00:47:41 2019][714823.157198] Lustre: 129251:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 00:50:52 2019][715014.757259] LNetError: 126422:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Sat Jan 26 00:50:52 2019][715014.769958] LNetError: 126422:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 13 previous similar messages [Sat Jan 26 00:50:55 2019][715017.704124] LNetError: 126426:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Sat Jan 26 00:50:55 2019][715017.716810] LNetError: 126426:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message [Sat Jan 26 00:50:58 2019][715019.939637] LNetError: 126432:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Sat Jan 26 00:50:58 2019][715019.952330] LNetError: 126432:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message [Sat Jan 26 00:51:08 2019][715030.467187] Lustre: fir-MDT0003: Client f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6) reconnecting [Sat Jan 26 00:51:08 2019][715030.477380] Lustre: fir-MDT0003: Connection restored to f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6) [Sat Jan 26 00:51:08 2019][715030.487764] Lustre: Skipped 1 previous similar message [Sat Jan 26 00:51:08 2019][715030.725401] LustreError: 127363:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffffa05be9fa6850 x1623670968621168/t0(0) o4->f9afcafe-b7fa-076a-0d87-fc0f78fd02d0@10.8.6.20@o2ib6:718/0 lens 488/448 e 0 to 0 dl 1548492883 ref 1 fl Interpret:/0/0 rc 0/0 [Sat Jan 26 00:51:08 2019][715030.749575] LustreError: 127363:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 11 previous similar messages [Sat Jan 26 00:51:08 2019][715030.759311] Lustre: fir-MDT0001: Bulk IO write error with f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6), client will retry: rc = -110 [Sat Jan 26 00:51:08 2019][715030.772482] Lustre: Skipped 13 previous similar messages [Sat Jan 26 00:51:10 2019][715032.498006] LNetError: 126430:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Sat Jan 26 00:51:10 2019][715032.510754] LNetError: 126430:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message [Sat Jan 26 00:51:36 2019][715058.639705] Lustre: fir-MDT0003: Client 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6) reconnecting [Sat Jan 26 00:51:36 2019][715058.649797] Lustre: Skipped 1 previous similar message [Sat Jan 26 00:51:36 2019][715058.655082] Lustre: fir-MDT0003: Connection restored to 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6) [Sat Jan 26 00:52:38 2019][715120.097627] LNetError: 126426:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Sat Jan 26 00:52:38 2019][715120.110382] LNetError: 126426:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message [Sat Jan 26 00:52:53 2019][715135.658134] LNetError: 126424:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Sat Jan 26 00:52:53 2019][715135.670828] LNetError: 126424:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages [Sat Jan 26 00:53:07 2019][715149.698973] Lustre: fir-MDT0003: Client 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6) reconnecting [Sat Jan 26 00:53:07 2019][715149.709062] Lustre: Skipped 4 previous similar messages [Sat Jan 26 00:53:07 2019][715149.714426] Lustre: fir-MDT0003: Connection restored to 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6) [Sat Jan 26 00:53:07 2019][715149.724772] Lustre: Skipped 4 previous similar messages [Sat Jan 26 00:53:12 2019][715154.830799] LNetError: 126428:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Sat Jan 26 01:06:38 2019][715960.818777] Lustre: 129259:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 01:06:38 2019][715960.830605] Lustre: 129259:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 01:17:09 2019][716591.194066] Lustre: 129280:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 01:17:09 2019][716591.205896] Lustre: 129280:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 01:48:16 2019][718457.937869] Lustre: 126477:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1548496260/real 1548496260] req@ffffa08a2d63a100 x1623664396752224/t0(0) o41->fir-MDT0002-osp-MDT0003@10.0.10.51@o2ib7:24/4 lens 224/368 e 0 to 1 dl 1548496267 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Sat Jan 26 01:48:16 2019][718457.966158] Lustre: 126477:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [Sat Jan 26 01:48:16 2019][718457.976084] Lustre: fir-MDT0002-osp-MDT0003: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Sat Jan 26 01:48:18 2019][718460.373054] Lustre: fir-MDT0003: Received new LWP connection from 10.0.10.51@o2ib7, removing former export from same NID [Sat Jan 26 01:48:18 2019][718460.384036] Lustre: fir-MDT0003: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [Sat Jan 26 01:48:18 2019][718460.392739] Lustre: Skipped 4 previous similar messages [Sat Jan 26 01:48:24 2019][718466.012935] Lustre: 126630:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1548496268/real 1548496268] req@ffffa06f553bf200 x1623664396754768/t0(0) o101->fir-MDT0000-osp-MDT0001@10.0.10.51@o2ib7:24/4 lens 328/344 e 0 to 1 dl 1548496275 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Sat Jan 26 01:48:24 2019][718466.041309] Lustre: 126630:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [Sat Jan 26 01:48:28 2019][718470.797199] Lustre: fir-MDT0001: Client f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6) reconnecting [Sat Jan 26 01:48:28 2019][718470.807288] Lustre: Skipped 4 previous similar messages [Sat Jan 26 01:48:28 2019][718470.812633] Lustre: fir-MDT0001: Connection restored to f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6) [Sat Jan 26 01:48:28 2019][718470.823020] Lustre: Skipped 2 previous similar messages [Sat Jan 26 01:48:29 2019][718470.946979] LustreError: 3361:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffffa05d706b2450 x1623671563132112/t0(0) o4->f9afcafe-b7fa-076a-0d87-fc0f78fd02d0@10.8.6.20@o2ib6:365/0 lens 488/448 e 1 to 0 dl 1548496305 ref 1 fl Interpret:/0/0 rc 0/0 [Sat Jan 26 01:48:29 2019][718470.967997] Lustre: fir-MDT0001: Bulk IO write error with f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6), client will retry: rc = -110 [Sat Jan 26 01:48:29 2019][718470.984053] LustreError: 3361:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 1 previous similar message [Sat Jan 26 01:48:30 2019][718472.149980] LustreError: 127263:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffffa05b447e0850 x1623623379261504/t0(0) o4->85c45b71-10aa-be9f-071b-9a9c50548db2@10.8.15.7@o2ib6:366/0 lens 488/448 e 1 to 0 dl 1548496306 ref 1 fl Interpret:/0/0 rc 0/0 [Sat Jan 26 01:48:30 2019][718472.159000] Lustre: fir-MDT0001: Bulk IO write error with 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6), client will retry: rc = -110 [Sat Jan 26 01:48:30 2019][718472.159001] Lustre: Skipped 1 previous similar message [Sat Jan 26 01:48:30 2019][718472.192444] LustreError: 127263:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 2 previous similar messages [Sat Jan 26 01:48:34 2019][718476.828020] LustreError: 129241:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffffa07b382da450 x1623671563132096/t0(0) o4->f9afcafe-b7fa-076a-0d87-fc0f78fd02d0@10.8.6.20@o2ib6:371/0 lens 488/448 e 1 to 0 dl 1548496311 ref 1 fl Interpret:/2/0 rc 0/0 [Sat Jan 26 01:48:34 2019][718476.852165] Lustre: fir-MDT0001: Bulk IO write error with f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6), client will retry: rc = -110 [Sat Jan 26 01:48:34 2019][718476.865281] Lustre: Skipped 2 previous similar messages [Sat Jan 26 01:48:35 2019][718477.027714] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.8.2.5@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Sat Jan 26 01:48:37 2019][718478.946056] Lustre: fir-MDT0001: Client 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6) reconnecting [Sat Jan 26 01:48:37 2019][718478.956146] Lustre: Skipped 4 previous similar messages [Sat Jan 26 01:48:37 2019][718479.425042] LustreError: 127227:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffffa05d988d1450 x1623568791052848/t0(0) o4->9b3a0481-8c2f-a606-18ad-7a06b856368e@10.8.3.11@o2ib6:386/0 lens 488/448 e 0 to 0 dl 1548496326 ref 1 fl Interpret:/0/0 rc 0/0 [Sat Jan 26 01:48:37 2019][718479.425061] Lustre: fir-MDT0001: Bulk IO write error with 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6), client will retry: rc = -110 [Sat Jan 26 01:48:37 2019][718479.425063] Lustre: Skipped 2 previous similar messages [Sat Jan 26 01:48:37 2019][718479.467594] LustreError: 127227:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 3 previous similar messages [Sat Jan 26 01:48:42 2019][718484.170311] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [Sat Jan 26 01:48:42 2019][718484.186641] LustreError: Skipped 3 previous similar messages [Sat Jan 26 01:48:42 2019][718484.214079] LustreError: 127693:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffffa07b382dbc50 x1623672577038288/t0(0) o4->9daa8e10-c154-ca96-d85d-3f6355b9ec3f@10.8.2.5@o2ib6:387/0 lens 488/448 e 0 to 0 dl 1548496327 ref 1 fl Interpret:/0/0 rc 0/0 [Sat Jan 26 01:48:42 2019][718484.238139] Lustre: fir-MDT0001: Bulk IO write error with 9daa8e10-c154-ca96-d85d-3f6355b9ec3f (at 10.8.2.5@o2ib6), client will retry: rc = -110 [Sat Jan 26 01:48:42 2019][718484.251175] Lustre: Skipped 3 previous similar messages [Sat Jan 26 01:48:43 2019][718485.598324] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.107@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Sat Jan 26 01:48:43 2019][718485.615778] LustreError: Skipped 4 previous similar messages [Sat Jan 26 01:48:51 2019][718493.301303] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.8.6.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Sat Jan 26 01:48:51 2019][718493.318585] LustreError: Skipped 7 previous similar messages [Sat Jan 26 01:48:56 2019][718498.598860] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.0.10.106@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Sat Jan 26 01:48:56 2019][718498.616317] LustreError: Skipped 11 previous similar messages [Sat Jan 26 01:49:06 2019][718508.391020] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.8.15.7@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Sat Jan 26 01:49:06 2019][718508.408304] LustreError: Skipped 1 previous similar message [Sat Jan 26 01:49:07 2019][718509.258534] Lustre: fir-MDT0002-osp-MDT0003: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [Sat Jan 26 01:49:07 2019][718509.268282] Lustre: Skipped 6 previous similar messages [Sat Jan 26 01:49:30 2019][718532.734110] Lustre: fir-MDT0001: Received new LWP connection from 10.0.10.51@o2ib7, removing former export from same NID [Sat Jan 26 01:49:30 2019][718532.745062] Lustre: Skipped 6 previous similar messages [Sat Jan 26 01:51:05 2019][718627.795087] Lustre: 129280:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 01:51:05 2019][718627.806917] Lustre: 129280:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 01:53:22 2019][718764.368475] Lustre: 128085:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 01:55:30 2019][718892.437323] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 01:59:03 2019][719105.374148] Lustre: 129267:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 01:59:03 2019][719105.385985] Lustre: 129267:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 02:05:06 2019][719467.984109] Lustre: 129794:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 02:23:24 2019][720566.857409] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 02:23:24 2019][720566.869250] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 02:37:44 2019][721426.916546] Lustre: 126623:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 02:48:48 2019][722090.771288] Lustre: 128085:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 02:48:48 2019][722090.783116] Lustre: 128085:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 02:51:39 2019][722261.193946] Lustre: 128092:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 02:51:39 2019][722261.205797] Lustre: 128092:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 02:59:28 2019][722730.548585] Lustre: 129262:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 02:59:28 2019][722730.560408] Lustre: 129262:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 03:23:45 2019][724187.360814] Lustre: 126625:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 03:33:19 2019][724761.439179] Lustre: 126623:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 03:37:19 2019][725001.896789] Lustre: 127063:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 03:39:39 2019][725141.257932] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 03:39:39 2019][725141.269761] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 04:02:38 2019][726520.966220] Lustre: 127247:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 04:04:55 2019][726657.221784] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 04:58:36 2019][729878.675532] Lustre: 128092:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 04:58:36 2019][729878.687358] Lustre: 128092:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 05:31:47 2019][731869.896417] Lustre: 127134:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 05:35:25 2019][732087.992233] Lustre: 126624:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 05:35:25 2019][732088.004057] Lustre: 126624:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 05:42:08 2019][732490.549454] Lustre: 127214:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 05:42:08 2019][732490.561278] Lustre: 127214:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 05:47:30 2019][732812.933367] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 06:22:34 2019][734916.665368] Lustre: 128085:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 06:22:34 2019][734916.677190] Lustre: 128085:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 06:24:56 2019][735058.961526] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 06:28:58 2019][735301.069627] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 06:29:30 2019][735332.301904] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 06:30:39 2019][735401.442253] Lustre: 129267:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 06:31:00 2019][735422.894768] Lustre: 129267:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 06:31:00 2019][735422.906590] Lustre: 129267:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 06:38:20 2019][735862.840759] Lustre: 126624:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 06:41:04 2019][736026.270217] Lustre: 129259:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 06:41:04 2019][736026.282044] Lustre: 129259:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 06:52:18 2019][736700.335810] Lustre: 129259:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 06:56:48 2019][736970.829075] Lustre: 126623:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 06:58:49 2019][737091.278932] Lustre: 126624:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 06:58:49 2019][737091.290757] Lustre: 126624:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 07:00:30 2019][737192.165261] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 07:00:30 2019][737192.177104] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 07:02:14 2019][737296.220282] Lustre: 127136:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 07:02:14 2019][737296.232103] Lustre: 127136:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 07:08:45 2019][737688.069714] Lustre: 127209:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 07:08:45 2019][737688.081542] Lustre: 127209:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 07:18:01 2019][738244.056269] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 07:18:01 2019][738244.068098] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 07:46:51 2019][739973.537360] Lustre: 129794:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 07:46:51 2019][739973.549184] Lustre: 129794:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 07:52:45 2019][740327.226487] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 08:08:31 2019][741274.037761] Lustre: 127247:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 08:08:31 2019][741274.049589] Lustre: 127247:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 08:13:22 2019][741564.905814] Lustre: 129251:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 08:20:46 2019][742008.553089] Lustre: 129257:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 08:20:46 2019][742008.564925] Lustre: 129257:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 08:35:17 2019][742880.072901] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 09:09:31 2019][744933.826236] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 09:51:53 2019][747476.131992] Lustre: 129280:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 10:07:54 2019][748436.325771] Lustre: 129259:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 10:37:57 2019][750239.734901] Lustre: 129257:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 10:39:35 2019][750337.693699] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 10:52:49 2019][751131.720029] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 10:52:52 2019][751134.455809] Lustre: 127136:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 10:53:23 2019][751165.344661] Lustre: 129251:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 10:53:23 2019][751165.356483] Lustre: 129251:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 10:54:59 2019][751262.061805] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 10:55:40 2019][751302.495477] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 10:55:40 2019][751302.507302] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 11:02:49 2019][751731.633421] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 11:02:49 2019][751731.645240] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 11:03:16 2019][751758.900678] Lustre: 127063:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 11:03:16 2019][751758.912495] Lustre: 127063:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 11:05:31 2019][751894.297786] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 11:05:31 2019][751894.309611] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 11:10:09 2019][752171.984701] Lustre: 127063:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 11:10:09 2019][752171.996528] Lustre: 127063:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 11:17:17 2019][752600.161186] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Sat Jan 26 11:17:17 2019][752600.171562] Lustre: Skipped 7 previous similar messages [Sat Jan 26 11:30:32 2019][753395.039548] Lustre: 127134:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:04:12 2019][755414.570677] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:04:12 2019][755414.582495] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 12:32:41 2019][757124.304560] Lustre: 129262:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:34:16 2019][757218.861180] Lustre: 2937:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:34:16 2019][757218.872829] Lustre: 2937:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 12:39:39 2019][757541.481871] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:45:52 2019][757914.468288] Lustre: 128085:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:45:52 2019][757914.480112] Lustre: 128085:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 13:04:44 2019][759046.475253] Lustre: 127209:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 13:04:44 2019][759046.487075] Lustre: 127209:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 13:09:20 2019][759323.095613] Lustre: 129257:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 13:09:20 2019][759323.107435] Lustre: 129257:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 13:11:09 2019][759431.589943] Lustre: 127136:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 13:11:09 2019][759431.601769] Lustre: 127136:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 13:14:24 2019][759626.965718] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 13:14:24 2019][759626.977542] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 13:19:47 2019][759949.610445] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 13:29:55 2019][760558.199390] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 13:29:55 2019][760558.211211] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 13:33:52 2019][760794.698236] Lustre: 129280:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 14:42:30 2019][764913.380420] Lustre: 127063:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 14:51:56 2019][765478.734415] Lustre: 128092:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 14:51:56 2019][765478.746236] Lustre: 128092:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 14:52:39 2019][765522.489581] Lustre: 2976:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 14:58:39 2019][765882.445168] Lustre: 129280:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 14:58:39 2019][765882.456990] Lustre: 129280:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 15:02:01 2019][766084.219982] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 15:02:01 2019][766084.231804] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 15:18:40 2019][767082.769798] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 15:20:30 2019][767192.592745] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 15:20:30 2019][767192.604569] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 15:22:44 2019][767326.876573] Lustre: 129794:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 15:22:44 2019][767326.888397] Lustre: 129794:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 15:28:03 2019][767646.496038] Lustre: 2937:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 15:28:03 2019][767646.507688] Lustre: 2937:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 15:29:45 2019][767747.867079] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 15:33:26 2019][767969.004855] Lustre: 128092:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 15:55:07 2019][769270.511136] Lustre: 129257:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 15:55:07 2019][769270.522962] Lustre: 129257:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 16:01:23 2019][769646.064045] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:01:23 2019][769646.075870] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 16:08:23 2019][770066.254879] Lustre: 129794:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:08:23 2019][770066.266697] Lustre: 129794:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 16:12:09 2019][770291.847046] Lustre: 129280:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:16:39 2019][770561.606290] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 17:29:43 2019][774946.151809] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 17:29:43 2019][774946.163634] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Sat Jan 26 17:31:14 2019][775037.500272] Lustre: 126625:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 17:33:35 2019][775177.968663] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Sat Jan 26 17:33:35 2019][775177.980490] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 20:25:42 2019][785504.860566] Lustre: 129251:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sat Jan 26 20:25:42 2019][785504.872397] Lustre: 129251:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Sun Jan 27 00:31:08 2019][800231.690553] Lustre: 2976:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 00:31:08 2019][800231.702207] Lustre: 2976:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 00:32:19 2019][800302.493364] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 00:42:18 2019][800901.619809] Lustre: 128085:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 00:59:20 2019][801923.589768] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds [Sun Jan 27 00:59:20 2019][801923.600114] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 2 previous similar messages [Sun Jan 27 00:59:20 2019][801923.610455] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.201@o2ib7 (7): c: 7, oc: 0, rc: 8 [Sun Jan 27 00:59:20 2019][801923.622612] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 2 previous similar messages [Sun Jan 27 00:59:20 2019][801923.769457] Lustre: fir-MDT0003: Client f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6) reconnecting [Sun Jan 27 00:59:20 2019][801923.779549] Lustre: Skipped 1 previous similar message [Sun Jan 27 00:59:20 2019][801923.784802] Lustre: 129264:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1548579725/real 0] req@ffffa06c532abc00 x1623665373486400/t0(0) o104->fir-MDT0003@10.8.3.11@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1548579732 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Sun Jan 27 00:59:20 2019][801923.811359] Lustre: 129264:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message [Sun Jan 27 00:59:20 2019][801923.821213] Lustre: fir-MDT0003: Connection restored to f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6) [Sun Jan 27 00:59:20 2019][801923.831556] Lustre: Skipped 1 previous similar message [Sun Jan 27 00:59:21 2019][801924.529771] Lustre: fir-MDT0003: Connection restored to 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6) [Sun Jan 27 00:59:21 2019][801924.540116] Lustre: Skipped 1 previous similar message [Sun Jan 27 00:59:22 2019][801925.589788] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Sun Jan 27 00:59:22 2019][801925.600130] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 1 previous similar message [Sun Jan 27 00:59:22 2019][801925.610386] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.107@o2ib7 (10): c: 0, oc: 0, rc: 8 [Sun Jan 27 00:59:22 2019][801925.622635] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 1 previous similar message [Sun Jan 27 00:59:23 2019][801926.082798] Lustre: 126481:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1548579728/real 0] req@ffffa08a5dfa1b00 x1623665373487600/t0(0) o13->fir-OST0026-osc-MDT0003@10.0.10.107@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1548579735 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Sun Jan 27 00:59:23 2019][801926.082802] Lustre: fir-OST0015-osc-MDT0001: Connection to fir-OST0015 (at 10.0.10.104@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Sun Jan 27 00:59:23 2019][801926.082804] Lustre: Skipped 3 previous similar messages [Sun Jan 27 00:59:23 2019][801926.131760] Lustre: 126481:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [Sun Jan 27 00:59:23 2019][801926.594809] Lustre: fir-OST000f-osc-MDT0001: Connection to fir-OST000f (at 10.0.10.104@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Sun Jan 27 00:59:23 2019][801926.610969] Lustre: Skipped 7 previous similar messages [Sun Jan 27 00:59:24 2019][801927.746810] Lustre: fir-OST0017-osc-MDT0003: Connection to fir-OST0017 (at 10.0.10.104@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Sun Jan 27 00:59:24 2019][801927.762962] Lustre: Skipped 7 previous similar messages [Sun Jan 27 00:59:25 2019][801928.434045] Lustre: mdt: This server is not able to keep up with reques4048] Lustre: 127710:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=3 reqQ=0 recA=10, svcEst=31, delay=10729 [Sun Jan 27 00:59:25 2019][801928.434060] LustreError: 127350:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffffa07b416a7050 x1623575400153952/t0(0) o4->9b3a0481-8c2f-a606-18ad-7a06b856368e@10.8.3.11@o2ib6:16/0 lens 488/448 e 1 to 0 dl 1548579761 ref 1 fl Interpret:/0/0 rc 0/0 [Sun Jan 27 00:59:25 2019][801928.434063] Lustre: 126467:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1548579728/real 0] req@ffffa06c5d7bd400 x1623665373487728/t0(0) o13->fir-OST0024-osc-MDT0001@10.0.10.107@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1548579735 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Sun Jan 27 00:59:25 2019][801928.434067] Lustre: 127710:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-6s), not sending early reply. Consider increasing at_early_margin (5)? req@ffffa05be9fa4050 x1623678107587488/t0(0) o4->f9afcafe-b7fa-076a-0d87-fc0f78fd02d0@10.8.6.20@o2ib6:741/0 lens 488/448 e 0 to 0 dl 1548579731 ref 2 fl Interpret:/0/0 rc 0/0 [Sun Jan 27 00:59:25 2019][801928.434070] Lustre: 126467:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [Sun Jan 27 00:59:25 2019][801928.434073] Lustre: 127710:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 11 previous similar messages [Sun Jan 27 00:59:25 2019][801928.434088] Lustre: fir-MDT0003: Bulk IO write error with 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6), client will retry: rc = -110 [Sun Jan 27 00:59:25 2019][801928.434099] LustreError: 127266:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffffa052e5f14450 x1623678107587536/t0(0) o4->f9afcafe-b7fa-076a-0d87-fc0f78fd02d0@10.8.6.20@o2ib6:11/0 lens 488/448 e 1 to 0 dl 1548579756 ref 1 fl Interpret:/0/0 rc 0/0 [Sun Jan 27 00:59:25 2019][801928.434179] LNetError: 126426:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) [Sun Jan 27 00:59:25 2019][801928.434219] Lustre: fir-MDT0003: Client fir-MDT0003-lwp-OST0015_UUID (at 10.0.10.104@o2ib7) reconnecting [Sun Jan 27 00:59:25 2019][801928.434221] Lustre: Skipped 2 previous similar messages [Sun Jan 27 00:59:25 2019][801928.434246] Lustre: fir-MDT0003: Connection restored to 10.0.10.104@o2ib7 (at 10.0.10.104@o2ib7) [Sun Jan 27 00:59:25 2019][801928.438095] Lustre: 126669:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:6s); client may timeout. req@ffffa05be9fa4050 x1623678107587488/t0(0) o4->f9afcafe-b7fa-076a-0d87-fc0f78fd02d0@10.8.6.20@o2ib6:741/0 lens 488/448 e 0 to 0 dl 1548579731 ref 1 fl Complete:/0/ffffffff rc -107/-1 [Sun Jan 27 00:59:25 2019][801928.438169] Lustre: 126669:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 8 previous similar messages [Sun Jan 27 00:59:25 2019][801928.669016] Lustre: Skipped 1 previous similar message [Sun Jan 27 00:59:28 2019][801931.696673] Lustre: fir-MDT0003: Connection restored to 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6) [Sun Jan 27 00:59:28 2019][801931.707019] Lustre: Skipped 36 previous similar messages [Sun Jan 27 00:59:29 2019][801932.438856] LustreError: 129240:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffffa06383c4c450 x1623575400163120/t0(0) o4->9b3a0481-8c2f-a606-18ad-7a06b856368e@10.8.3.11@o2ib6:35/0 lens 488/448 e 0 to 0 dl 1548579780 ref 1 fl Interpret:/0/0 rc 0/0 [Sun Jan 27 00:59:29 2019][801932.462902] LustreError: 129240:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 1 previous similar message [Sun Jan 27 00:59:29 2019][801932.472400] Lustre: fir-MDT0003: Bulk IO write error with 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6), client will retry: rc = -110 [Sun Jan 27 00:59:29 2019][801932.485518] Lustre: Skipped 4 previous similar messages [Sun Jan 27 00:59:29 2019][801932.874023] Lustre: fir-MDT0003: Client f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6) reconnecting [Sun Jan 27 00:59:29 2019][801932.884109] Lustre: Skipped 12 previous similar messages [Sun Jan 27 00:59:36 2019][801939.501880] Lustre: fir-MDT0003: Connection restored to 9daa8e10-c154-ca96-d85d-3f6355b9ec3f (at 10.8.2.5@o2ib6) [Sun Jan 27 00:59:36 2019][801939.512141] Lustre: Skipped 1 previous similar message [Sun Jan 27 00:59:37 2019][801940.448912] LustreError: 129302:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffffa05b5d845c50 x1623679192349632/t0(0) o4->9daa8e10-c154-ca96-d85d-3f6355b9ec3f@10.8.2.5@o2ib6:35/0 lens 488/448 e 0 to 0 dl 1548579780 ref 1 fl Interpret:/0/0 rc 0/0 [Sun Jan 27 00:59:37 2019][801940.453936] Lustre: fir-MDT0003: Bulk IO write error with 9daa8e10-c154-ca96-d85d-3f6355b9ec3f (at 10.8.2.5@o2ib6), client will retry: rc = -110 [Sun Jan 27 00:59:37 2019][801940.453938] Lustre: Skipped 3 previous similar messages [Sun Jan 27 00:59:37 2019][801940.491220] LustreError: 129302:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 6 previous similar messages [Sun Jan 27 01:00:09 2019][801972.494838] Lustre: fir-MDT0003: Client 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) reconnecting [Sun Jan 27 01:00:09 2019][801972.504933] Lustre: Skipped 1 previous similar message [Sun Jan 27 01:00:09 2019][801972.510194] Lustre: fir-MDT0003: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Sun Jan 27 01:00:10 2019][801973.437181] LustreError: 130109:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffffa05be9fa4850 x1623629564446864/t0(0) o4->85c45b71-10aa-be9f-071b-9a9c50548db2@10.8.15.7@o2ib6:48/0 lens 488/448 e 1 to 0 dl 1548579793 ref 1 fl Interpret:/2/0 rc 0/0 [Sun Jan 27 01:00:10 2019][801973.454202] Lustre: fir-MDT0003: Bulk IO write error with 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6), client will retry: rc = -110 [Sun Jan 27 01:00:10 2019][801973.454204] Lustre: Skipped 3 previous similar messages [Sun Jan 27 01:00:10 2019][801973.479645] LustreError: 130109:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 2 previous similar messages [Sun Jan 27 01:04:39 2019][802242.682631] Lustre: 126624:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 01:07:50 2019][802433.357221] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 01:07:50 2019][802433.369068] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sun Jan 27 01:20:14 2019][803177.661292] Lustre: 127970:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 01:51:52 2019][805075.575579] perf: interrupt took too long (4108 > 3915), lowering kernel.perf_event_max_sample_rate to 48000 [Sun Jan 27 01:52:05 2019][805088.213690] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 01:52:05 2019][805088.225512] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 02:05:55 2019][805918.339985] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.8.2.5@o2ib6 ns: mdt-fir-MDT0003_UUID lock: ffffa05e2e15e0c0/0x8c1de53716c795d9 lrc: 3/0,0 mode: EX/EX res: [0x280000c7c:0xd1c2:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x60000400000020 nid: 10.8.2.5@o2ib6 remote: 0x63bf45ab0827ded5 expref: 105792 pid: 126626 timeout: 805911 lvb_type: 3 [Sun Jan 27 02:05:55 2019][805918.381581] Lustre: fir-MDT0003: Connection restored to 9daa8e10-c154-ca96-d85d-3f6355b9ec3f (at 10.8.2.5@o2ib6) [Sun Jan 27 02:05:55 2019][805918.391846] Lustre: Skipped 1 previous similar message [Sun Jan 27 02:05:55 2019][805918.760952] LustreError: 129278:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa06c5df7ec00 x1623665435521120/t0(0) o104->fir-MDT0003@10.8.2.5@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Sun Jan 27 02:06:01 2019][805924.061274] LustreError: 129256:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa0612bfa9e00 x1623665435524304/t0(0) o104->fir-MDT0003@10.8.2.5@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Sun Jan 27 02:06:34 2019][805957.058729] LustreError: 126608:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa08a323a1b00 x1623665435535824/t0(0) o105->fir-MDT0003@10.8.2.5@o2ib6:15/16 lens 304/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Sun Jan 27 02:06:39 2019][805962.516604] Lustre: 129278:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43:1s); client may timeout. req@ffffa0740de5c500 x1623679357375472/t8769185605(0) o36->9daa8e10-c154-ca96-d85d-3f6355b9ec3f@10.8.2.5@o2ib6:250/0 lens 528/424 e 0 to 0 dl 1548583770 ref 1 fl Complete:/0/0 rc 0/0 [Sun Jan 27 02:06:39 2019][805962.545592] Lustre: 129278:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2 previous similar messages [Sun Jan 27 02:06:43 2019][805966.910199] LustreError: 129251:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa05966b67500 x1623665435621584/t0(0) o104->fir-MDT0003@10.8.2.5@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Sun Jan 27 02:07:09 2019][805992.422416] LustreError: 1418:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 143360 [Sun Jan 27 02:07:10 2019][805993.586254] LustreError: 130108:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Sun Jan 27 02:07:15 2019][805998.763073] LustreError: 1418:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 53248 GRANT, real grant 0 [Sun Jan 27 02:07:15 2019][805998.776976] LustreError: 1418:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 86 previous similar messages [Sun Jan 27 02:07:20 2019][806003.312998] LustreError: 1416:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 02:07:20 2019][806003.326993] LustreError: 1416:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 240 previous similar messages [Sun Jan 27 02:07:25 2019][806008.640639] LustreError: 127231:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 02:07:25 2019][806008.654808] LustreError: 127231:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 144 previous similar messages [Sun Jan 27 02:07:35 2019][806018.454639] LustreError: 126671:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 02:07:35 2019][806018.468799] LustreError: 126671:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 412 previous similar messages [Sun Jan 27 02:07:53 2019][806036.090318] LustreError: 3361:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 02:07:53 2019][806036.104309] LustreError: 3361:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 742 previous similar messages [Sun Jan 27 02:08:25 2019][806069.004885] LustreError: 129293:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 02:08:25 2019][806069.019051] LustreError: 129293:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 1339 previous similar messages [Sun Jan 27 02:09:30 2019][806133.159221] LustreError: 127319:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 02:09:30 2019][806133.173390] LustreError: 127319:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 2986 previous similar messages [Sun Jan 27 02:11:41 2019][806264.662205] LustreError: 127699:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 02:11:41 2019][806264.676373] LustreError: 127699:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 7370 previous similar messages [Sun Jan 27 02:15:58 2019][806521.105208] LustreError: 127235:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 02:15:58 2019][806521.119374] LustreError: 127235:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 15224 previous similar messages [Sun Jan 27 02:24:30 2019][807033.250512] LustreError: 1418:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 02:24:30 2019][807033.264509] LustreError: 1418:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 29844 previous similar messages [Sun Jan 27 02:34:30 2019][807633.476649] LustreError: 130109:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 02:34:30 2019][807633.490817] LustreError: 130109:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 37897 previous similar messages [Sun Jan 27 02:36:47 2019][807770.498915] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 02:36:47 2019][807770.510743] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 02:44:17 2019][808221.053267] Lustre: 129267:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 02:44:31 2019][808234.406336] LustreError: 127319:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 02:44:31 2019][808234.420503] LustreError: 127319:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 29853 previous similar messages [Sun Jan 27 02:54:31 2019][808835.080355] LustreError: 127266:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 02:54:32 2019][808835.094530] LustreError: 127266:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 29184 previous similar messages [Sun Jan 27 03:04:33 2019][809437.050538] LustreError: 126668:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 03:04:33 2019][809437.064704] LustreError: 126668:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 31310 previous similar messages [Sun Jan 27 03:14:34 2019][810037.244940] LustreError: 129300:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 03:14:34 2019][810037.259108] LustreError: 129300:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 34617 previous similar messages [Sun Jan 27 03:24:36 2019][810639.686657] LustreError: 130112:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 03:24:36 2019][810639.700821] LustreError: 130112:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 32420 previous similar messages [Sun Jan 27 03:34:36 2019][811239.822213] LustreError: 127227:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 03:34:36 2019][811239.836375] LustreError: 127227:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 35870 previous similar messages [Sun Jan 27 03:44:36 2019][811839.939723] LustreError: 129283:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 03:44:36 2019][811839.953889] LustreError: 129283:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 29414 previous similar messages [Sun Jan 27 03:47:21 2019][812004.332238] Lustre: 127209:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 03:47:21 2019][812004.344069] Lustre: 127209:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 03:54:36 2019][812440.091284] LustreError: 127693:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 03:54:36 2019][812440.105451] LustreError: 127693:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 25235 previous similar messages [Sun Jan 27 04:04:36 2019][813040.095893] LustreError: 130108:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Sun Jan 27 04:04:36 2019][813040.109976] LustreError: 130108:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 32745 previous similar messages [Sun Jan 27 04:14:39 2019][813642.442692] LustreError: 127656:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 04:14:39 2019][813642.456859] LustreError: 127656:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 33493 previous similar messages [Sun Jan 27 04:14:46 2019][813649.193376] LNetError: 126423:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Sun Jan 27 04:14:46 2019][813649.205898] LNetError: 126423:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 7 previous similar messages [Sun Jan 27 04:24:40 2019][814243.542106] LustreError: 126669:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 04:24:40 2019][814243.556272] LustreError: 126669:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 33005 previous similar messages [Sun Jan 27 04:34:40 2019][814843.866014] LustreError: 127700:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 04:34:40 2019][814843.880177] LustreError: 127700:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 44643 previous similar messages [Sun Jan 27 04:44:41 2019][815444.694742] LustreError: 127699:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 04:44:41 2019][815444.708908] LustreError: 127699:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 119788 previous similar messages [Sun Jan 27 04:54:41 2019][816044.937331] LustreError: 127261:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 04:54:41 2019][816044.951497] LustreError: 127261:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 65323 previous similar messages [Sun Jan 27 05:04:45 2019][816648.977171] LustreError: 130112:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 05:04:45 2019][816648.991338] LustreError: 130112:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 57515 previous similar messages [Sun Jan 27 05:14:47 2019][817250.265102] LustreError: 127656:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 05:14:47 2019][817250.279260] LustreError: 127656:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 65600 previous similar messages [Sun Jan 27 05:24:48 2019][817851.541656] LustreError: 127225:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 05:24:48 2019][817851.555858] LustreError: 127225:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 62631 previous similar messages [Sun Jan 27 05:34:48 2019][818452.152514] LustreError: 129239:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 05:34:48 2019][818452.166676] LustreError: 129239:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 53490 previous similar messages [Sun Jan 27 05:44:49 2019][819053.165382] LustreError: 129241:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 05:44:49 2019][819053.179551] LustreError: 129241:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 53081 previous similar messages [Sun Jan 27 05:54:52 2019][819655.883528] LustreError: 127218:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 05:54:52 2019][819655.897685] LustreError: 127218:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 66014 previous similar messages [Sun Jan 27 06:04:55 2019][820258.974924] LustreError: 131047:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 06:04:55 2019][820258.989090] LustreError: 131047:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 33839 previous similar messages [Sun Jan 27 06:08:30 2019][820474.090517] Lustre: 129251:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 06:14:57 2019][820861.207018] LustreError: 130204:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 06:14:57 2019][820861.221182] LustreError: 130204:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 31790 previous similar messages [Sun Jan 27 06:24:59 2019][821462.626947] LustreError: 3361:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 06:24:59 2019][821462.640944] LustreError: 3361:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 34984 previous similar messages [Sun Jan 27 06:27:13 2019][821597.136166] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.0.10.3@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Sun Jan 27 06:35:01 2019][822064.660524] LustreError: 130112:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 06:35:01 2019][822064.674694] LustreError: 130112:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 31143 previous similar messages [Sun Jan 27 06:45:02 2019][822665.961245] LustreError: 127222:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 06:45:02 2019][822665.975413] LustreError: 127222:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 27224 previous similar messages [Sun Jan 27 06:55:05 2019][823268.643631] LustreError: 1012:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Sun Jan 27 06:55:05 2019][823268.657531] LustreError: 1012:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 34334 previous similar messages [Sun Jan 27 07:05:05 2019][823868.902874] LustreError: 127227:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 07:05:05 2019][823868.917038] LustreError: 127227:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 40580 previous similar messages [Sun Jan 27 07:08:54 2019][824097.511794] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 07:15:06 2019][824469.458162] LustreError: 127221:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 07:15:06 2019][824469.472334] LustreError: 127221:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 44711 previous similar messages [Sun Jan 27 07:25:08 2019][825071.481391] LustreError: 130108:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Sun Jan 27 07:25:08 2019][825071.495467] LustreError: 130108:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 52133 previous similar messages [Sun Jan 27 07:35:09 2019][825673.137549] LustreError: 129239:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Sun Jan 27 07:35:09 2019][825673.151627] LustreError: 129239:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 31230 previous similar messages [Sun Jan 27 07:36:29 2019][825753.104693] Lustre: 129267:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 07:36:29 2019][825753.116514] Lustre: 129267:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 07:45:11 2019][826274.663932] LustreError: 127221:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Sun Jan 27 07:45:11 2019][826274.678014] LustreError: 127221:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 27130 previous similar messages [Sun Jan 27 07:48:43 2019][826486.884838] Lustre: 128092:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 08:02:56 2019][827339.741938] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 08:02:56 2019][827339.753787] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 08:13:28 2019][827971.593364] Lustre: 127134:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 08:25:02 2019][828665.866510] LustreError: 129284:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Sun Jan 27 08:25:02 2019][828665.880675] LustreError: 129284:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 21596 previous similar messages [Sun Jan 27 09:16:40 2019][831763.509384] Lustre: 129259:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 09:48:50 2019][833694.134957] Lustre: 128092:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 10:33:13 2019][836356.696682] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 11:06:01 2019][838324.666480] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 11:06:01 2019][838324.678302] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 11:24:59 2019][839462.493346] Lustre: 129280:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 11:34:16 2019][840020.397921] Lustre: 127136:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 11:34:16 2019][840020.409745] Lustre: 127136:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sun Jan 27 11:47:01 2019][840784.541214] Lustre: 129251:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 11:47:01 2019][840784.553037] Lustre: 129251:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 11:47:52 2019][840836.256280] Lustre: 128092:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 11:47:52 2019][840836.268100] Lustre: 128092:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 11:49:02 2019][840905.882147] Lustre: 2937:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 11:49:02 2019][840905.893796] Lustre: 2937:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 11:49:21 2019][840924.524761] Lustre: 128092:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 11:49:21 2019][840924.536595] Lustre: 128092:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 11:50:11 2019][840974.926087] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 11:50:11 2019][840974.937908] Lustre: 129276:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 11:52:22 2019][841105.537860] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 11:58:51 2019][841494.674452] Lustre: 129259:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 11:58:51 2019][841494.686271] Lustre: 129259:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 12:06:56 2019][841980.350831] Lustre: 2976:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 12:06:56 2019][841980.362480] Lustre: 2976:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 12:10:44 2019][842207.886968] Lustre: 129259:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 12:10:44 2019][842207.898788] Lustre: 129259:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 12:25:20 2019][843083.826178] Lustre: 129259:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 12:25:20 2019][843083.837999] Lustre: 129259:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 13:20:01 2019][846365.270623] Lustre: 129259:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 13:39:37 2019][847540.682423] Lustre: 126625:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 13:46:07 2019][847930.966037] Lustre: 2976:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 14:17:02 2019][849786.099226] Lustre: 127136:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 14:17:02 2019][849786.111052] Lustre: 127136:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 14:18:43 2019][849886.826925] Lustre: 128092:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 14:18:47 2019][849891.011214] Lustre: 128085:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 14:18:47 2019][849891.023037] Lustre: 128085:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 14:19:12 2019][849916.331441] Lustre: 127247:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 14:19:12 2019][849916.343264] Lustre: 127247:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 14:22:50 2019][850134.502338] Lustre: 126624:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 14:22:50 2019][850134.514159] Lustre: 126624:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 14:47:05 2019][851589.245003] Lustre: 127063:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 14:47:05 2019][851589.256823] Lustre: 127063:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 18:23:32 2019][864576.029421] Lustre: 129245:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 18:37:05 2019][865389.706675] Lustre: 127214:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 18:40:35 2019][865599.740190] Lustre: 127134:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 18:40:35 2019][865599.752007] Lustre: 127134:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 19:24:39 2019][868243.535602] Lustre: 128084:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 19:24:39 2019][868243.547423] Lustre: 128084:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 19:32:03 2019][868687.348821] Lustre: 128084:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 20:02:00 2019][870484.692321] Lustre: 129267:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Sun Jan 27 20:02:00 2019][870484.704148] Lustre: 129267:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 20:31:26 2019][872250.202248] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Sun Jan 27 20:31:26 2019][872250.212595] Lustre: Skipped 1 previous similar message [Sun Jan 27 21:00:13 2019][873977.502856] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Sun Jan 27 21:00:13 2019][873977.513213] Lustre: Skipped 1 previous similar message [Sun Jan 27 21:29:36 2019][875740.939179] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Sun Jan 27 21:29:36 2019][875740.949534] Lustre: Skipped 1 previous similar message [Sun Jan 27 21:58:49 2019][877493.737216] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Sun Jan 27 21:58:49 2019][877493.747573] Lustre: Skipped 1 previous similar message [Sun Jan 27 22:29:22 2019][879326.134885] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Sun Jan 27 22:29:22 2019][879326.145229] Lustre: Skipped 1 previous similar message [Sun Jan 27 22:44:02 2019][880206.780201] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Sun Jan 27 22:44:02 2019][880206.790554] Lustre: Skipped 1 previous similar message [Sun Jan 27 23:12:59 2019][881943.585652] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Sun Jan 27 23:12:59 2019][881943.596011] Lustre: Skipped 1 previous similar message [Sun Jan 27 23:41:26 2019][883650.783825] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Sun Jan 27 23:41:26 2019][883650.794180] Lustre: Skipped 1 previous similar message [Sun Jan 27 23:55:54 2019][884518.593757] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Sun Jan 27 23:55:54 2019][884518.604103] Lustre: Skipped 1 previous similar message [Mon Jan 28 00:24:04 2019][886209.031298] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 00:24:04 2019][886209.041649] Lustre: Skipped 1 previous similar message [Mon Jan 28 00:52:04 2019][887888.130729] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 00:52:04 2019][887888.141083] Lustre: Skipped 1 previous similar message [Mon Jan 28 01:06:27 2019][888752.074377] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 01:06:27 2019][888752.084735] Lustre: Skipped 1 previous similar message [Mon Jan 28 01:33:52 2019][890396.762730] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 01:33:52 2019][890396.773091] Lustre: Skipped 1 previous similar message [Mon Jan 28 01:48:04 2019][891248.863641] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 01:48:04 2019][891248.873982] Lustre: Skipped 1 previous similar message [Mon Jan 28 02:15:31 2019][892896.132858] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 02:15:31 2019][892896.143213] Lustre: Skipped 1 previous similar message [Mon Jan 28 02:43:17 2019][894561.331933] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 02:43:17 2019][894561.342290] Lustre: Skipped 1 previous similar message [Mon Jan 28 03:11:37 2019][896261.922252] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 03:11:37 2019][896261.932605] Lustre: Skipped 1 previous similar message [Mon Jan 28 03:40:12 2019][897976.903148] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 03:40:12 2019][897976.913498] Lustre: Skipped 1 previous similar message [Mon Jan 28 04:08:49 2019][899693.570946] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 04:08:49 2019][899693.581300] Lustre: Skipped 1 previous similar message [Mon Jan 28 04:37:05 2019][901390.171986] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 04:37:05 2019][901390.182337] Lustre: Skipped 1 previous similar message [Mon Jan 28 04:44:23 2019][901827.705112] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.3@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Mon Jan 28 04:45:13 2019][901877.836954] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.0.10.3@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Mon Jan 28 04:46:00 2019][901924.298244] Lustre: fir-MDT0003: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:46:00 2019][901924.298269] Lustre: fir-MDT0001: Connection restored to c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) [Mon Jan 28 04:46:00 2019][901924.318677] Lustre: Skipped 2 previous similar messages [Mon Jan 28 04:46:44 2019][901968.956568] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.3@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Mon Jan 28 04:46:44 2019][901968.973863] LustreError: Skipped 1 previous similar message [Mon Jan 28 04:47:09 2019][901993.997604] Lustre: fir-MDT0003: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:47:09 2019][901994.007724] Lustre: fir-MDT0003: Connection restored to c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) [Mon Jan 28 04:47:09 2019][901994.018067] Lustre: Skipped 1 previous similar message [Mon Jan 28 04:47:31 2019][902015.540109] Lustre: fir-MDT0003: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:47:31 2019][902015.540127] Lustre: fir-MDT0001: Connection restored to c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) [Mon Jan 28 04:47:31 2019][902015.560551] Lustre: Skipped 1 previous similar message [Mon Jan 28 04:48:00 2019][902045.219820] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.0.10.3@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Mon Jan 28 04:49:07 2019][902111.568513] Lustre: fir-MDT0001: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:49:07 2019][902111.568639] Lustre: fir-MDT0003: Connection restored to c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) [Mon Jan 28 04:49:07 2019][902111.568641] Lustre: Skipped 1 previous similar message [Mon Jan 28 04:49:07 2019][902111.594166] Lustre: Skipped 1 previous similar message [Mon Jan 28 04:49:57 2019][902161.795546] Lustre: fir-MDT0001: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:49:57 2019][902161.795547] Lustre: fir-MDT0003: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:49:57 2019][902161.795576] Lustre: fir-MDT0003: Connection restored to c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) [Mon Jan 28 04:49:57 2019][902161.795578] Lustre: Skipped 1 previous similar message [Mon Jan 28 04:50:05 2019][902169.398933] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.3@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Mon Jan 28 04:50:05 2019][902169.416212] LustreError: Skipped 1 previous similar message [Mon Jan 28 04:50:50 2019][902214.824525] Lustre: fir-MDT0003: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:50:50 2019][902214.834635] Lustre: fir-MDT0003: Connection restored to c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) [Mon Jan 28 04:50:50 2019][902214.844983] Lustre: Skipped 1 previous similar message [Mon Jan 28 04:51:40 2019][902264.736013] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.3@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Mon Jan 28 04:51:40 2019][902264.753297] LustreError: Skipped 1 previous similar message [Mon Jan 28 04:52:41 2019][902325.996610] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.0.10.3@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Mon Jan 28 05:05:37 2019][903102.228654] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 05:05:37 2019][903102.239024] Lustre: Skipped 1 previous similar message [Mon Jan 28 05:33:40 2019][904785.291578] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 05:33:40 2019][904785.301941] Lustre: Skipped 1 previous similar message [Mon Jan 28 06:01:22 2019][906446.856506] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 06:01:22 2019][906446.866856] Lustre: Skipped 1 previous similar message [Mon Jan 28 06:30:06 2019][908170.692625] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 06:30:06 2019][908170.702979] Lustre: Skipped 1 previous similar message [Mon Jan 28 06:44:55 2019][909060.123347] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 06:44:55 2019][909060.133696] Lustre: Skipped 1 previous similar message [Mon Jan 28 06:59:28 2019][909933.271212] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 06:59:28 2019][909933.281565] Lustre: Skipped 1 previous similar message [Mon Jan 28 07:27:25 2019][911609.651548] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 07:27:25 2019][911609.661895] Lustre: Skipped 1 previous similar message [Mon Jan 28 07:41:40 2019][912465.142420] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 07:41:40 2019][912465.152777] Lustre: Skipped 1 previous similar message [Mon Jan 28 08:10:01 2019][914165.621330] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 08:10:01 2019][914165.631678] Lustre: Skipped 1 previous similar message [Mon Jan 28 08:38:27 2019][915871.526700] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 08:38:27 2019][915871.537047] Lustre: Skipped 1 previous similar message [Mon Jan 28 09:05:57 2019][917521.870277] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 09:05:57 2019][917521.880639] Lustre: Skipped 1 previous similar message [Mon Jan 28 09:33:58 2019][919203.110315] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 09:33:58 2019][919203.120664] Lustre: Skipped 1 previous similar message [Mon Jan 28 09:47:39 2019][920023.617003] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 09:47:39 2019][920023.627350] Lustre: Skipped 1 previous similar message [Mon Jan 28 10:15:00 2019][921664.876219] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 10:15:00 2019][921664.886567] Lustre: Skipped 1 previous similar message [Mon Jan 28 10:43:58 2019][923403.382595] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 10:43:58 2019][923403.392945] Lustre: Skipped 1 previous similar message [Mon Jan 28 11:16:01 2019][925326.415729] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 11:16:01 2019][925326.426074] Lustre: Skipped 1 previous similar message [Mon Jan 28 11:46:33 2019][927157.633910] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 11:46:33 2019][927157.644260] Lustre: Skipped 1 previous similar message [Mon Jan 28 12:15:04 2019][928869.266097] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 12:15:04 2019][928869.276467] Lustre: Skipped 1 previous similar message [Mon Jan 28 12:43:22 2019][930566.621961] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 12:43:22 2019][930566.632304] Lustre: Skipped 1 previous similar message [Mon Jan 28 13:11:56 2019][932280.678240] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 13:11:56 2019][932280.688651] Lustre: Skipped 1 previous similar message [Mon Jan 28 13:40:20 2019][933984.875021] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 13:40:20 2019][933984.885380] Lustre: Skipped 1 previous similar message [Mon Jan 28 14:09:52 2019][935757.345865] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 14:09:52 2019][935757.356211] Lustre: Skipped 1 previous similar message [Mon Jan 28 14:24:39 2019][936644.499921] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 14:24:39 2019][936644.510265] Lustre: Skipped 1 previous similar message [Mon Jan 28 14:53:51 2019][938396.333836] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Mon Jan 28 14:53:51 2019][938396.344188] Lustre: Skipped 1 previous similar message [Mon Jan 28 15:02:36 2019][938921.318311] Lustre: fir-MDT0003: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 15:02:36 2019][938921.328443] Lustre: fir-MDT0003: Connection restored to c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) [Mon Jan 28 15:05:19 2019][939084.578151] Lustre: fir-MDT0001: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 15:05:19 2019][939084.578195] Lustre: fir-MDT0003: Connection restored to c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) [Mon Jan 28 15:05:19 2019][939084.578197] Lustre: Skipped 1 previous similar message [Mon Jan 28 15:05:19 2019][939084.578221] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.3@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Mon Jan 28 15:05:19 2019][939084.621102] Lustre: Skipped 2 previous similar messages [Mon Jan 28 15:09:43 2019][939348.577458] Lustre: fir-MDT0001: haven't heard from client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) in 228 seconds. I think it's dead, and I am evicting it. exp ffffa06b06fb9400, cur 1548717156 expire 1548717006 last 1548716928 [Mon Jan 28 15:09:52 2019][939357.026242] Lustre: fir-MDT0003: haven't heard from client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) in 230 seconds. I think it's dead, and I am evicting it. exp ffffa05b3b2b9800, cur 1548717164 expire 1548717014 last 1548716934 [Mon Jan 28 15:13:04 2019][939549.699337] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds [Mon Jan 28 15:13:04 2019][939549.709680] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.3@o2ib7 (108): c: 8, oc: 0, rc: 8 [Mon Jan 28 15:34:35 2019][940840.293974] Lustre: fir-MDT0001: Connection restored to c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) [Mon Jan 28 15:34:35 2019][940840.304326] Lustre: Skipped 2 previous similar messages [Mon Jan 28 18:40:37 2019][952002.163922] Lustre: fir-MDT0001: Connection restored to 75d3df1c-c05f-80f2-4e0d-3991a06b4cf2 (at 10.8.2.9@o2ib6) [Mon Jan 28 18:40:38 2019][952003.163986] Lustre: fir-MDT0003: Connection restored to 75d3df1c-c05f-80f2-4e0d-3991a06b4cf2 (at 10.8.2.9@o2ib6) [Mon Jan 28 18:42:43 2019][952128.748389] Lustre: fir-MDT0001: Connection restored to 45d9bd1b-c260-5ee3-c014-76ee236dd98f (at 10.8.6.31@o2ib6) [Mon Jan 28 18:42:43 2019][952128.758742] Lustre: Skipped 1 previous similar message [Mon Jan 28 18:55:46 2019][952911.227627] LustreError: 127698:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Mon Jan 28 18:55:46 2019][952911.241796] LustreError: 127698:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 36 previous similar messages [Mon Jan 28 19:21:32 2019][954457.532078] LustreError: 130108:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Mon Jan 28 19:21:37 2019][954462.566268] LustreError: 127218:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Mon Jan 28 19:21:37 2019][954462.580348] LustreError: 127218:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 8 previous similar messages [Mon Jan 28 19:21:42 2019][954467.590446] LustreError: 130117:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Mon Jan 28 19:21:42 2019][954467.604523] LustreError: 130117:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 79 previous similar messages [Mon Jan 28 19:21:47 2019][954472.646487] LustreError: 127222:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Mon Jan 28 19:21:47 2019][954472.660567] LustreError: 127222:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 77 previous similar messages [Mon Jan 28 19:21:57 2019][954482.726689] LustreError: 127218:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Mon Jan 28 19:21:57 2019][954482.740766] LustreError: 127218:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 190 previous similar messages [Mon Jan 28 19:22:17 2019][954502.870694] LustreError: 3384:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Mon Jan 28 19:22:17 2019][954502.884591] LustreError: 3384:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 318 previous similar messages [Mon Jan 28 19:22:57 2019][954542.795213] LustreError: 130108:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Mon Jan 28 19:22:57 2019][954542.809372] LustreError: 130108:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 937 previous similar messages [Mon Jan 28 19:24:12 2019][954617.769070] LustreError: 129241:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Mon Jan 28 19:24:12 2019][954617.783236] LustreError: 129241:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 2306 previous similar messages [Mon Jan 28 19:26:43 2019][954768.211874] LustreError: 127702:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Mon Jan 28 19:26:43 2019][954768.226038] LustreError: 127702:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 1863 previous similar messages [Mon Jan 28 21:25:38 2019][961903.258578] Lustre: fir-MDT0001: haven't heard from client 9d0731b3-b468-e234-aad6-42e89a84a6d4 (at 10.9.101.60@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa06b06fb9000, cur 1548739711 expire 1548739561 last 1548739484 [Mon Jan 28 21:25:39 2019][961904.613502] Lustre: fir-MDT0003: haven't heard from client 9d0731b3-b468-e234-aad6-42e89a84a6d4 (at 10.9.101.60@o2ib4) in 228 seconds. I think it's dead, and I am evicting it. exp ffffa05b3b2bfc00, cur 1548739712 expire 1548739562 last 1548739484 [Mon Jan 28 21:49:13 2019][963318.514220] LustreError: 129285:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Mon Jan 28 21:49:13 2019][963318.528382] LustreError: 129285:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 184 previous similar messages [Mon Jan 28 21:50:40 2019][963405.935871] LustreError: 130112:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Mon Jan 28 21:50:40 2019][963405.949953] LustreError: 130112:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 33 previous similar messages [Mon Jan 28 22:37:14 2019][966199.621604] Lustre: fir-MDT0001: Connection restored to 9d0731b3-b468-e234-aad6-42e89a84a6d4 (at 10.9.101.60@o2ib4) [Mon Jan 28 22:37:14 2019][966199.632130] Lustre: Skipped 1 previous similar message [Tue Jan 29 02:05:04 2019][978669.484701] LustreError: 127226:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Tue Jan 29 02:05:13 2019][978678.966959] LustreError: 127218:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Tue Jan 29 02:05:13 2019][978678.981129] LustreError: 127218:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 17 previous similar messages [Tue Jan 29 02:05:33 2019][978699.220759] LustreError: 130108:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Tue Jan 29 02:05:34 2019][978699.234837] LustreError: 130108:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 24 previous similar messages [Tue Jan 29 02:06:11 2019][978736.860648] LustreError: 1012:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Tue Jan 29 02:06:11 2019][978736.874637] LustreError: 1012:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 90 previous similar messages [Tue Jan 29 02:07:29 2019][978814.834930] LustreError: 130108:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Tue Jan 29 02:07:29 2019][978814.849093] LustreError: 130108:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 965 previous similar messages [Tue Jan 29 02:10:00 2019][978965.542993] LustreError: 129239:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Tue Jan 29 02:10:00 2019][978965.557159] LustreError: 129239:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 2452 previous similar messages [Tue Jan 29 02:15:02 2019][979268.001636] LustreError: 130117:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Tue Jan 29 02:15:02 2019][979268.015801] LustreError: 130117:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 12212 previous similar messages [Tue Jan 29 02:25:02 2019][979867.848211] LustreError: 127696:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Tue Jan 29 02:25:02 2019][979867.862382] LustreError: 127696:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 3891 previous similar messages [Tue Jan 29 02:35:05 2019][980470.342520] LustreError: 127235:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 131072 GRANT, real grant 0 [Tue Jan 29 02:35:05 2019][980470.356682] LustreError: 127235:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 9178 previous similar messages [Tue Jan 29 02:45:07 2019][981072.893864] LustreError: 127702:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Tue Jan 29 02:45:07 2019][981072.908032] LustreError: 127702:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 19662 previous similar messages [Tue Jan 29 02:55:09 2019][981675.173163] LustreError: 130204:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Tue Jan 29 02:55:09 2019][981675.187335] LustreError: 130204:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 11362 previous similar messages [Tue Jan 29 03:05:10 2019][982276.172167] LustreError: 127697:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Tue Jan 29 03:05:10 2019][982276.186328] LustreError: 127697:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 7363 previous similar messages [Tue Jan 29 03:15:12 2019][982878.115698] LustreError: 127227:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Tue Jan 29 03:15:12 2019][982878.129862] LustreError: 127227:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 11647 previous similar messages [Tue Jan 29 03:25:13 2019][983478.426928] LustreError: 127265:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Tue Jan 29 03:25:13 2019][983478.441089] LustreError: 127265:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 17754 previous similar messages [Tue Jan 29 03:35:17 2019][984082.929893] LustreError: 127219:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Tue Jan 29 03:35:17 2019][984082.943972] LustreError: 127219:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 12568 previous similar messages [Tue Jan 29 03:45:17 2019][984683.279605] LustreError: 127698:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Tue Jan 29 03:45:17 2019][984683.293772] LustreError: 127698:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 29161 previous similar messages [Tue Jan 29 03:55:28 2019][985293.955165] LustreError: 127264:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Tue Jan 29 03:55:28 2019][985293.969327] LustreError: 127264:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 21019 previous similar messages [Tue Jan 29 09:28:57 2019][1005302.577707] LustreError: 130204:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 155648 GRANT, real grant 0 [Tue Jan 29 09:28:57 2019][1005302.591956] LustreError: 130204:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 9 previous similar messages [Tue Jan 29 11:22:56 2019][1012141.836695] Lustre: fir-MDT0001: Client 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) reconnecting [Tue Jan 29 11:22:56 2019][1012141.841060] Lustre: fir-MDT0003: Connection restored to 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) [Tue Jan 29 11:22:56 2019][1012141.857127] Lustre: Skipped 1 previous similar message [Tue Jan 29 11:23:18 2019][1012163.801377] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.9.0.1@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Tue Jan 29 11:23:18 2019][1012163.818655] LustreError: Skipped 2 previous similar messages [Tue Jan 29 11:23:43 2019][1012188.998391] Lustre: fir-MDT0001: Client 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) reconnecting [Tue Jan 29 11:23:43 2019][1012188.998439] Lustre: fir-MDT0003: Connection restored to 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) [Tue Jan 29 11:23:43 2019][1012188.998442] Lustre: Skipped 1 previous similar message [Tue Jan 29 11:23:43 2019][1012189.024137] Lustre: Skipped 1 previous similar message [Tue Jan 29 11:24:24 2019][1012230.585686] Lustre: fir-MDT0001: Client 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) reconnecting [Tue Jan 29 11:24:24 2019][1012230.595808] Lustre: fir-MDT0001: Connection restored to 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) [Tue Jan 29 11:24:24 2019][1012230.606157] Lustre: Skipped 1 previous similar message [Tue Jan 29 11:24:48 2019][1012254.363656] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.0.1@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Tue Jan 29 12:01:30 2019][1014456.196597] Lustre: 2937:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Tue Jan 29 12:01:30 2019][1014456.208332] Lustre: 2937:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Jan 29 15:43:50 2019][1027796.576453] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Tue Jan 29 15:43:50 2019][1027796.586892] Lustre: Skipped 1 previous similar message [Tue Jan 29 17:18:16 2019][1033462.390922] Lustre: fir-MDT0001: Connection restored to 8ae4cc68-204d-d2bb-4908-94f49692744b (at 10.8.20.15@o2ib6) [Tue Jan 29 17:18:17 2019][1033463.390184] Lustre: fir-MDT0003: Connection restored to 8ae4cc68-204d-d2bb-4908-94f49692744b (at 10.8.20.15@o2ib6) [Tue Jan 29 17:38:34 2019][1034680.842875] Lustre: fir-MDT0003: haven't heard from client 8ae4cc68-204d-d2bb-4908-94f49692744b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa0680fa60c00, cur 1548812488 expire 1548812338 last 1548812261 [Tue Jan 29 17:40:15 2019][1034781.107543] Lustre: 127250:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Tue Jan 29 17:46:18 2019][1035144.550645] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Tue Jan 29 17:46:18 2019][1035144.562553] Lustre: 129258:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Jan 29 18:04:24 2019][1036230.671820] Lustre: 129259:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Tue Jan 29 18:15:07 2019][1036873.188486] Lustre: 129267:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Tue Jan 29 18:15:07 2019][1036873.200396] Lustre: 129267:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Wed Jan 30 12:06:07 2019][1101134.378317] Lustre: fir-MDT0001: haven't heard from client 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa06858aff000, cur 1548878941 expire 1548878791 last 1548878714 [Wed Jan 30 12:06:07 2019][1101134.400107] Lustre: Skipped 1 previous similar message [Fri Feb 1 15:06:05 2019][1284734.984411] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Fri Feb 1 15:06:05 2019][1284734.994857] Lustre: Skipped 1 previous similar message [Fri Feb 1 15:12:20 2019][1285109.629262] Lustre: fir-MDT0001: Connection restored to d24c46ad-c948-5081-831c-1314be2aea23 (at 10.8.6.34@o2ib6) [Fri Feb 1 15:12:20 2019][1285109.639708] Lustre: Skipped 1 previous similar message [Mon Feb 4 07:58:54 2019][1518306.382821] Lustre: fir-MDT0003: haven't heard from client 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa05b3b2bd000, cur 1549296109 expire 1549295959 last 1549295882 [Mon Feb 4 07:58:54 2019][1518306.404525] Lustre: Skipped 1 previous similar message [Mon Feb 4 08:03:20 2019][1518572.012523] Lustre: fir-MDT0001: Connection restored to 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) [Mon Feb 4 08:03:20 2019][1518572.022870] Lustre: Skipped 1 previous similar message [Mon Feb 4 09:26:08 2019][1523540.456155] Lustre: fir-MDT0001: haven't heard from client a0fd8789-8ae9-fb75-85b8-13d1c4cef8d2 (at 10.9.101.60@o2ib4) in 228 seconds. I think it's dead, and I am evicting it. exp ffffa06fde2fd400, cur 1549301343 expire 1549301193 last 1549301115 [Mon Feb 4 09:26:08 2019][1523540.478120] Lustre: Skipped 1 previous similar message [Mon Feb 4 15:24:06 2019][1545018.416095] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 15:24:06 2019][1545018.426541] Lustre: Skipped 1 previous similar message [Mon Feb 4 15:29:40 2019][1545353.090583] Lustre: fir-MDT0001: Connection restored to 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6) [Mon Feb 4 15:29:40 2019][1545353.101021] Lustre: Skipped 1 previous similar message [Mon Feb 4 15:30:16 2019][1545388.709013] LustreError: 129974:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Mon Feb 4 15:30:16 2019][1545388.723177] LustreError: 129974:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 331 previous similar messages [Mon Feb 4 15:30:26 2019][1545398.757392] LustreError: 127697:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Mon Feb 4 15:30:26 2019][1545398.771567] LustreError: 127697:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 101 previous similar messages [Mon Feb 4 15:30:52 2019][1545424.363305] LustreError: 129974:0:(tgt_grant.c:742:tgt_grant_check()) fir-MDT0003: cli 9daa8e10-c154-ca96-d85d-3f6355b9ec3f claims 28672 GRANT, real grant 0 [Mon Feb 4 15:30:52 2019][1545424.377469] LustreError: 129974:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 2092 previous similar messages [Mon Feb 4 16:03:51 2019][1547403.788378] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 16:03:51 2019][1547403.798812] Lustre: Skipped 1 previous similar message [Mon Feb 4 16:32:11 2019][1549103.931082] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 16:32:11 2019][1549103.941530] Lustre: Skipped 1 previous similar message [Mon Feb 4 17:01:21 2019][1550854.019472] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 17:01:21 2019][1550854.029918] Lustre: Skipped 1 previous similar message [Mon Feb 4 17:29:37 2019][1552550.210763] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 17:29:37 2019][1552550.221202] Lustre: Skipped 1 previous similar message [Mon Feb 4 17:59:43 2019][1554356.215214] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 17:59:43 2019][1554356.215215] Lustre: fir-MDT0003: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 18:30:30 2019][1556203.238287] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 18:30:30 2019][1556203.248740] Lustre: Skipped 1 previous similar message [Mon Feb 4 18:59:17 2019][1557930.356544] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 18:59:17 2019][1557930.366986] Lustre: Skipped 1 previous similar message [Mon Feb 4 19:27:26 2019][1559619.305147] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 19:27:26 2019][1559619.315598] Lustre: Skipped 1 previous similar message [Mon Feb 4 19:56:37 2019][1561369.912800] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 19:56:37 2019][1561369.923349] Lustre: Skipped 1 previous similar message [Mon Feb 4 20:24:52 2019][1563064.504995] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 20:24:52 2019][1563064.515436] Lustre: Skipped 1 previous similar message [Mon Feb 4 20:38:47 2019][1563899.663739] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 20:38:47 2019][1563899.674192] Lustre: Skipped 1 previous similar message [Mon Feb 4 21:07:20 2019][1565612.717310] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 21:07:20 2019][1565612.727758] Lustre: Skipped 1 previous similar message [Mon Feb 4 21:35:59 2019][1567331.838767] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 21:35:59 2019][1567331.849220] Lustre: Skipped 1 previous similar message [Mon Feb 4 22:03:55 2019][1569007.796654] Lustre: fir-MDT0003: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 22:03:55 2019][1569007.807093] Lustre: Skipped 1 previous similar message [Mon Feb 4 22:32:29 2019][1570722.463666] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 22:32:29 2019][1570722.474119] Lustre: Skipped 1 previous similar message [Mon Feb 4 23:01:42 2019][1572474.988095] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 23:01:42 2019][1572474.998532] Lustre: Skipped 1 previous similar message [Mon Feb 4 23:30:28 2019][1574201.309639] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 23:30:28 2019][1574201.320105] Lustre: Skipped 1 previous similar message [Mon Feb 4 23:44:30 2019][1575042.956568] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 23:44:30 2019][1575042.967021] Lustre: Skipped 1 previous similar message [Mon Feb 4 23:58:43 2019][1575896.226078] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Mon Feb 4 23:58:43 2019][1575896.236511] Lustre: Skipped 1 previous similar message [Tue Feb 5 00:27:00 2019][1577593.594787] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 00:27:00 2019][1577593.605233] Lustre: Skipped 1 previous similar message [Tue Feb 5 00:56:06 2019][1579339.105282] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 00:56:06 2019][1579339.115715] Lustre: Skipped 1 previous similar message [Tue Feb 5 01:24:37 2019][1581049.756650] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 01:24:37 2019][1581049.767094] Lustre: Skipped 1 previous similar message [Tue Feb 5 01:52:49 2019][1582742.490401] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 01:52:49 2019][1582742.500841] Lustre: Skipped 1 previous similar message [Tue Feb 5 02:21:04 2019][1584437.444951] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 02:21:04 2019][1584437.455401] Lustre: Skipped 1 previous similar message [Tue Feb 5 02:48:55 2019][1586107.928877] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 02:48:55 2019][1586107.939314] Lustre: Skipped 1 previous similar message [Tue Feb 5 03:18:05 2019][1587857.802198] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 03:18:05 2019][1587857.812634] Lustre: Skipped 1 previous similar message [Tue Feb 5 03:46:48 2019][1589581.158054] Lustre: fir-MDT0003: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 03:46:48 2019][1589581.168491] Lustre: Skipped 1 previous similar message [Tue Feb 5 04:15:16 2019][1591288.979858] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 04:15:16 2019][1591288.990294] Lustre: Skipped 1 previous similar message [Tue Feb 5 04:43:44 2019][1592997.225978] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 04:43:44 2019][1592997.236422] Lustre: Skipped 1 previous similar message [Tue Feb 5 05:16:36 2019][1594969.506245] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 05:16:36 2019][1594969.516698] Lustre: Skipped 1 previous similar message [Tue Feb 5 05:45:17 2019][1596690.649204] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 05:45:17 2019][1596690.659642] Lustre: Skipped 1 previous similar message [Tue Feb 5 06:13:38 2019][1598390.990652] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 06:13:38 2019][1598391.001090] Lustre: Skipped 1 previous similar message [Tue Feb 5 06:27:41 2019][1599234.414147] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 06:27:41 2019][1599234.424581] Lustre: Skipped 1 previous similar message [Tue Feb 5 06:54:53 2019][1600866.191968] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 06:54:53 2019][1600866.202400] Lustre: Skipped 1 previous similar message [Tue Feb 5 07:07:58 2019][1601651.073639] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 07:07:58 2019][1601651.084078] Lustre: Skipped 1 previous similar message [Tue Feb 5 07:34:05 2019][1603218.597189] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 07:34:05 2019][1603218.607632] Lustre: Skipped 1 previous similar message [Tue Feb 5 08:00:01 2019][1604774.934079] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 08:00:01 2019][1604774.944521] Lustre: Skipped 1 previous similar message [Tue Feb 5 08:26:12 2019][1606345.202659] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 08:26:12 2019][1606345.213094] Lustre: Skipped 1 previous similar message [Tue Feb 5 08:52:29 2019][1607922.208014] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 08:52:29 2019][1607922.218447] Lustre: Skipped 1 previous similar message [Tue Feb 5 09:18:25 2019][1609478.454997] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 09:18:25 2019][1609478.465535] Lustre: Skipped 1 previous similar message [Tue Feb 5 09:39:00 2019][1610713.598850] Lustre: fir-MDT0001: Connection restored to 9d0731b3-b468-e234-aad6-42e89a84a6d4 (at 10.9.101.60@o2ib4) [Tue Feb 5 09:39:00 2019][1610713.609463] Lustre: Skipped 1 previous similar message [Tue Feb 5 09:44:19 2019][1611032.555116] Lustre: fir-MDT0001: Connection restored to 2df312f9-aff0-54ca-41b5-f915ec943155 (at 10.8.11.9@o2ib6) [Tue Feb 5 09:44:19 2019][1611032.565556] Lustre: Skipped 1 previous similar message [Tue Feb 5 09:44:43 2019][1611056.953577] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 09:44:43 2019][1611056.964015] Lustre: Skipped 1 previous similar message [Tue Feb 5 09:45:39 2019][1611112.593467] Lustre: fir-MDT0001: Connection restored to 9d0731b3-b468-e234-aad6-42e89a84a6d4 (at 10.9.101.60@o2ib4) [Tue Feb 5 09:45:39 2019][1611112.604077] Lustre: Skipped 1 previous similar message [Tue Feb 5 09:49:47 2019][1611361.048202] Lustre: fir-MDT0001: Connection restored to 9d0731b3-b468-e234-aad6-42e89a84a6d4 (at 10.9.101.60@o2ib4) [Tue Feb 5 09:49:47 2019][1611361.058812] Lustre: Skipped 1 previous similar message [Tue Feb 5 10:00:33 2019][1612006.760282] Lustre: fir-MDT0003: haven't heard from client 28764730-adc8-1b1a-0aa7-9bb3363d4889 (at 10.9.101.59@o2ib4) in 228 seconds. I think it's dead, and I am evicting it. exp ffffa05b3b2bc000, cur 1549389809 expire 1549389659 last 1549389581 [Tue Feb 5 10:00:33 2019][1612006.782255] Lustre: Skipped 3 previous similar messages [Tue Feb 5 10:10:15 2019][1612588.731294] Lustre: fir-MDT0001: Connection restored to 2df312f9-aff0-54ca-41b5-f915ec943155 (at 10.8.11.9@o2ib6) [Tue Feb 5 10:10:15 2019][1612588.741733] Lustre: Skipped 1 previous similar message [Tue Feb 5 10:10:59 2019][1612632.494754] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 10:10:59 2019][1612632.505203] Lustre: Skipped 1 previous similar message [Tue Feb 5 10:11:12 2019][1612645.327516] Lustre: fir-MDT0003: Connection restored to 8ae4cc68-204d-d2bb-4908-94f49692744b (at 10.8.20.15@o2ib6) [Tue Feb 5 10:11:12 2019][1612645.338058] Lustre: Skipped 1 previous similar message [Tue Feb 5 10:34:47 2019][1614061.056680] Lustre: fir-MDT0001: Connection restored to 28764730-adc8-1b1a-0aa7-9bb3363d4889 (at 10.9.101.59@o2ib4) [Tue Feb 5 10:34:47 2019][1614061.067300] Lustre: Skipped 1 previous similar message [Tue Feb 5 10:36:35 2019][1614168.170783] Lustre: fir-MDT0001: Connection restored to 2df312f9-aff0-54ca-41b5-f915ec943155 (at 10.8.11.9@o2ib6) [Tue Feb 5 10:36:35 2019][1614168.181224] Lustre: Skipped 1 previous similar message [Tue Feb 5 10:36:51 2019][1614184.775610] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 10:36:51 2019][1614184.786052] Lustre: Skipped 1 previous similar message [Tue Feb 5 10:37:04 2019][1614197.848570] Lustre: fir-MDT0003: Connection restored to 8ae4cc68-204d-d2bb-4908-94f49692744b (at 10.8.20.15@o2ib6) [Tue Feb 5 10:37:04 2019][1614197.859093] Lustre: Skipped 1 previous similar message [Tue Feb 5 10:44:56 2019][1614669.515852] Lustre: fir-MDT0001: Connection restored to 28764730-adc8-1b1a-0aa7-9bb3363d4889 (at 10.9.101.59@o2ib4) [Tue Feb 5 10:44:56 2019][1614669.526475] Lustre: Skipped 1 previous similar message [Tue Feb 5 11:20:44 2019][1616817.822156] Lustre: fir-MDT0001: Connection restored to 2df312f9-aff0-54ca-41b5-f915ec943155 (at 10.8.11.9@o2ib6) [Tue Feb 5 11:20:44 2019][1616817.832591] Lustre: Skipped 1 previous similar message [Tue Feb 5 11:21:04 2019][1616837.758750] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 11:21:04 2019][1616837.769184] Lustre: Skipped 1 previous similar message [Tue Feb 5 11:21:10 2019][1616843.501114] Lustre: fir-MDT0001: Connection restored to 8ae4cc68-204d-d2bb-4908-94f49692744b (at 10.8.20.15@o2ib6) [Tue Feb 5 11:21:10 2019][1616843.511642] Lustre: Skipped 1 previous similar message [Tue Feb 5 11:22:23 2019][1616916.962772] Lustre: fir-MDT0001: Connection restored to 5fd45b8a-8ee3-7336-9b2c-6620fd288834 (at 10.8.1.29@o2ib6) [Tue Feb 5 11:22:23 2019][1616916.973216] Lustre: Skipped 1 previous similar message [Tue Feb 5 11:25:19 2019][1617093.021213] Lustre: fir-MDT0001: Connection restored to 154a1d06-667e-f425-5a30-53b3df77dc1d (at 10.9.107.69@o2ib4) [Tue Feb 5 11:25:19 2019][1617093.031824] Lustre: Skipped 1 previous similar message [Tue Feb 5 11:33:31 2019][1617584.506353] Lustre: fir-MDT0001: Connection restored to e288d2fa-1494-4b6c-352f-7c16658a712b (at 10.8.1.1@o2ib6) [Tue Feb 5 11:33:31 2019][1617584.516704] Lustre: Skipped 3 previous similar messages [Tue Feb 5 11:38:18 2019][1617871.818577] Lustre: fir-MDT0001: Connection restored to 904d4093-60f5-6c01-6013-a990fb837712 (at 10.8.1.33@o2ib6) [Tue Feb 5 11:38:18 2019][1617871.829027] Lustre: Skipped 1 previous similar message [Tue Feb 5 11:38:22 2019][1617875.717157] Lustre: fir-MDT0003: haven't heard from client f844fdc2-4b3f-a5e3-d40e-580f6f5c3bcb (at 10.9.101.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa065f483fc00, cur 1549395678 expire 1549395528 last 1549395451 [Tue Feb 5 11:38:22 2019][1617875.739118] Lustre: Skipped 1 previous similar message [Tue Feb 5 11:38:27 2019][1617880.641492] Lustre: fir-MDT0001: haven't heard from client e19ebed6-7d88-5c59-21b2-1eb456a47be8 (at 10.9.101.60@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa05210694800, cur 1549395683 expire 1549395533 last 1549395456 [Tue Feb 5 11:38:27 2019][1617880.663458] Lustre: Skipped 1 previous similar message [Tue Feb 5 11:39:32 2019][1617945.219064] Lustre: fir-MDT0001: Connection restored to 2708a86e-3625-5509-9d0d-aaed4fc61a05 (at 10.8.1.31@o2ib6) [Tue Feb 5 11:39:32 2019][1617945.229504] Lustre: Skipped 3 previous similar messages [Tue Feb 5 11:46:08 2019][1618341.899043] Lustre: fir-MDT0003: Connection restored to 2df312f9-aff0-54ca-41b5-f915ec943155 (at 10.8.11.9@o2ib6) [Tue Feb 5 11:46:08 2019][1618341.909488] Lustre: Skipped 3 previous similar messages [Tue Feb 5 12:01:33 2019][1619267.016149] Lustre: fir-MDT0001: Connection restored to 3c005caf-e304-dd1a-234a-65ceb646a930 (at 10.8.1.2@o2ib6) [Tue Feb 5 12:01:33 2019][1619267.026497] Lustre: Skipped 5 previous similar messages [Tue Feb 5 12:05:14 2019][1619487.922760] Lustre: fir-MDT0001: Connection restored to 8f14391a-cd28-b53b-3ae4-9c533f7347f0 (at 10.8.1.4@o2ib6) [Tue Feb 5 12:05:14 2019][1619487.933116] Lustre: Skipped 1 previous similar message [Tue Feb 5 12:11:59 2019][1619892.845736] Lustre: fir-MDT0001: Connection restored to 2df312f9-aff0-54ca-41b5-f915ec943155 (at 10.8.11.9@o2ib6) [Tue Feb 5 12:11:59 2019][1619892.856175] Lustre: Skipped 1 previous similar message [Tue Feb 5 12:16:03 2019][1620136.334003] Lustre: fir-MDT0001: Connection restored to 8f14391a-cd28-b53b-3ae4-9c533f7347f0 (at 10.8.1.4@o2ib6) [Tue Feb 5 12:16:03 2019][1620136.344367] Lustre: Skipped 7 previous similar messages [Tue Feb 5 12:27:07 2019][1620800.567905] Lustre: fir-MDT0001: haven't heard from client 0efd883e-065d-0727-44f7-75a6230140a1 (at 10.9.0.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa06b06fb8c00, cur 1549398603 expire 1549398453 last 1549398376 [Tue Feb 5 12:27:07 2019][1620800.589614] Lustre: Skipped 1 previous similar message [Tue Feb 5 12:37:33 2019][1621427.009251] Lustre: fir-MDT0001: Connection restored to 2df312f9-aff0-54ca-41b5-f915ec943155 (at 10.8.11.9@o2ib6) [Tue Feb 5 12:37:33 2019][1621427.019686] Lustre: Skipped 1 previous similar message [Tue Feb 5 12:40:42 2019][1621616.158989] Lustre: fir-MDT0001: Connection restored to 775ad82c-450c-c16e-bb21-700cda6d4afd (at 10.9.103.39@o2ib4) [Tue Feb 5 12:40:42 2019][1621616.169601] Lustre: Skipped 5 previous similar messages [Tue Feb 5 13:03:37 2019][1622990.884732] Lustre: fir-MDT0001: Connection restored to 2df312f9-aff0-54ca-41b5-f915ec943155 (at 10.8.11.9@o2ib6) [Tue Feb 5 13:03:37 2019][1622990.895169] Lustre: Skipped 5 previous similar messages [Tue Feb 5 13:03:52 2019][1623005.853426] Lustre: fir-MDT0001: Connection restored to 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) [Tue Feb 5 13:03:52 2019][1623005.863863] Lustre: Skipped 1 previous similar message [Tue Feb 5 13:17:50 2019][1623843.954412] Lustre: fir-MDT0001: Connection restored to 0efd883e-065d-0727-44f7-75a6230140a1 (at 10.9.0.2@o2ib4) [Tue Feb 5 13:17:50 2019][1623843.964785] Lustre: Skipped 3 previous similar messages [Tue Feb 5 13:21:17 2019][1624051.217886] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Tue Feb 5 13:21:17 2019][1624051.228323] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.201@o2ib7 (17): c: 6, oc: 0, rc: 8 [Tue Feb 5 13:21:31 2019][1624065.218003] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds [Tue Feb 5 13:21:31 2019][1624065.228433] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 6 previous similar messages [Tue Feb 5 13:21:31 2019][1624065.238861] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.212@o2ib7 (41): c: 6, oc: 0, rc: 8 [Tue Feb 5 13:21:31 2019][1624065.251192] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 6 previous similar messages [Tue Feb 5 13:21:44 2019][1624078.126115] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549401836/real 1549401836] req@ffffa05a8d816600 x1623666992584064/t0(0) o106->fir-MDT0001@10.8.1.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401880 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Tue Feb 5 13:21:44 2019][1624078.153447] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [Tue Feb 5 13:21:44 2019][1624078.163462] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.2@o2ib6 from [Tue Feb 5 13:21:44 2019][1624078.174169] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 5 previous similar messages [Tue Feb 5 13:21:45 2019][1624078.626100] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549401881/real 1549401881] req@ffffa05a8d816600 x1623666992584064/t0(0) o106->fir-MDT0001@10.8.1.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401925 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:21:45 2019][1624078.653778] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 62039 previous similar messages [Tue Feb 5 13:21:45 2019][1624078.664126] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.2@o2ib6 from [Tue Feb 5 13:21:45 2019][1624078.674831] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 62039 previous similar messages [Tue Feb 5 13:21:46 2019][1624079.626108] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549401882/real 1549401882] req@ffffa05a8d816600 x1623666992584064/t0(0) o106->fir-MDT0001@10.8.1.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401926 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:21:46 2019][1624079.653788] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 131577 previous similar messages [Tue Feb 5 13:21:46 2019][1624079.664229] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.2@o2ib6 from [Tue Feb 5 13:21:46 2019][1624079.674934] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 131577 previous similar messages [Tue Feb 5 13:21:48 2019][1624081.626130] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549401884/real 1549401884] req@ffffa05a8d816600 x1623666992584064/t0(0) o106->fir-MDT0001@10.8.1.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401928 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:21:48 2019][1624081.653829] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 271461 previous similar messages [Tue Feb 5 13:21:48 2019][1624081.664260] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.2@o2ib6 from [Tue Feb 5 13:21:48 2019][1624081.674967] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 271461 previous similar messages [Tue Feb 5 13:21:52 2019][1624085.626157] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549401888/real 1549401888] req@ffffa05a8d816600 x1623666992584064/t0(0) o106->fir-MDT0001@10.8.1.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401932 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:21:52 2019][1624085.653836] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 551679 previous similar messages [Tue Feb 5 13:21:52 2019][1624085.664266] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.2@o2ib6 from [Tue Feb 5 13:21:52 2019][1624085.674959] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 551679 previous similar messages [Tue Feb 5 13:22:00 2019][1624093.626229] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549401896/real 1549401896] req@ffffa05a8d816600 x1623666992584064/t0(0) o106->fir-MDT0001@10.8.1.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401940 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:22:00 2019][1624093.653912] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1116663 previous similar messages [Tue Feb 5 13:22:00 2019][1624093.664438] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.2@o2ib6 from [Tue Feb 5 13:22:00 2019][1624093.675148] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 1116663 previous similar messages [Tue Feb 5 13:22:16 2019][1624109.626354] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549401912/real 1549401912] req@ffffa05a8d816600 x1623666992584064/t0(0) o106->fir-MDT0001@10.8.1.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401956 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:22:16 2019][1624109.654036] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2236645 previous similar messages [Tue Feb 5 13:22:16 2019][1624109.664562] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.2@o2ib6 from [Tue Feb 5 13:22:16 2019][1624109.675278] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 2236645 previous similar messages [Tue Feb 5 13:22:48 2019][1624141.626613] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549401944/real 1549401944] req@ffffa05a8d816600 x1623666992584064/t0(0) o106->fir-MDT0001@10.8.1.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401988 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:22:48 2019][1624141.654314] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4505468 previous similar messages [Tue Feb 5 13:22:48 2019][1624141.664836] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.2@o2ib6 from [Tue Feb 5 13:22:48 2019][1624141.675546] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 4505468 previous similar messages [Tue Feb 5 13:23:52 2019][1624205.627131] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549402008/real 1549402008] req@ffffa05a8d816600 x1623666992584064/t0(0) o106->fir-MDT0001@10.8.1.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549402052 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:23:52 2019][1624205.654812] Lustre: 127136:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8955953 previous similar messages [Tue Feb 5 13:23:52 2019][1624205.665337] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.2@o2ib6 from [Tue Feb 5 13:23:52 2019][1624205.676046] LNetError: 127136:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 8955953 previous similar messages [Tue Feb 5 13:24:11 2019][1624224.784320] Lustre: fir-MDT0003: haven't heard from client ba761368-8a01-ecd0-1153-4bf3475f3fd0 (at 10.8.1.25@o2ib6) in 229 seconds. I think it's dead, and I am evicting it. exp ffffa07771dc5800, cur 1549402027 expire 1549401877 last 1549401798 [Tue Feb 5 13:24:11 2019][1624224.806118] Lustre: Skipped 1 previous similar message [Tue Feb 5 13:24:21 2019][1624234.576368] LNet: Service thread pid 127136 was inactive for 200.44s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Tue Feb 5 13:24:21 2019][1624234.593555] Pid: 127136, comm: mdt00_005 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Tue Feb 5 13:24:21 2019][1624234.603551] Call Trace: [Tue Feb 5 13:24:21 2019][1624234.606186] [] 0xffffffffffffffff [Tue Feb 5 13:24:21 2019][1624234.611392] LustreError: dumping log to /tmp/lustre-log.1549402037.127136 [Tue Feb 5 13:24:21 2019][1624234.633417] Lustre: fir-MDT0001: haven't heard from client 9daa8e10-c154-ca96-d85d-3f6355b9ec3f (at 10.8.2.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa06858afe400, cur 1549402037 expire 1549401887 last 1549401810 [Tue Feb 5 13:24:21 2019][1624234.655411] Lustre: Skipped 21 previous similar messages [Tue Feb 5 13:24:21 2019][1624234.661788] LNet: Service thread pid 127136 completed after 200.53s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Tue Feb 5 13:26:16 2019][1624350.220348] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.204@o2ib7: 0 seconds [Tue Feb 5 13:26:17 2019][1624351.220347] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.201@o2ib7: 0 seconds [Tue Feb 5 13:26:17 2019][1624351.230778] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 1 previous similar message [Tue Feb 5 13:26:18 2019][1624352.220339] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.202@o2ib7: 0 seconds [Tue Feb 5 13:26:18 2019][1624352.230767] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 2 previous similar messages [Tue Feb 5 13:26:31 2019][1624365.220436] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.212@o2ib7: 0 seconds [Tue Feb 5 13:26:31 2019][1624365.230863] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 1 previous similar message [Tue Feb 5 13:31:18 2019][1624652.222773] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.201@o2ib7: 1 seconds [Tue Feb 5 13:31:32 2019][1624666.222863] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.212@o2ib7: 0 seconds [Tue Feb 5 13:31:32 2019][1624666.233295] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 6 previous similar messages [Tue Feb 5 13:36:17 2019][1624951.225186] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.204@o2ib7: 0 seconds [Tue Feb 5 13:41:19 2019][1625253.227590] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.210@o2ib7: 1 seconds [Tue Feb 5 13:41:19 2019][1625253.238024] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 7 previous similar messages [Tue Feb 5 13:56:19 2019][1626153.234841] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.204@o2ib7: 0 seconds [Tue Feb 5 13:56:37 2019][1626171.234985] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.212@o2ib7: 0 seconds [Tue Feb 5 13:56:37 2019][1626171.245414] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 6 previous similar messages [Tue Feb 5 14:00:44 2019][1626417.370904] Lustre: fir-MDT0003: Connection restored to 0efd883e-065d-0727-44f7-75a6230140a1 (at 10.9.0.2@o2ib4) [Tue Feb 5 14:00:44 2019][1626417.381251] Lustre: Skipped 1 previous similar message [Tue Feb 5 14:01:09 2019][1626442.715489] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.0.1@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Tue Feb 5 14:01:09 2019][1626442.732788] LustreError: Skipped 1 previous similar message [Tue Feb 5 14:01:34 2019][1626467.782781] Lustre: fir-MDT0001: Connection restored to 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) [Tue Feb 5 14:01:34 2019][1626467.793127] Lustre: Skipped 1 previous similar message [Tue Feb 5 14:24:36 2019][1627850.160221] Lustre: fir-MDT0001: Connection restored to 6da5269e-e6e7-e930-ea8b-e990b1fd18b0 (at 10.9.101.72@o2ib4) [Tue Feb 5 14:24:36 2019][1627850.170835] Lustre: Skipped 1 previous similar message [Tue Feb 5 14:24:37 2019][1627850.848334] Lustre: fir-MDT0001: Connection restored to f00399f0-46cf-c80f-b0b6-1b044a6fb9c6 (at 10.9.101.40@o2ib4) [Tue Feb 5 14:24:37 2019][1627850.858936] Lustre: Skipped 3 previous similar messages [Tue Feb 5 14:24:38 2019][1627851.881011] Lustre: fir-MDT0001: Connection restored to 64c2e10f-595e-bb8e-efbd-0544992f523c (at 10.9.101.49@o2ib4) [Tue Feb 5 14:24:38 2019][1627851.891621] Lustre: Skipped 17 previous similar messages [Tue Feb 5 14:24:40 2019][1627854.031676] Lustre: fir-MDT0001: Connection restored to 4c5b69ea-d1f1-0261-ea03-15f22270fb92 (at 10.9.101.2@o2ib4) [Tue Feb 5 14:24:40 2019][1627854.042194] Lustre: Skipped 23 previous similar messages [Tue Feb 5 14:24:44 2019][1627858.164628] Lustre: fir-MDT0001: Connection restored to e2d946d5-12d9-285b-e4ca-825370f3d1df (at 10.9.101.22@o2ib4) [Tue Feb 5 14:24:44 2019][1627858.175236] Lustre: Skipped 9 previous similar messages [Tue Feb 5 14:53:03 2019][1629557.166972] Lustre: fir-MDT0001: Connection restored to 44a223c4-dd83-0a85-5c05-20e7d86a6af5 (at 10.8.1.12@o2ib6) [Tue Feb 5 14:53:03 2019][1629557.177406] Lustre: Skipped 1 previous similar message [Tue Feb 5 14:53:06 2019][1629560.036435] Lustre: fir-MDT0001: Connection restored to 254b94e8-a059-c2e7-f0ce-7ccba72fe51c (at 10.8.1.34@o2ib6) [Tue Feb 5 14:53:06 2019][1629560.046872] Lustre: Skipped 1 previous similar message [Tue Feb 5 14:53:35 2019][1629588.432979] Lustre: fir-MDT0003: Connection restored to e6f15ae7-1884-d676-5174-cffd0284b92d (at 10.8.1.30@o2ib6) [Tue Feb 5 14:53:35 2019][1629588.443411] Lustre: Skipped 1 previous similar message [Tue Feb 5 14:54:22 2019][1629636.054625] Lustre: fir-MDT0001: Connection restored to cafb4a30-f828-c84f-d0a5-247d32765a3e (at 10.8.1.27@o2ib6) [Tue Feb 5 14:54:22 2019][1629636.065077] Lustre: Skipped 1 previous similar message [Tue Feb 5 14:55:05 2019][1629679.099277] Lustre: fir-MDT0001: Connection restored to 763f631c-7b84-895c-764f-d88426b5fe26 (at 10.8.1.3@o2ib6) [Tue Feb 5 14:55:05 2019][1629679.109625] Lustre: Skipped 3 previous similar messages [Tue Feb 5 14:55:24 2019][1629697.857049] Lustre: fir-MDT0001: Connection restored to a811a23e-a19d-35f5-63e7-a6dd4ef128fb (at 10.8.1.28@o2ib6) [Tue Feb 5 14:55:24 2019][1629697.867491] Lustre: Skipped 1 previous similar message [Tue Feb 5 14:56:17 2019][1629750.323965] Lustre: fir-MDT0001: Connection restored to ba7fdf11-52aa-7842-bf5d-1305febabf57 (at 10.8.1.11@o2ib6) [Tue Feb 5 14:56:17 2019][1629750.334402] Lustre: Skipped 3 previous similar messages [Tue Feb 5 14:57:24 2019][1629818.256720] Lustre: fir-MDT0001: Connection restored to 9ea86c7e-c197-f9f9-5450-9cd734e89e2d (at 10.8.1.9@o2ib6) [Tue Feb 5 14:57:24 2019][1629818.267071] Lustre: Skipped 15 previous similar messages [Tue Feb 5 14:59:36 2019][1629949.341217] Lustre: fir-MDT0001: Connection restored to 277b3821-f326-dcfd-7723-1e7e3e6174f1 (at 10.8.1.21@o2ib6) [Tue Feb 5 14:59:36 2019][1629949.351648] Lustre: Skipped 37 previous similar messages [Tue Feb 5 15:15:13 2019][1630886.396539] Lustre: fir-MDT0001: Connection restored to ac16def5-1a59-80e5-2e16-45b58fcd0330 (at 10.8.2.8@o2ib6) [Tue Feb 5 15:15:13 2019][1630886.406890] Lustre: Skipped 13 previous similar messages [Tue Feb 5 15:15:46 2019][1630919.364110] Lustre: fir-MDT0001: Connection restored to 4600638c-d686-27e8-2646-fd49b60d2ae1 (at 10.9.102.13@o2ib4) [Tue Feb 5 15:15:46 2019][1630919.374716] Lustre: Skipped 17 previous similar messages [Tue Feb 5 15:16:51 2019][1630984.597901] Lustre: fir-MDT0001: Connection restored to fa5fc8be-f740-e109-5125-43d4f0bf6a14 (at 10.9.102.33@o2ib4) [Tue Feb 5 15:16:51 2019][1630984.608509] Lustre: Skipped 7 previous similar messages [Tue Feb 5 15:19:18 2019][1631132.221182] Lustre: fir-MDT0001: Connection restored to 78a94154-9621-e13c-f5a0-60f10c4683a5 (at 10.9.102.71@o2ib4) [Tue Feb 5 15:19:18 2019][1631132.231795] Lustre: Skipped 19 previous similar messages [Tue Feb 5 15:23:35 2019][1631388.518562] Lustre: fir-MDT0001: Connection restored to 26c5771f-6bff-b79c-941d-a328fa48123c (at 10.9.102.6@o2ib4) [Tue Feb 5 15:23:35 2019][1631388.529111] Lustre: Skipped 125 previous similar messages [Tue Feb 5 15:32:18 2019][1631911.539351] Lustre: fir-MDT0001: Connection restored to 282817f5-6ae5-e49a-959f-04d16934f700 (at 10.9.101.12@o2ib4) [Tue Feb 5 15:32:18 2019][1631911.549975] Lustre: Skipped 37 previous similar messages [Tue Feb 5 15:44:00 2019][1632614.226652] Lustre: fir-MDT0001: Connection restored to 8085461b-ece8-9665-6f8f-29041438e681 (at 10.9.102.56@o2ib4) [Tue Feb 5 15:44:00 2019][1632614.237269] Lustre: Skipped 65 previous similar messages [Tue Feb 5 15:54:17 2019][1633230.425650] Lustre: fir-MDT0001: Connection restored to 6ff78f59-4d7f-d121-1724-154727e4138f (at 10.9.0.64@o2ib4) [Tue Feb 5 15:54:17 2019][1633230.436090] Lustre: Skipped 3 previous similar messages [Tue Feb 5 16:09:58 2019][1634171.391783] Lustre: fir-MDT0001: Connection restored to ba1e1fbb-a72d-16b5-8ad9-93cddc13d3a5 (at 10.9.0.63@o2ib4) [Tue Feb 5 16:09:58 2019][1634171.402219] Lustre: Skipped 4 previous similar messages [Tue Feb 5 16:22:21 2019][1634914.899599] Lustre: fir-MDT0001: Connection restored to 55a5f43f-2070-1278-9f4b-9a1845c8940d (at 10.8.11.22@o2ib6) [Tue Feb 5 16:22:21 2019][1634914.910125] Lustre: Skipped 5 previous similar messages [Tue Feb 5 16:33:53 2019][1635606.981196] Lustre: fir-MDT0001: Connection restored to 7220072f-2a24-f2fe-045d-d370434701d9 (at 10.9.104.3@o2ib4) [Tue Feb 5 16:33:53 2019][1635606.991721] Lustre: Skipped 1 previous similar message [Tue Feb 5 16:34:59 2019][1635672.495587] Lustre: 126632:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549413467/real 1549413467] req@ffffa08b57fc0600 x1623667029680048/t0(0) o104->fir-MDT0001@10.9.0.81@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1549413474 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Tue Feb 5 16:34:59 2019][1635672.523010] Lustre: 126632:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4060430 previous similar messages [Tue Feb 5 16:35:06 2019][1635679.533147] LustreError: 126632:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.0.81@o2ib4) returned error from blocking AST (req@ffffa08b57fc0600 x1623667029680048 status -107 rc -107), evict it ns: mdt-fir-MDT0001_UUID lock: ffffa06ac78e86c0/0x8c1de5388f670863 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.9.0.81@o2ib4 remote: 0xaa31c5400b8b1042 expref: 7 pid: 129255 timeout: 1635815 lvb_type: 0 [Tue Feb 5 16:35:06 2019][1635679.576089] LustreError: 138-a: fir-MDT0001: A client on nid 10.9.0.81@o2ib4 was evicted due to a lock blocking callback time out: rc -107 [Tue Feb 5 16:35:06 2019][1635679.588723] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 14s: evicting client at 10.9.0.81@o2ib4 ns: mdt-fir-MDT0001_UUID lock: ffffa06ac78e86c0/0x8c1de5388f670863 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x8:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.9.0.81@o2ib4 remote: 0xaa31c5400b8b1042 expref: 8 pid: 129255 timeout: 0 lvb_type: 0 [Tue Feb 5 16:35:11 2019][1635684.895863] Lustre: fir-MDT0003: haven't heard from client 61d0fcbf-b5c3-a78f-9cdf-55e61d025648 (at 10.9.0.81@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa0750a306000, cur 1549413487 expire 1549413337 last 1549413260 [Tue Feb 5 16:35:11 2019][1635684.917658] Lustre: Skipped 21 previous similar messages [Tue Feb 5 16:46:05 2019][1636338.879662] Lustre: fir-MDT0001: Connection restored to a60735d0-fd55-3a91-e138-5b2efd5cb4f9 (at 10.9.104.8@o2ib4) [Tue Feb 5 16:46:05 2019][1636338.890186] Lustre: Skipped 3 previous similar messages [Tue Feb 5 17:00:23 2019][1637197.159661] Lustre: fir-MDT0001: Connection restored to 2df312f9-aff0-54ca-41b5-f915ec943155 (at 10.8.11.9@o2ib6) [Tue Feb 5 17:00:23 2019][1637197.170102] Lustre: Skipped 5 previous similar messages [Tue Feb 5 17:07:58 2019][1637651.704944] Lustre: fir-MDT0003: haven't heard from client 5475bc15-b344-a614-5762-811f76d6d107 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa063771de800, cur 1549415454 expire 1549415304 last 1549415227 [Tue Feb 5 17:19:53 2019][1638367.051312] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.9.0.1@o2ib4 ns: mdt-fir-MDT0003_UUID lock: ffffa0570af73a80/0x8c1de538aea261d2 lrc: 3/0,0 mode: PR/PR res: [0x280000c1b:0x635e:0x0].0x0 bits 0x5b/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.9.0.1@o2ib4 remote: 0xc4e69a80f22a6999 expref: 986614 pid: 129262 timeout: 1638353 lvb_type: 0 [Tue Feb 5 17:19:53 2019][1638367.089706] LustreError: 51825:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.9.0.1@o2ib4 arrived at 1549416169 with bad export cookie 10096477970625601052 [Tue Feb 5 17:19:53 2019][1638367.105264] LustreError: 51825:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 9 previous similar messages [Tue Feb 5 17:19:53 2019][1638367.105599] Lustre: fir-MDT0003: Connection restored to 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) [Tue Feb 5 17:20:14 2019][1638387.545843] Lustre: 129795:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (118:1s); client may timeout. req@ffffa076aa2ca700 x1624565594509360/t8838884737(0) o36->673a177f-1230-a616-2a0b-8995b4882735@10.9.0.1@o2ib4:658/0 lens 504/424 e 3 to 0 dl 1549416188 ref 1 fl Complete:/0/0 rc 0/0 [Tue Feb 5 17:23:40 2019][1638593.774498] Lustre: fir-MDT0003: haven't heard from client 673a177f-1230-a616-2a0b-8995b4882735 (at 10.9.0.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa05a13dff000, cur 1549416396 expire 1549416246 last 1549416169 [Tue Feb 5 17:23:40 2019][1638593.796203] Lustre: Skipped 1 previous similar message [Tue Feb 5 17:29:56 2019][1638969.714344] Lustre: fir-MDT0001: haven't heard from client 673a177f-1230-a616-2a0b-8995b4882735 (at 10.9.0.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa08b4df40800, cur 1549416772 expire 1549416622 last 1549416545 [Tue Feb 5 17:31:06 2019][1639040.276617] Lustre: fir-MDT0001: Connection restored to 0732bf9f-f5f9-3f7f-d05c-6eafb128d46c (at 10.8.0.66@o2ib6) [Tue Feb 5 17:31:06 2019][1639040.287050] Lustre: Skipped 1 previous similar message [Tue Feb 5 17:31:43 2019][1639076.526819] Lustre: DEBUG MARKER: Tue Feb 5 17:34:38 2019 [Tue Feb 5 17:31:43 2019][1639076.526819] [Tue Feb 5 17:43:51 2019][1639804.721044] Lustre: fir-MDT0003: haven't heard from client db4d27b6-f63b-4d46-2c87-13eadaadf021 (at 10.9.0.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa08b5eb1e800, cur 1549417607 expire 1549417457 last 1549417380 [Tue Feb 5 17:44:33 2019][1639846.717277] Lustre: fir-MDT0001: Connection restored to 0efd883e-065d-0727-44f7-75a6230140a1 (at 10.9.0.2@o2ib4) [Tue Feb 5 17:44:33 2019][1639846.727630] Lustre: Skipped 5 previous similar messages [Tue Feb 5 18:16:12 2019][1641745.587199] Lustre: fir-MDT0001: Connection restored to 55a5f43f-2070-1278-9f4b-9a1845c8940d (at 10.8.11.22@o2ib6) [Tue Feb 5 18:16:12 2019][1641745.597724] Lustre: Skipped 3 previous similar messages [Tue Feb 5 18:18:02 2019][1641855.737472] Lustre: fir-MDT0001: haven't heard from client a60735d0-fd55-3a91-e138-5b2efd5cb4f9 (at 10.9.104.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa072c61d1400, cur 1549419658 expire 1549419508 last 1549419431 [Tue Feb 5 18:18:02 2019][1641855.759353] Lustre: Skipped 1 previous similar message [Tue Feb 5 18:41:38 2019][1643271.810497] Lustre: fir-MDT0001: Connection restored to 55a5f43f-2070-1278-9f4b-9a1845c8940d (at 10.8.11.22@o2ib6) [Tue Feb 5 18:41:38 2019][1643271.810498] Lustre: fir-MDT0003: Connection restored to 55a5f43f-2070-1278-9f4b-9a1845c8940d (at 10.8.11.22@o2ib6) [Tue Feb 5 19:00:55 2019][1644428.617065] Lustre: fir-MDT0001: Connection restored to dfbc3ca1-e3f2-2f8c-ef1d-6d31c339e5cc (at 10.9.103.31@o2ib4) [Tue Feb 5 19:00:55 2019][1644428.627677] Lustre: Skipped 1 previous similar message [Tue Feb 5 19:02:11 2019][1644505.022556] Lustre: fir-MDT0001: Connection restored to 7faa9bb1-5fa1-02dc-cb52-0e15585b3329 (at 10.9.103.42@o2ib4) [Tue Feb 5 19:02:11 2019][1644505.033167] Lustre: Skipped 1 previous similar message [Tue Feb 5 19:02:14 2019][1644508.473479] Lustre: fir-MDT0001: Connection restored to 775ad82c-450c-c16e-bb21-700cda6d4afd (at 10.9.103.39@o2ib4) [Tue Feb 5 19:02:14 2019][1644508.484096] Lustre: Skipped 1 previous similar message [Tue Feb 5 19:21:02 2019][1645636.127465] Lustre: fir-MDT0001: Connection restored to 55a5f43f-2070-1278-9f4b-9a1845c8940d (at 10.8.11.22@o2ib6) [Tue Feb 5 19:21:02 2019][1645636.127466] Lustre: fir-MDT0003: Connection restored to 55a5f43f-2070-1278-9f4b-9a1845c8940d (at 10.8.11.22@o2ib6) [Tue Feb 5 19:46:19 2019][1647152.626239] Lustre: fir-MDT0001: Connection restored to 55a5f43f-2070-1278-9f4b-9a1845c8940d (at 10.8.11.22@o2ib6) [Tue Feb 5 19:46:19 2019][1647152.626240] Lustre: fir-MDT0003: Connection restored to 55a5f43f-2070-1278-9f4b-9a1845c8940d (at 10.8.11.22@o2ib6) [Tue Feb 5 20:11:57 2019][1648690.821642] Lustre: fir-MDT0001: Connection restored to 55a5f43f-2070-1278-9f4b-9a1845c8940d (at 10.8.11.22@o2ib6) [Tue Feb 5 20:11:57 2019][1648690.821643] Lustre: fir-MDT0003: Connection restored to 55a5f43f-2070-1278-9f4b-9a1845c8940d (at 10.8.11.22@o2ib6) [Tue Feb 5 20:12:12 2019][1648706.377778] Lustre: fir-MDT0001: Connection restored to b34755b7-d181-850b-8754-b1f884364504 (at 10.9.103.44@o2ib4) [Tue Feb 5 20:12:12 2019][1648706.388392] Lustre: Skipped 1 previous similar message [Tue Feb 5 20:13:22 2019][1648776.271585] Lustre: fir-MDT0003: Connection restored to d295bc3f-a6cd-b523-283f-d68b438ddc06 (at 10.8.3.13@o2ib6) [Tue Feb 5 20:13:22 2019][1648776.271586] Lustre: fir-MDT0001: Connection restored to d295bc3f-a6cd-b523-283f-d68b438ddc06 (at 10.8.3.13@o2ib6) [Tue Feb 5 20:13:57 2019][1648811.399871] Lustre: fir-MDT0001: Connection restored to 61f87f92-e850-3b4a-54a2-35a6e7dde0d3 (at 10.8.3.35@o2ib6) [Tue Feb 5 20:13:57 2019][1648811.410306] Lustre: Skipped 1 previous similar message [Tue Feb 5 20:15:18 2019][1648892.012698] Lustre: fir-MDT0001: Connection restored to ecb1b75a-8cd7-f25c-3a61-8ac620b0c3af (at 10.8.3.19@o2ib6) [Tue Feb 5 20:15:18 2019][1648892.023137] Lustre: Skipped 1 previous similar message [Tue Feb 5 20:17:59 2019][1649052.934437] Lustre: fir-MDT0001: Connection restored to 6b010079-d818-22c1-0543-18450a32bb73 (at 10.8.3.7@o2ib6) [Tue Feb 5 20:17:59 2019][1649052.944792] Lustre: Skipped 1 previous similar message [Tue Feb 5 20:37:23 2019][1650216.888982] Lustre: fir-MDT0001: Connection restored to 55a5f43f-2070-1278-9f4b-9a1845c8940d (at 10.8.11.22@o2ib6) [Tue Feb 5 20:37:23 2019][1650216.899507] Lustre: Skipped 1 previous similar message [Tue Feb 5 20:39:24 2019][1650337.927388] Lustre: fir-MDT0001: Connection restored to c62ed1f1-3793-7a2f-05b0-df79888e04df (at 10.8.0.65@o2ib6) [Tue Feb 5 20:39:24 2019][1650337.937825] Lustre: Skipped 1 previous similar message [Tue Feb 5 20:41:07 2019][1650440.878509] Lustre: fir-MDT0003: haven't heard from client f9688477-20ef-48d9-107d-c35a4ee3a4ab (at 10.9.0.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa054d7ae4800, cur 1549428243 expire 1549428093 last 1549428016 [Tue Feb 5 20:41:07 2019][1650440.900216] Lustre: Skipped 3 previous similar messages [Tue Feb 5 20:41:09 2019][1650442.845577] Lustre: fir-MDT0001: haven't heard from client f9688477-20ef-48d9-107d-c35a4ee3a4ab (at 10.9.0.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa067e5b93800, cur 1549428245 expire 1549428095 last 1549428018 [Tue Feb 5 20:44:30 2019][1650643.710900] Lustre: fir-MDT0001: Connection restored to 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) [Tue Feb 5 20:44:30 2019][1650643.721246] Lustre: Skipped 1 previous similar message [Tue Feb 5 21:02:32 2019][1651725.688126] Lustre: fir-MDT0001: Connection restored to 55a5f43f-2070-1278-9f4b-9a1845c8940d (at 10.8.11.22@o2ib6) [Tue Feb 5 21:02:32 2019][1651725.698649] Lustre: Skipped 1 previous similar message [Tue Feb 5 21:09:30 2019][1652143.628136] Lustre: fir-MDT0001: Connection restored to 81ce647f-387b-b748-de78-f885bffe2ce2 (at 10.8.3.33@o2ib6) [Tue Feb 5 21:09:30 2019][1652143.638574] Lustre: Skipped 1 previous similar message [Tue Feb 5 21:27:56 2019][1653249.974805] Lustre: fir-MDT0003: Connection restored to 55a5f43f-2070-1278-9f4b-9a1845c8940d (at 10.8.11.22@o2ib6) [Tue Feb 5 21:27:56 2019][1653249.985332] Lustre: Skipped 1 previous similar message [Tue Feb 5 21:40:22 2019][1653995.759386] Lustre: fir-MDT0001: Connection restored to 2df312f9-aff0-54ca-41b5-f915ec943155 (at 10.8.11.9@o2ib6) [Tue Feb 5 21:40:22 2019][1653995.769814] Lustre: Skipped 1 previous similar message [Tue Feb 5 21:45:02 2019][1654276.457739] Lustre: fir-MDT0001: Connection restored to 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6) [Tue Feb 5 21:45:02 2019][1654276.468179] Lustre: Skipped 1 previous similar message [Tue Feb 5 21:49:13 2019][1654526.839707] Lustre: fir-MDT0003: haven't heard from client 81ce647f-387b-b748-de78-f885bffe2ce2 (at 10.8.3.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa0783d0bc000, cur 1549432329 expire 1549432179 last 1549432102 [Tue Feb 5 21:51:05 2019][1654639.262381] Lustre: fir-MDT0001: Connection restored to 98df58b1-8a0c-b210-852f-29bcfcaceadc (at 10.9.103.40@o2ib4) [Tue Feb 5 21:51:05 2019][1654639.272993] Lustre: Skipped 1 previous similar message [Tue Feb 5 21:51:12 2019][1654645.860474] Lustre: fir-MDT0001: Connection restored to 276945c0-0d00-01d6-84a8-1f1c209c1b8f (at 10.9.103.36@o2ib4) [Tue Feb 5 21:51:12 2019][1654645.871085] Lustre: Skipped 1 previous similar message [Tue Feb 5 21:51:26 2019][1654660.482156] Lustre: fir-MDT0001: Connection restored to 302c6952-5b6c-1588-cbd4-2c54f063f559 (at 10.9.103.37@o2ib4) [Tue Feb 5 21:51:26 2019][1654660.492770] Lustre: Skipped 3 previous similar messages [Tue Feb 5 21:51:42 2019][1654676.409410] Lustre: fir-MDT0001: Connection restored to a550b456-cf1e-625d-c988-86c8427ac97c (at 10.9.103.38@o2ib4) [Tue Feb 5 21:51:42 2019][1654676.420018] Lustre: Skipped 1 previous similar message [Tue Feb 5 21:52:35 2019][1654728.916464] Lustre: fir-MDT0001: Connection restored to b34755b7-d181-850b-8754-b1f884364504 (at 10.9.103.44@o2ib4) [Tue Feb 5 21:52:35 2019][1654728.927073] Lustre: Skipped 3 previous similar messages [Tue Feb 5 21:53:25 2019][1654779.195136] Lustre: fir-MDT0001: Connection restored to 775ad82c-450c-c16e-bb21-700cda6d4afd (at 10.9.103.39@o2ib4) [Tue Feb 5 21:53:25 2019][1654779.205742] Lustre: Skipped 1 previous similar message [Tue Feb 5 21:55:13 2019][1654886.896243] Lustre: fir-MDT0001: Connection restored to 55a5f43f-2070-1278-9f4b-9a1845c8940d (at 10.8.11.22@o2ib6) [Tue Feb 5 21:55:13 2019][1654886.906769] Lustre: Skipped 3 previous similar messages [Tue Feb 5 21:57:22 2019][1655015.924736] Lustre: fir-MDT0001: Connection restored to 21210b5c-1bf9-2672-2025-9c6097d35430 (at 10.9.103.25@o2ib4) [Tue Feb 5 21:57:22 2019][1655015.935343] Lustre: Skipped 15 previous similar messages [Tue Feb 5 22:09:47 2019][1655761.563187] Lustre: fir-MDT0001: Connection restored to fa385709-7326-78a8-5385-8636f6dd528a (at 10.8.3.26@o2ib6) [Tue Feb 5 22:09:47 2019][1655761.573624] Lustre: Skipped 41 previous similar messages [Tue Feb 5 22:18:21 2019][1656275.069012] Lustre: fir-MDT0001: Connection restored to 59e5976b-e92e-fe58-4cc6-1a51bbca02ac (at 10.8.3.14@o2ib6) [Tue Feb 5 22:18:21 2019][1656275.079451] Lustre: Skipped 99 previous similar messages [Tue Feb 5 22:32:43 2019][1657136.725567] Lustre: fir-MDT0001: Connection restored to 77116eb9-21ab-2adf-64d6-a8cbcc3d83d7 (at 10.9.102.39@o2ib4) [Tue Feb 5 22:32:43 2019][1657136.736175] Lustre: Skipped 234 previous similar messages [Tue Feb 5 22:45:34 2019][1657908.440797] Lustre: fir-MDT0001: Connection restored to 449ab5a0-1671-c50c-bbbe-21371abb55d6 (at 10.9.107.56@o2ib4) [Tue Feb 5 22:45:34 2019][1657908.451401] Lustre: Skipped 63 previous similar messages [Tue Feb 5 22:59:09 2019][1658722.891565] Lustre: fir-MDT0001: Connection restored to 24d9d171-71c4-5c61-0044-18351be88bb7 (at 10.9.114.7@o2ib4) [Tue Feb 5 22:59:09 2019][1658722.902086] Lustre: Skipped 49 previous similar messages [Tue Feb 5 23:09:25 2019][1659338.878548] Lustre: fir-MDT0003: haven't heard from client 59e5976b-e92e-fe58-4cc6-1a51bbca02ac (at 10.8.3.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa05daeb4f800, cur 1549437141 expire 1549436991 last 1549436914 [Tue Feb 5 23:09:25 2019][1659338.900341] Lustre: Skipped 11 previous similar messages [Tue Feb 5 23:30:12 2019][1660585.841931] Lustre: fir-MDT0001: Connection restored to 6f23ad32-0dd1-26f7-1bbe-7fefdeb50a2a (at 10.8.19.1@o2ib6) [Tue Feb 5 23:30:12 2019][1660585.852369] Lustre: Skipped 55 previous similar messages [Tue Feb 5 23:42:26 2019][1661320.482070] Lustre: fir-MDT0003: Connection restored to 8ae4cc68-204d-d2bb-4908-94f49692744b (at 10.8.20.15@o2ib6) [Tue Feb 5 23:42:26 2019][1661320.492592] Lustre: Skipped 1 previous similar message [Tue Feb 5 23:47:00 2019][1661594.198275] Lustre: fir-MDT0001: Connection restored to 20d572d4-8f70-51c7-3e05-1d3dc78a02cc (at 10.8.27.16@o2ib6) [Tue Feb 5 23:47:00 2019][1661594.208798] Lustre: Skipped 1 previous similar message [Tue Feb 5 23:49:35 2019][1661749.582993] Lustre: fir-MDT0001: Connection restored to a4f2c4c1-03c5-819a-6852-1875c7d76a33 (at 10.8.19.7@o2ib6) [Tue Feb 5 23:49:35 2019][1661749.593439] Lustre: Skipped 1 previous similar message [Tue Feb 5 23:51:13 2019][1661847.640456] Lustre: fir-MDT0001: Connection restored to f1ecbb01-7b7c-667a-3a9f-343625d79564 (at 10.8.22.13@o2ib6) [Tue Feb 5 23:51:13 2019][1661847.650982] Lustre: Skipped 1 previous similar message [Tue Feb 5 23:52:29 2019][1661922.962138] Lustre: fir-MDT0001: Connection restored to 7a28562b-3b5d-09cc-c2f6-568559bac302 (at 10.8.18.15@o2ib6) [Tue Feb 5 23:52:29 2019][1661922.972689] Lustre: Skipped 104 previous similar messages [Tue Feb 5 23:55:06 2019][1662080.630828] Lustre: fir-MDT0001: Connection restored to 5d3c8a4e-536f-dbf4-56cc-734fa5187919 (at 10.8.3.6@o2ib6) [Tue Feb 5 23:55:06 2019][1662080.641183] Lustre: Skipped 196 previous similar messages [Tue Feb 5 23:58:48 2019][1662301.906139] Lustre: fir-MDT0001: haven't heard from client 225eea3d-b7b7-fb2f-9cfe-041d20b24f14 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa06729486800, cur 1549440104 expire 1549439954 last 1549439877 [Tue Feb 5 23:58:48 2019][1662301.928020] Lustre: Skipped 63 previous similar messages [Wed Feb 6 00:00:51 2019][1662425.421514] Lustre: fir-MDT0001: Connection restored to 4bb06e63-20b8-d623-0f05-97260407842f (at 10.8.20.10@o2ib6) [Wed Feb 6 00:00:51 2019][1662425.432039] Lustre: Skipped 34 previous similar messages [Wed Feb 6 00:21:29 2019][1663663.651217] LNetError: 129244:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.14.7@tcp from [Wed Feb 6 00:21:29 2019][1663663.661912] LNetError: 129244:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 4060429 previous similar messages [Wed Feb 6 00:21:29 2019][1663663.673268] Lustre: 129244:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549441465/real 1549441465] req@ffffa06237ce9500 x1623667885921344/t0(0) o104->fir-MDT0001@10.10.14.7@tcp:15/16 lens 296/224 e 0 to 1 dl 1549441476 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 [Wed Feb 6 00:21:29 2019][1663663.700974] Lustre: 129244:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message [Wed Feb 6 00:21:31 2019][1663665.673266] Lustre: 129244:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549441467/real 1549441467] req@ffffa06237ce9500 x1623667885921344/t0(0) o104->fir-MDT0001@10.10.14.7@tcp:15/16 lens 296/224 e 0 to 1 dl 1549441478 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:21:31 2019][1663665.700953] Lustre: 129244:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 266387 previous similar messages [Wed Feb 6 00:21:35 2019][1663669.673290] Lustre: 129244:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549441471/real 1549441471] req@ffffa06237ce9500 x1623667885921344/t0(0) o104->fir-MDT0001@10.10.14.7@tcp:15/16 lens 296/224 e 0 to 1 dl 1549441482 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:21:35 2019][1663669.700968] Lustre: 129244:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 664355 previous similar messages [Wed Feb 6 00:21:43 2019][1663677.673362] Lustre: 126626:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549441479/real 1549441479] req@ffffa0680f90b900 x1623667885946512/t0(0) o104->fir-MDT0001@10.10.14.7@tcp:15/16 lens 296/224 e 0 to 1 dl 1549441490 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:21:43 2019][1663677.701049] Lustre: 126626:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1293765 previous similar messages [Wed Feb 6 00:21:45 2019][1663679.651370] LNetError: 126626:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.14.7@tcp from [Wed Feb 6 00:21:45 2019][1663679.662061] LNetError: 126626:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 2540551 previous similar messages [Wed Feb 6 00:21:59 2019][1663693.673497] Lustre: 129244:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549441495/real 1549441495] req@ffffa06237ce9500 x1623667885921344/t0(0) o104->fir-MDT0001@10.10.14.7@tcp:15/16 lens 296/224 e 0 to 1 dl 1549441506 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:21:59 2019][1663693.701179] Lustre: 129244:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2587407 previous similar messages [Wed Feb 6 00:22:17 2019][1663711.651627] LNetError: 126626:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.14.7@tcp from [Wed Feb 6 00:22:17 2019][1663711.662322] LNetError: 126626:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 6094166 previous similar messages [Wed Feb 6 00:22:31 2019][1663725.673746] Lustre: 129244:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549441527/real 1549441527] req@ffffa06237ce9500 x1623667885921344/t0(0) o104->fir-MDT0001@10.10.14.7@tcp:15/16 lens 296/224 e 0 to 1 dl 1549441538 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:22:31 2019][1663725.701428] Lustre: 129244:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7409589 previous similar messages [Wed Feb 6 00:23:21 2019][1663775.652141] LNetError: 127390:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.11@tcp from [Wed Feb 6 00:23:21 2019][1663775.663002] LNetError: 127390:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 16321045 previous similar messages [Wed Feb 6 00:23:28 2019][1663782.292284] Lustre: fir-MDT0001: Connection restored to 8ae4cc68-204d-d2bb-4908-94f49692744b (at 10.8.20.15@o2ib6) [Wed Feb 6 00:23:28 2019][1663782.302810] Lustre: Skipped 5 previous similar messages [Wed Feb 6 00:23:35 2019][1663789.674263] Lustre: 126626:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549441591/real 1549441591] req@ffffa0680f90a100 x1623667885946496/t0(0) o104->fir-MDT0001@10.10.14.7@tcp:15/16 lens 296/224 e 0 to 1 dl 1549441602 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:23:35 2019][1663789.701951] Lustre: 126626:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 16345309 previous similar messages [Wed Feb 6 00:24:00 2019][1663814.256473] LustreError: 129244:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.14.7@tcp) failed to reply to blocking AST (req@ffffa06237ce9500 x1623667885921344 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffffa05d53ea8d80/0x8c1de5393f13def3 lrc: 4/0,0 mode: PR/PR res: [0x240000406:0x382:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.14.7@tcp remote: 0xbeac4b43c26ef6d4 expref: 19 pid: 129249 timeout: 1663949 lvb_type: 0 [Wed Feb 6 00:24:00 2019][1663814.299150] LustreError: 138-a: fir-MDT0001: A client on nid 10.10.14.7@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:24:00 2019][1663814.311693] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.14.7@tcp ns: mdt-fir-MDT0001_UUID lock: ffffa05d53ea8d80/0x8c1de5393f13def3 lrc: 3/0,0 mode: PR/PR res: [0x240000406:0x382:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.14.7@tcp remote: 0xbeac4b43c26ef6d4 expref: 20 pid: 129249 timeout: 0 lvb_type: 0 [Wed Feb 6 00:24:00 2019][1663814.349210] LustreError: 126626:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa0637cd04000 ns: mdt-fir-MDT0001_UUID lock: ffffa06388247bc0/0x8c1de5393f1687eb lrc: 1/0,0 mode: EX/EX res: [0x240000cb1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.14.7@tcp remote: 0xbeac4b43c26efcca expref: 4 pid: 126626 timeout: 0 lvb_type: 3 [Wed Feb 6 00:24:50 2019][1663864.719872] LNet: Service thread pid 127411 was inactive for 200.09s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:24:50 2019][1663864.737064] Pid: 127411, comm: mdt03_011 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:24:50 2019][1663864.747067] Call Trace: [Wed Feb 6 00:24:50 2019][1663864.749699] [] 0xffffffffffffffff [Wed Feb 6 00:24:51 2019][1663864.754901] LustreError: dumping log to /tmp/lustre-log.1549441666.127411 [Wed Feb 6 00:24:51 2019][1663865.231876] LNet: Service thread pid 128092 was inactive for 200.45s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:24:51 2019][1663865.249075] Pid: 128092, comm: mdt00_013 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:24:51 2019][1663865.259072] Call Trace: [Wed Feb 6 00:24:51 2019][1663865.261708] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.268829] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.276109] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.282955] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:24:51 2019][1663865.289691] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:24:51 2019][1663865.296085] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:24:51 2019][1663865.302394] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:24:51 2019][1663865.309831] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:24:51 2019][1663865.316746] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:24:51 2019][1663865.324965] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:24:51 2019][1663865.331358] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:24:51 2019][1663865.337572] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:24:51 2019][1663865.344309] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:24:51 2019][1663865.350176] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.357299] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.365189] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.371685] [] kthread+0xd1/0xe0 [Wed Feb 6 00:24:51 2019][1663865.376773] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:24:51 2019][1663865.383422] [] 0xffffffffffffffff [Wed Feb 6 00:24:51 2019][1663865.388615] LustreError: dumping log to /tmp/lustre-log.1549441667.128092 [Wed Feb 6 00:24:51 2019][1663865.506477] Pid: 129260, comm: mdt02_017 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:24:51 2019][1663865.516477] Call Trace: [Wed Feb 6 00:24:51 2019][1663865.519122] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.526243] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.533523] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.540369] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:24:51 2019][1663865.547110] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:24:51 2019][1663865.553505] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:24:51 2019][1663865.559816] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:24:51 2019][1663865.567252] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:24:51 2019][1663865.574160] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:24:51 2019][1663865.582376] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:24:51 2019][1663865.588772] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:24:51 2019][1663865.594984] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:24:51 2019][1663865.601729] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:24:51 2019][1663865.607606] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.614720] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.622605] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.629110] [] kthread+0xd1/0xe0 [Wed Feb 6 00:24:51 2019][1663865.634195] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:24:51 2019][1663865.640842] [] 0xffffffffffffffff [Wed Feb 6 00:24:51 2019][1663865.646039] Pid: 127136, comm: mdt00_005 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:24:51 2019][1663865.656036] Call Trace: [Wed Feb 6 00:24:51 2019][1663865.658667] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.665775] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.673058] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.679894] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:24:51 2019][1663865.686628] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:24:51 2019][1663865.693025] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:24:51 2019][1663865.699324] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:24:51 2019][1663865.706760] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:24:51 2019][1663865.713671] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:24:51 2019][1663865.721886] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:24:51 2019][1663865.728282] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:24:51 2019][1663865.734504] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:24:51 2019][1663865.741238] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:24:51 2019][1663865.747107] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.754221] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.762109] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663865.768607] [] kthread+0xd1/0xe0 [Wed Feb 6 00:24:51 2019][1663865.773695] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:24:52 2019][1663865.780336] [] 0xffffffffffffffff [Wed Feb 6 00:24:52 2019][1663865.785511] Pid: 128085, comm: mdt00_012 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:24:52 2019][1663865.795511] Call Trace: [Wed Feb 6 00:24:52 2019][1663865.798142] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 00:24:52 2019][1663865.805242] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:24:52 2019][1663865.812522] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:24:52 2019][1663865.819369] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:24:52 2019][1663865.826105] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:24:52 2019][1663865.832499] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:24:52 2019][1663865.838800] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:24:52 2019][1663865.846236] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:24:52 2019][1663865.853143] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:24:52 2019][1663865.861361] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:24:52 2019][1663865.867755] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:24:52 2019][1663865.873979] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:24:52 2019][1663865.880712] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:24:52 2019][1663865.886588] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:24:52 2019][1663865.893696] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:24:52 2019][1663865.901581] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:24:52 2019][1663865.908081] [] kthread+0xd1/0xe0 [Wed Feb 6 00:24:52 2019][1663865.913159] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:24:52 2019][1663865.919799] [] 0xffffffffffffffff [Wed Feb 6 00:24:52 2019][1663865.924978] LNet: Service thread pid 129267 was inactive for 200.75s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:24:52 2019][1663866.256879] LustreError: dumping log to /tmp/lustre-log.1549441668.127196 [Wed Feb 6 00:24:52 2019][1663866.767883] LNet: Service thread pid 127066 was inactive for 200.55s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:24:52 2019][1663866.781007] LNet: Skipped 7 previous similar messages [Wed Feb 6 00:24:54 2019][1663867.792889] LNet: Service thread pid 127205 was inactive for 200.74s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:24:54 2019][1663867.806005] LNet: Skipped 3 previous similar messages [Wed Feb 6 00:24:54 2019][1663867.811234] LustreError: dumping log to /tmp/lustre-log.1549441669.127205 [Wed Feb 6 00:24:55 2019][1663868.815906] LustreError: dumping log to /tmp/lustre-log.1549441670.126631 [Wed Feb 6 00:24:59 2019][1663872.911939] LNet: Service thread pid 129373 was inactive for 200.35s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:24:59 2019][1663872.925059] LNet: Skipped 3 previous similar messages [Wed Feb 6 00:24:59 2019][1663872.930296] LustreError: dumping log to /tmp/lustre-log.1549441675.129373 [Wed Feb 6 00:25:01 2019][1663874.959952] LustreError: dumping log to /tmp/lustre-log.1549441677.14718 [Wed Feb 6 00:25:02 2019][1663875.983968] LustreError: dumping log to /tmp/lustre-log.1549441678.129265 [Wed Feb 6 00:25:03 2019][1663877.007969] LNet: Service thread pid 127063 was inactive for 200.54s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:25:03 2019][1663877.021086] LNet: Skipped 3 previous similar messages [Wed Feb 6 00:25:03 2019][1663877.026323] LustreError: dumping log to /tmp/lustre-log.1549441679.127063 [Wed Feb 6 00:25:05 2019][1663879.567994] LustreError: dumping log to /tmp/lustre-log.1549441681.129266 [Wed Feb 6 00:25:10 2019][1663884.176028] LustreError: dumping log to /tmp/lustre-log.1549441686.127317 [Wed Feb 6 00:25:11 2019][1663885.200034] LNet: Service thread pid 127248 was inactive for 200.15s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:25:11 2019][1663885.213153] LNet: Skipped 4 previous similar messages [Wed Feb 6 00:25:11 2019][1663885.218382] LustreError: dumping log to /tmp/lustre-log.1549441687.127248 [Wed Feb 6 00:25:12 2019][1663886.224039] LustreError: dumping log to /tmp/lustre-log.1549441688.127197 [Wed Feb 6 00:25:21 2019][1663895.440121] LustreError: dumping log to /tmp/lustre-log.1549441697.129278 [Wed Feb 6 00:25:23 2019][1663897.489135] LustreError: dumping log to /tmp/lustre-log.1549441699.129374 [Wed Feb 6 00:25:26 2019][1663900.560158] LustreError: dumping log to /tmp/lustre-log.1549441702.129371 [Wed Feb 6 00:25:27 2019][1663901.584166] LNet: Service thread pid 129797 was inactive for 200.42s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:25:27 2019][1663901.597282] LNet: Skipped 6 previous similar messages [Wed Feb 6 00:25:27 2019][1663901.602515] LustreError: dumping log to /tmp/lustre-log.1549441703.129797 [Wed Feb 6 00:25:28 2019][1663902.609173] LustreError: dumping log to /tmp/lustre-log.1549441704.129253 [Wed Feb 6 00:25:29 2019][1663903.653178] LNetError: 127411:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.115.6@tcp from [Wed Feb 6 00:25:29 2019][1663903.663955] LNetError: 127411:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 35152155 previous similar messages [Wed Feb 6 00:25:30 2019][1663904.656194] LustreError: dumping log to /tmp/lustre-log.1549441706.128538 [Wed Feb 6 00:25:33 2019][1663907.728220] LustreError: dumping log to /tmp/lustre-log.1549441709.127247 [Wed Feb 6 00:25:43 2019][1663917.675287] Lustre: 127134:0:(client.c:2132:ptlrpc_expire_one_requed due to network error: [sent 1549441719/real 1549441719] req@ffffa0596e36fb00 x1623667886056768/t0(0) o104->fir-MDT0003@10.10.114.11@tcp:15/16 lens 296/224 e 0 to 1 dl 1549441730 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:25:43 2019][1663917.703144] Lustre: 127134:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 35722551 previous similar messages [Wed Feb 6 00:25:59 2019][1663933.328425] LustreError: dumping log to /tmp/lustre-log.1549441735.129262 [Wed Feb 6 00:26:01 2019][1663935.376443] LNet: Service thread pid 129252 was inactive for 200.41s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:26:01 2019][1663935.389559] LNet: Skipped 4 previous similar messages [Wed Feb 6 00:26:01 2019][1663935.394786] LustreError: dumping log to /tmp/lustre-log.1549441737.129252 [Wed Feb 6 00:26:02 2019][1663936.400450] LustreError: dumping log to /tmp/lustre-log.1549441738.127198 [Wed Feb 6 00:26:03 2019][1663937.424469] LustreError: dumping log to /tmp/lustre-log.1549441739.2937 [Wed Feb 6 00:26:07 2019][1663941.008488] LustreError: dumping log to /tmp/lustre-log.1549441743.128267 [Wed Feb 6 00:26:30 2019][1663964.753677] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:26:30 2019][1663964.769832] Lustre: Skipped 7 previous similar messages [Wed Feb 6 00:26:30 2019][1663964.775254] LustreError: 129260:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441466, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa073ea646780/0x8c1de5393f15d270 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 76 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935180444abd expref: -99 pid: 129260 timeout: 0 lvb_type: 0 [Wed Feb 6 00:26:31 2019][1663964.780860] Lustre: fir-MDT0000-osp-MDT0003: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [Wed Feb 6 00:26:31 2019][1663964.780862] Lustre: Skipped 1 previous similar message [Wed Feb 6 00:26:31 2019][1663964.830859] LustreError: 129260:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Feb 6 00:26:31 2019][1663965.257684] LustreError: 129251:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.14.7@tcp) failed to reply to blocking AST (req@ffffa05ad8f5f200 x1623667886036864 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffffa0702a1460c0/0x8c1de5393f666ff8 lrc: 4/0,0 mode: PR/PR res: [0x240000cc8:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.14.7@tcp remote: 0xbeac4b43c26efd3a expref: 14 pid: 127501 timeout: 1664100 lvb_type: 0 [Wed Feb 6 00:26:31 2019][1663965.300185] LustreError: 138-a: fir-MDT0001: A client on nid 10.10.14.7@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:26:31 2019][1663965.312720] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.14.7@tcp ns: mdt-fir-MDT0001_UUID lock: ffffa0702a1460c0/0x8c1de5393f666ff8 lrc: 3/0,0 mode: PR/PR res: [0x240000cc8:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.14.7@tcp remote: 0xbeac4b43c26efd3a expref: 15 pid: 127501 timeout: 0 lvb_type: 0 [Wed Feb 6 00:26:31 2019][1663965.349870] LustreError: 129251:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa066e55c1400 ns: mdt-fir-MDT0001_UUID lock: ffffa059a5f17080/0x8c1de5393f667014 lrc: 1/0,0 mode: --/PW res: [0x240000cc8:0x1:0x0].0x0 bits 0x40/0x0 rrc: 3 type: IBT flags: 0x54a01000000000 nid: 10.10.14.7@tcp remote: 0xbeac4b43c26efd48 expref: 9 pid: 129251 timeout: 0 lvb_type: 0 [Wed Feb 6 00:26:31 2019][1663965.388691] LustreError: 127212:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441467, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa065469ae540/0x8c1de5393f164a88 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 76 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518044fb4c expref: -99 pid: 127212 timeout: 0 lvb_type: 0 [Wed Feb 6 00:26:31 2019][1663965.429111] LustreError: 127212:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 7 previous similar messages [Wed Feb 6 00:26:32 2019][1663966.257695] LustreError: 127411:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.114.15@tcp) failed to reply to blocking AST (req@ffffa086d7f26000 x1623667886038048 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffffa0548f6c3840/0x8c1de5393f66d793 lrc: 4/0,0 mode: PR/PR res: [0x240000406:0x4f2:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.114.15@tcp remote: 0x5a89b021a3479c79 expref: 6 pid: 129276 timeout: 1664101 lvb_type: 0 [Wed Feb 6 00:26:32 2019][1663966.300633] LustreError: 138-a: fir-MDT0001: A client on nid 10.10.114.15@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:26:32 2019][1663966.313686] LNet: Service thread pid 127411 completed after 301.68s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 00:26:32 2019][1663966.330098] LNet: Skipped 10 previous similar messages [Wed Feb 6 00:27:09 2019][1664003.258010] LustreError: 127456:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.114.11@tcp) failed to reply to blocking AST (req@ffffa086d1ad1800 x1623667886051584 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa089d33c9f80/0x8c1de5393f7b6bad lrc: 4/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 55 type: IBT flags: 0x60200400000020 nid: 10.10.114.11@tcp remote: 0xdbb126caab60e599 expref: 14 pid: 127130 timeout: 1664138 lvb_type: 0 [Wed Feb 6 00:27:09 2019][1664003.301039] LustreError: 127456:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message [Wed Feb 6 00:27:09 2019][1664003.311308] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.114.11@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:27:09 2019][1664003.324004] LustreError: Skipped 1 previous similar message [Wed Feb 6 00:27:09 2019][1664003.329778] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.114.11@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa089d33c9f80/0x8c1de5393f7b6bad lrc: 3/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 55 type: IBT flags: 0x60200400000020 nid: 10.10.114.11@tcp remote: 0xdbb126caab60e599 expref: 15 pid: 127130 timeout: 0 lvb_type: 0 [Wed Feb 6 00:27:09 2019][1664003.367340] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages [Wed Feb 6 00:27:09 2019][1664003.377629] LustreError: 127134:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa089f07ecc00 ns: mdt-fir-MDT0003_UUID lock: ffffa05901a66540/0x8c1de5393f81a871 lrc: 1/0,0 mode: EX/EX res: [0x280000d67:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.114.11@tcp remote: 0xdbb126caab60e5d1 expref: 4 pid: 127134 timeout: 0 lvb_type: 3 [Wed Feb 6 00:29:01 2019][1664115.254899] LustreError: 126626:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441617, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa060628de9c0/0x8c1de5393f66de46 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 95 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935180490427 expref: -99 pid: 126626 timeout: 0 lvb_type: 0 [Wed Feb 6 00:29:01 2019][1664115.295327] LustreError: 126626:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 4 previous similar messages [Wed Feb 6 00:29:02 2019][1664116.258908] LustreError: 126627:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.14.7@tcp) failed to reply to blocking AST (req@ffffa066e2079800 x1623667886105568 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffffa0682b6bc140/0x8c1de5393f94bea9 lrc: 4/0,0 mode: PR/PR res: [0x240000cb1:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.14.7@tcp remote: 0xbeac4b43c26efd6b expref: 12 pid: 126627 timeout: 1664251 lvb_type: 0 [Wed Feb 6 00:29:02 2019][1664116.301409] LustreError: 138-a: fir-MDT0001: A client on nid 10.10.14.7@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:29:02 2019][1664116.313947] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.14.7@tcp ns: mdt-fir-MDT0001_UUID lock: ffffa0682b6bc140/0x8c1de5393f94bea9 lrc: 3/0,0 mode: PR/PR res: [0x240000cb1:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.14.7@tcp remote: 0xbeac4b43c26efd6b expref: 13 pid: 126627 timeout: 0 lvb_type: 0 [Wed Feb 6 00:29:02 2019][1664116.351124] LustreError: 126627:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa077ef1ef800 ns: mdt-fir-MDT0001_UUID lock: ffffa0682b6bee40/0x8c1de5393f94bee1 lrc: 3/0,0 mode: PW/PW res: [0x240000cb1:0x5:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x50200000000000 nid: 10.10.14.7@tcp remote: 0xbeac4b43c26efd79 expref: 3 pid: 126627 timeout: 0 lvb_type: 0 [Wed Feb 6 00:29:02 2019][1664116.414647] Lustre: fir-MDT0001: Connection restored to 82eeb05c-9f6a-d608-7f3e-29db42f96cfa (at 10.10.14.7@tcp) [Wed Feb 6 00:29:02 2019][1664116.424994] Lustre: Skipped 5 previous similar messages [Wed Feb 6 00:29:14 2019][1664128.020003] LustreError: 17828:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441630, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa089bd35d100/0x8c1de5393f6d682d lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 100 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351804917b4 expref: -99 pid: 17828 timeout: 0 lvb_type: 0 [Wed Feb 6 00:29:14 2019][1664128.060341] LustreError: 17828:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 2 previous similar messages [Wed Feb 6 00:29:18 2019][1664132.697039] LustreError: 17838:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441634, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa0705cbc8900/0x8c1de5393f6fbcb5 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 100 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351804919f2 expref: -99 pid: 17838 timeout: 0 lvb_type: 0 [Wed Feb 6 00:29:36 2019][1664150.669181] LustreError: 127297:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441652, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa070042b7080/0x8c1de5393f7a5773 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 105 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935180495a03 expref: -99 pid: 127297 timeout: 0 lvb_type: 0 [Wed Feb 6 00:29:38 2019][1664152.466190] LNet: Service thread pid 126628 was inactive for 337.03s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:29:38 2019][1664152.479306] LNet: Skipped 4 previous similar messages [Wed Feb 6 00:29:38 2019][1664152.484536] LustreError: dumping log to /tmp/lustre-log.1549441954.126628 [Wed Feb 6 00:29:39 2019][1664153.490198] LustreError: dumping log to /tmp/lustre-log.1549441955.17839 [Wed Feb 6 00:29:40 2019][1664154.259212] LustreError: 129255:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.112.8@tcp) failed to reply to blocking AST (req@ffffa063ae665d00 x1623667886185616 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa064331a18c0/0x8c1de5393f9bba03 lrc: 4/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 63 type: IBT flags: 0x60200400000020 nid: 10.10.112.8@tcp remote: 0x8ea3e4d27142f257 expref: 9 pid: 126629 timeout: 1664289 lvb_type: 0 [Wed Feb 6 00:29:40 2019][1664154.301982] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.112.8@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:29:40 2019][1664154.314613] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.8@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa064331a18c0/0x8c1de5393f9bba03 lrc: 3/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 63 type: IBT flags: 0x60200400000020 nid: 10.10.112.8@tcp remote: 0x8ea3e4d27142f257 expref: 10 pid: 126629 timeout: 0 lvb_type: 0 [Wed Feb 6 00:29:45 2019][1664159.655249] LNetError: 129595:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.115.2@tcp from [Wed Feb 6 00:29:45 2019][1664159.666030] LNetError: 129595:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 56032007 previous similar messages [Wed Feb 6 00:29:56 2019][1664169.843344] LustreError: 17860:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441671, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa059f2839d40/0x8c1de5393f85baad lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 109 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935180495e94 expref: -99 pid: 17860 timeout: 0 lvb_type: 0 [Wed Feb 6 00:29:56 2019][1664169.883687] LustreError: 17860:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Feb 6 00:29:59 2019][1664173.677355] Lustre: 127198:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549441975/real 1549441975] req@ffffa086d5ac2400 x1623667886271696/t0(0) o104->fir-MDT0001@10.10.14.7@tcp:15/16 lens 296/224 e 0 to 1 dl 1549441986 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:29:59 2019][1664173.705039] Lustre: 127198:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 55038953 previous similar messages [Wed Feb 6 00:30:17 2019][1664191.378506] LNet: Service thread pid 17828 was inactive for 363.35s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:30:17 2019][1664191.395617] LNet: Skipped 3 previous similar messages [Wed Feb 6 00:30:17 2019][1664191.400852] Pid: 17828, comm: mdt03_022 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:30:17 2019][1664191.410762] Call Trace: [Wed Feb 6 00:30:17 2019][1664191.413400] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:30:17 2019][1664191.420510] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:30:17 2019][1664191.427791] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:30:17 2019][1664191.434636] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:30:17 2019][1664191.441370] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:30:17 2019][1664191.447770] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:30:17 2019][1664191.454076] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:30:17 2019][1664191.461529] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:30:17 2019][1664191.468438] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:30:17 2019][1664191.476653] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:30:17 2019][1664191.483050] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:30:17 2019][1664191.489273] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:30:17 2019][1664191.496016] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:30:17 2019][1664191.501892] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:30:17 2019][1664191.509016] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:30:17 2019][1664191.516901] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:30:17 2019][1664191.523425] [] kthread+0xd1/0xe0 [Wed Feb 6 00:30:17 2019][1664191.528517] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:30:17 2019][1664191.535162] [] 0xffffffffffffffff [Wed Feb 6 00:30:17 2019][1664191.540357] LustreError: dumping log to /tmp/lustre-log.1549441993.17828 [Wed Feb 6 00:30:37 2019][1664211.054670] LustreError: 17827:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441713, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa0896a393cc0/0x8c1de5393f91d4a4 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 110 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935180496cf6 expref: -99 pid: 17827 timeout: 0 lvb_type: 0 [Wed Feb 6 00:30:37 2019][1664211.095003] LustreError: 17827:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 5 previous similar messages [Wed Feb 6 00:30:46 2019][1664220.050736] LNet: Service thread pid 17838 was inactive for 387.35s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:30:46 2019][1664220.067841] Pid: 17838, comm: mdt02_029 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:30:46 2019][1664220.077754] Call Trace: [Wed Feb 6 00:30:46 2019][1664220.080414] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:30:46 2019][1664220.087529] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:30:46 2019][1664220.094828] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:30:46 2019][1664220.101675] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:30:46 2019][1664220.108432] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:30:46 2019][1664220.114832] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:30:46 2019][1664220.121162] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:30:46 2019][1664220.128603] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:30:46 2019][1664220.135541] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:30:46 2019][1664220.143760] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:30:46 2019][1664220.150161] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:30:46 2019][1664220.156380] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:30:46 2019][1664220.163121] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:30:46 2019][1664220.169004] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:30:46 2019][1664220.176131] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:30:46 2019][1664220.184039] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:30:46 2019][1664220.190544] [] kthread+0xd1/0xe0 [Wed Feb 6 00:30:46 2019][1664220.195655] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:30:46 2019][1664220.202305] [] 0xffffffffffffffff [Wed Feb 6 00:30:46 2019][1664220.207519] LustreError: dumping log to /tmp/lustre-log.1549442022.17838 [Wed Feb 6 00:31:03 2019][1664237.458878] LNet: Service thread pid 127297 was inactive for 386.78s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:31:03 2019][1664237.476071] Pid: 127297, comm: mdt02_010 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:31:03 2019][1664237.486072] Call Trace: [Wed Feb 6 00:31:03 2019][1664237.488727] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:31:03 2019][1664237.495849] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:31:03 2019][1664237.503146] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:31:03 2019][1664237.509999] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:31:03 2019][1664237.516757] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:31:03 2019][1664237.523156] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:31:03 2019][1664237.529479] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:31:03 2019][1664237.536919] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:31:03 2019][1664237.543848] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:31:03 2019][1664237.552069] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:31:03 2019][1664237.558477] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:31:03 2019][1664237.564705] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:31:03 2019][1664237.571445] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:31:03 2019][1664237.577336] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:31:03 2019][1664237.584458] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:31:03 2019][1664237.592366] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:31:03 2019][1664237.598873] [] kthread+0xd1/0xe0 [Wed Feb 6 00:31:03 2019][1664237.603979] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:31:03 2019][1664237.610628] [] 0xffffffffffffffff [Wed Feb 6 00:31:03 2019][1664237.615839] LustreError: dumping log to /tmp/lustre-log.1549442039.127297 [Wed Feb 6 00:31:32 2019][1664266.167111] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:31:32 2019][1664266.183264] Lustre: Skipped 1 previous similar message [Wed Feb 6 00:31:33 2019][1664267.260121] LustreError: 127198:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.14.7@tcp) failed to reply to blocking AST (req@ffffa086d5ac2400 x1623667886271696 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffffa053af9a8b40/0x8c1de5393fa81f5d lrc: 4/0,0 mode: PR/PR res: [0x240000cb1:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.10.14.7@tcp remote: 0xbeac4b43c26efdbf expref: 12 pid: 129259 timeout: 1664402 lvb_type: 0 [Wed Feb 6 00:31:33 2019][1664267.302625] LustreError: 138-a: fir-MDT0001: A client on nid 10.10.14.7@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:31:33 2019][1664267.315162] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.14.7@tcp ns: mdt-fir-MDT0001_UUID lock: ffffa053af9a8b40/0x8c1de5393fa81f5d lrc: 3/0,0 mode: PR/PR res: [0x240000cb1:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.10.14.7@tcp remote: 0xbeac4b43c26efdbf expref: 13 pid: 129259 timeout: 0 lvb_type: 0 [Wed Feb 6 00:31:33 2019][1664267.352352] LustreError: 127198:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa065aec6ec00 ns: mdt-fir-MDT0001_UUID lock: ffffa07237f56540/0x8c1de5393fa81f79 lrc: 3/0,0 mode: PW/PW res: [0x240000cb1:0x5:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT flags: 0x50200000000000 nid: 10.10.14.7@tcp remote: 0xbeac4b43c26efdcd expref: 3 pid: 127198 timeout: 0 lvb_type: 0 [Wed Feb 6 00:31:45 2019][1664278.860216] LustreError: 17908:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441780, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa0736dac7500/0x8c1de5393f980ab5 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 113 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351804c22e4 expref: -99 pid: 17908 timeout: 0 lvb_type: 0 [Wed Feb 6 00:31:45 2019][1664278.900549] LustreError: 17908:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 44 previous similar messages [Wed Feb 6 00:32:04 2019][1664297.875361] LNet: Service thread pid 17829 was inactive for 437.05s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:32:04 2019][1664297.892464] Pid: 17829, comm: mdt03_023 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:32:04 2019][1664297.902370] Call Trace: [Wed Feb 6 00:32:04 2019][1664297.905005] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:32:04 2019][1664297.912117] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:32:04 2019][1664297.919398] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:32:04 2019][1664297.926259] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:32:04 2019][1664297.933003] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:32:04 2019][1664297.939401] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:32:04 2019][1664297.945709] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:32:04 2019][1664297.953146] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:32:04 2019][1664297.960053] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:32:04 2019][1664297.968272] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:32:04 2019][1664297.974670] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:32:04 2019][1664297.980897] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:32:04 2019][1664297.987633] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:32:04 2019][1664297.993522] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:32:04 2019][1664298.000641] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:32:04 2019][1664298.008530] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:32:04 2019][1664298.015027] [] kthread+0xd1/0xe0 [Wed Feb 6 00:32:04 2019][1664298.020116] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:32:04 2019][1664298.026762] [] 0xffffffffffffffff [Wed Feb 6 00:32:04 2019][1664298.031966] LustreError: dumping log to /tmp/lustre-log.1549442100.17829 [Wed Feb 6 00:32:12 2019][1664306.260437] LustreError: 129595:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.115.2@tcp) failed to reply to blocking AST (req@ffffa07647cc2400 x1623667886302816 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa05555922ac0/0x8c1de5393f9bbace lrc: 4/0,0 mode: PR/PR res: [0x280000d7d:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.115.2@tcp remote: 0xb8e1c27d85cfe834 expref: 14 pid: 128084 timeout: 1664441 lvb_type: 0 [Wed Feb 6 00:32:12 2019][1664306.303117] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.115.2@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:32:12 2019][1664306.315741] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.115.2@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa05555922ac0/0x8c1de5393f9bbace lrc: 3/0,0 mode: PR/PR res: [0x280000d7d:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.115.2@tcp remote: 0xb8e1c27d85cfe834 expref: 15 pid: 128084 timeout: 0 lvb_type: 0 [Wed Feb 6 00:32:12 2019][1664306.353304] LustreError: 129595:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa0887c335000 ns: mdt-fir-MDT0003_UUID lock: ffffa06f1d303600/0x8c1de5393faab84f lrc: 1/0,0 mode: EX/EX res: [0x280000d7d:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.115.2@tcp remote: 0xb8e1c27d85cfe8ab expref: 4 pid: 129595 timeout: 0 lvb_type: 3 [Wed Feb 6 00:32:19 2019][1664313.260556] LustreError: 17939:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa04efc3e8600 x1623667886418256/t0(0) o104->fir-MDT0003@10.10.114.11@tcp:15/16 lens 296/224 e 0 to 1 dl 1549442126 ref 1 fl Rpc:eX/2/ffffffff rc -5/-1 [Wed Feb 6 00:32:38 2019][1664332.691649] LNet: Service thread pid 17860 was inactive for 462.84s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:32:38 2019][1664332.708755] Pid: 17860, comm: mdt00_027 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:32:38 2019][1664332.718663] Call Trace: [Wed Feb 6 00:32:38 2019][1664332.721304] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:32:38 2019][1664332.728417] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:32:38 2019][1664332.735707] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:32:38 2019][1664332.742543] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:32:38 2019][1664332.749301] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:32:38 2019][1664332.755692] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:32:38 2019][1664332.762016] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:32:38 2019][1664332.769456] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:32:38 2019][1664332.776377] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:32:39 2019][1664332.784596] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:32:39 2019][1664332.791007] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:32:39 2019][1664332.797224] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:32:39 2019][1664332.803981] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:32:39 2019][1664332.809852] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:32:39 2019][1664332.816981] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:32:39 2019][1664332.824871] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:32:39 2019][1664332.831383] [] kthread+0xd1/0xe0 [Wed Feb 6 00:32:39 2019][1664332.836477] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:32:39 2019][1664332.843126] [] 0xffffffffffffffff [Wed Feb 6 00:32:39 2019][1664332.848315] LustreError: dumping log to /tmp/lustre-log.1549442134.17860 [Wed Feb 6 00:32:44 2019][1664337.811685] LNet: Service thread pid 129256 was inactive for 462.97s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:32:44 2019][1664337.824807] LNet: Skipped 2 previous similar messages [Wed Feb 6 00:32:44 2019][1664337.830045] LustreError: dumping log to /tmp/lustre-log.1549442139.129256 [Wed Feb 6 00:33:11 2019][1664365.459915] LustreError: dumping log to /tmp/lustre-log.1549442167.17875 [Wed Feb 6 00:33:19 2019][1664372.943382] Lustre: fir-MDT0003: haven't heard from client 5b464c0c-042f-5769-56f1-e3068a3e016c (at 10.9.105.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa05854eea800, cur 1549442175 expire 1549442025 last 1549441948 [Wed Feb 6 00:33:19 2019][1664372.965342] Lustre: Skipped 1 previous similar message [Wed Feb 6 00:33:19 2019][1664372.972103] LNet: Service thread pid 126626 completed after 557.71s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 00:33:19 2019][1664372.988516] LNet: Skipped 35 previous similar messages [Wed Feb 6 00:33:21 2019][1664375.699991] LustreError: dumping log to /tmp/lustre-log.1549442177.17848 [Wed Feb 6 00:33:29 2019][1664382.938969] LNet: Service thread pid 17828 completed after 554.91s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 00:33:29 2019][1664382.955306] LNet: Skipped 7 previous similar messages [Wed Feb 6 00:33:29 2019][1664383.011416] Lustre: fir-MDT0001: haven't heard from client 5b464c0c-042f-5769-56f1-e3068a3e016c (at 10.9.105.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa087e473e800, cur 1549442185 expire 1549442035 last 1549441958 [Wed Feb 6 00:33:29 2019][1664383.033378] Lustre: Skipped 1 previous similar message [Wed Feb 6 00:33:57 2019][1664411.462283] LustreError: 129228:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441913, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa0645502bf00/0x8c1de5393fa79cb7 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 108 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351804e9f66 expref: -99 pid: 129228 timeout: 0 lvb_type: 0 [Wed Feb 6 00:33:57 2019][1664411.502788] LustreError: 129228:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 38 previous similar messages [Wed Feb 6 00:34:04 2019][1664418.261343] LustreError: 129245:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.14.7@tcp) failed to reply to blocking AST (req@ffffa05a9df48000 x1623667886392560 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffffa07265eeda00/0x8c1de5393fb3336e lrc: 4/0,0 mode: PR/PR res: [0x240000cb1:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.10.14.7@tcp remote: 0xbeac4b43c26efe05 expref: 12 pid: 129265 timeout: 1664553 lvb_type: 0 [Wed Feb 6 00:34:04 2019][1664418.303844] LustreError: 129245:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages [Wed Feb 6 00:34:04 2019][1664418.314196] LustreError: 138-a: fir-MDT0001: A client on nid 10.10.14.7@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:34:04 2019][1664418.326707] LustreError: Skipped 3 previous similar messages [Wed Feb 6 00:34:04 2019][1664418.332569] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.14.7@tcp ns: mdt-fir-MDT0001_UUID lock: ffffa07265eeda00/0x8c1de5393fb3336e lrc: 3/0,0 mode: PR/PR res: [0x240000cb1:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.10.14.7@tcp remote: 0xbeac4b43c26efe05 expref: 13 pid: 129265 timeout: 0 lvb_type: 0 [Wed Feb 6 00:34:04 2019][1664418.369666] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [Wed Feb 6 00:34:04 2019][1664418.379670] LustreError: 129245:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa060d4172000 ns: mdt-fir-MDT0001_UUID lock: ffffa051d0608000/0x8c1de5393fb3338a lrc: 1/0,0 mode: --/PW res: [0x240000cb1:0x5:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT flags: 0x54a01000000000 nid: 10.10.14.7@tcp remote: 0xbeac4b43c26efe13 expref: 5 pid: 129245 timeout: 0 lvb_type: 0 [Wed Feb 6 00:34:04 2019][1664418.414110] LustreError: 129245:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message [Wed Feb 6 00:34:04 2019][1664418.453615] Lustre: fir-MDT0001: Connection restored to 82eeb05c-9f6a-d608-7f3e-29db42f96cfa (at 10.10.14.7@tcp) [Wed Feb 6 00:34:04 2019][1664418.463962] Lustre: Skipped 6 previous similar messages [Wed Feb 6 00:36:27 2019][1664561.149503] Lustre: 17827:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:36:27 2019][1664561.149503] req@ffffa086d5fd5100 x1624702166554432/t0(0) o36->41732ea7-65b1-83d7-b5d1-efb7145a422b@10.9.112.4@o2ib4:413/0 lens 528/2888 e 2 to 0 dl 1549442368 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:36:32 2019][1664566.613542] Lustre: 129595:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:36:32 2019][1664566.613542] req@ffffa0763121b900 x1624669771612960/t0(0) o36->47ce4289-7e25-8c66-9590-6b36cdee8e22@10.9.101.1@o2ib4:418/0 lens 520/2888 e 3 to 0 dl 1549442373 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:36:32 2019][1664566.642703] Lustre: 129595:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 32 previous similar messages [Wed Feb 6 00:36:34 2019][1664568.509559] Lustre: 129249:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:36:34 2019][1664568.509559] req@ffffa062d7b3d100 x1624701456922144/t0(0) o36->582c0bf9-0c65-cb8a-6846-32e5a6d43ca1@10.9.107.43@o2ib4:420/0 lens 520/2888 e 2 to 0 dl 1549442375 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:36:38 2019][1664572.580370] Lustre: fir-MDT0003: Client 47ce4289-7e25-8c66-9590-6b36cdee8e22 (at 10.9.101.1@o2ib4) reconnecting [Wed Feb 6 00:36:39 2019][1664573.624587] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:36:39 2019][1664573.632603] Lustre: 127297:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:36:39 2019][1664573.632603] req@ffffa07629c38600 x1624699694503584/t0(0) o36->86d44073-2a86-f18b-4a0f-e98051cdbb2e@10.9.105.51@o2ib4:425/0 lens 528/2888 e 3 to 0 dl 1549442380 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:36:39 2019][1664573.669980] Lustre: Skipped 1 previous similar message [Wed Feb 6 00:36:40 2019][1664574.440633] Lustre: fir-MDT0003: Client 582c0bf9-0c65-cb8a-6846-32e5a6d43ca1 (at 10.9.107.43@o2ib4) reconnecting [Wed Feb 6 00:36:44 2019][1664578.641641] Lustre: 129595:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:36:44 2019][1664578.641641] req@ffffa078bc021200 x1624705101504768/t0(0) o36->eca85409-9804-5945-41db-99b8b236d7bc@10.8.7.30@o2ib6:430/0 lens 528/2888 e 3 to 0 dl 1549442385 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:36:44 2019][1664578.670716] Lustre: 129595:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Feb 6 00:36:46 2019][1664579.916708] Lustre: fir-MDT0003: Client 86d44073-2a86-f18b-4a0f-e98051cdbb2e (at 10.9.105.51@o2ib4) reconnecting [Wed Feb 6 00:36:50 2019][1664583.995122] Lustre: fir-MDT0003: Client eca85409-9804-5945-41db-99b8b236d7bc (at 10.8.7.30@o2ib6) reconnecting [Wed Feb 6 00:36:50 2019][1664584.005291] Lustre: Skipped 1 previous similar message [Wed Feb 6 00:36:56 2019][1664590.661734] Lustre: 17855:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:36:56 2019][1664590.661734] req@ffffa073ca8e5400 x1624700571722384/t0(0) o36->9f0fc7f1-3017-a377-d5ca-3f45fac7d96c@10.9.106.2@o2ib4:442/0 lens 528/2888 e 2 to 0 dl 1549442397 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:36:56 2019][1664590.690802] Lustre: 17855:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [Wed Feb 6 00:36:57 2019][1664591.262737] LustreError: 127205:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.114.11@tcp) failed to reply to blocking AST (req@ffffa0682e721800 x1623667886510080 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa05847288d80/0x8c1de5393fb7eeec lrc: 4/0,0 mode: PR/PR res: [0x280000d67:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.114.11@tcp remote: 0xdbb126caab60e6fe expref: 10 pid: 17897 timeout: 1664726 lvb_type: 0 [Wed Feb 6 00:36:57 2019][1664591.262744] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.114.10@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:36:57 2019][1664591.262746] LustreError: Skipped 1 previous similar message [Wed Feb 6 00:36:57 2019][1664591.262770] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.114.10@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa089f0251440/0x8c1de5393fb7eed7 lrc: 3/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 42 type: IBT flags: 0x60200400000020 nid: 10.10.114.10@tcp remote: 0xd3460787bfacf2a3 expref: 10 pid: 127134 timeout: 0 lvb_type: 0 [Wed Feb 6 00:36:57 2019][1664591.262773] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [Wed Feb 6 00:36:57 2019][1664591.262943] LustreError: 129250:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa059e4695000 ns: mdt-fir-MDT0003_UUID lock: ffffa0705bb41b00/0x8c1de5393fb82a6c lrc: 3/0,0 mode: PW/PW res: [0x280000d7d:0x1:0x0].0x0 bits 0x40/0x0 rrc: 3 type: IBT flags: 0x50200000000000 nid: 10.10.115.2@tcp remote: 0xb8e1c27d85cfe9c3 expref: 8 pid: 129250 timeout: 0 lvb_type: 0 [Wed Feb 6 00:36:57 2019][1664591.405940] LustreError: 127205:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 4 previous similar messages [Wed Feb 6 00:37:02 2019][1664596.216795] Lustre: fir-MDT0003: Client 9f0fc7f1-3017-a377-d5ca-3f45fac7d96c (at 10.9.106.2@o2ib4) reconnecting [Wed Feb 6 00:37:02 2019][1664596.227056] Lustre: Skipped 8 previous similar messages [Wed Feb 6 00:37:14 2019][1664608.597880] Lustre: 17994:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:37:14 2019][1664608.597880] req@ffffa06aef2fa100 x1624697980980848/t0(0) o36->d5be83ac-c435-a05d-4965-bc79fa913f84@10.9.103.4@o2ib4:460/0 lens 528/2888 e 2 to 0 dl 1549442415 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:37:14 2019][1664608.626945] Lustre: 17994:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [Wed Feb 6 00:37:15 2019][1664609.173872] LNet: Service thread pid 127248 was inactive for 637.66s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:37:15 2019][1664609.191065] Pid: 127248, comm: mdt02_008 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:37:15 2019][1664609.201065] Call Trace: [Wed Feb 6 00:37:15 2019][1664609.203700] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:37:15 2019][1664609.210814] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:37:15 2019][1664609.218102] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:37:15 2019][1664609.224941] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:37:15 2019][1664609.231715] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:37:15 2019][1664609.238114] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:37:15 2019][1664609.244436] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:37:15 2019][1664609.251876] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:37:15 2019][1664609.258798] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:37:15 2019][1664609.267017] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:37:15 2019][1664609.273435] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:37:15 2019][1664609.279661] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:37:15 2019][1664609.286417] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:37:15 2019][1664609.292298] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:37:15 2019][1664609.299451] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:37:15 2019][1664609.307334] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:37:15 2019][1664609.313849] [] kthread+0xd1/0xe0 [Wed Feb 6 00:37:15 2019][1664609.318932] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:37:15 2019][1664609.325592] [] 0xffffffffffffffff [Wed Feb 6 00:37:15 2019][1664609.330790] LustreError: dumping log to /tmp/lustre-log.1549442411.127248 [Wed Feb 6 00:37:20 2019][1664614.672387] Lustre: fir-MDT0003: Client d5be83ac-c435-a05d-4965-bc79fa913f84 (at 10.9.103.4@o2ib4) reconnecting [Wed Feb 6 00:37:20 2019][1664614.682648] Lustre: Skipped 8 previous similar messages [Wed Feb 6 00:37:21 2019][1664615.317922] Pid: 17908, comm: mdt02_036 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:37:21 2019][1664615.327833] Call Trace: [Wed Feb 6 00:37:21 2019][1664615.330470] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:37:21 2019][1664615.337591] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:37:21 2019][1664615.344878] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:37:21 2019][1664615.351728] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:37:21 2019][1664615.358473] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:37:21 2019][1664615.364875] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:37:21 2019][1664615.371197] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:37:21 2019][1664615.378637] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:37:21 2019][1664615.385558] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:37:21 2019][1664615.393777] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:37:21 2019][1664615.400189] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:37:21 2019][1664615.406412] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:37:21 2019][1664615.413160] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:37:21 2019][1664615.419033] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:37:21 2019][1664615.426170] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:37:21 2019][1664615.434065] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:37:21 2019][1664615.440584] [] kthread+0xd1/0xe0 [Wed Feb 6 00:37:21 2019][1664615.445675] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:37:21 2019][1664615.452336] [] 0xffffffffffffffff [Wed Feb 6 00:37:21 2019][1664615.457533] LustreError: dumping log to /tmp/lustre-log.1549442417.17908 [Wed Feb 6 00:37:25 2019][1664619.413957] Pid: 129266, comm: mdt02_019 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:37:25 2019][1664619.423951] Call Trace: [Wed Feb 6 00:37:25 2019][1664619.426586] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:37:25 2019][1664619.433709] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:37:25 2019][1664619.441001] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:37:25 2019][1664619.447856] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:37:25 2019][1664619.454623] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:37:25 2019][1664619.461013] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:37:25 2019][1664619.467344] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:37:25 2019][1664619.474791] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:37:25 2019][1664619.481714] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:37:25 2019][1664619.489934] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:37:25 2019][1664619.496341] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:37:25 2019][1664619.502569] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:37:25 2019][1664619.509301] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:37:25 2019][1664619.515182] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:37:25 2019][1664619.522313] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:37:25 2019][1664619.530222] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:37:25 2019][1664619.536725] [] kthread+0xd1/0xe0 [Wed Feb 6 00:37:25 2019][1664619.541827] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:37:25 2019][1664619.548478] [] 0xffffffffffffffff [Wed Feb 6 00:37:25 2019][1664619.553691] LustreError: dumping log to /tmp/lustre-log.1549442421.129266 [Wed Feb 6 00:37:29 2019][1664623.510984] Pid: 127135, comm: mdt02_004 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:37:29 2019][1664623.520985] Call Trace: [Wed Feb 6 00:37:29 2019][1664623.523624] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:37:29 2019][1664623.530745] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:37:29 2019][1664623.538025] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:37:29 2019][1664623.544863] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:37:29 2019][1664623.551605] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:37:29 2019][1664623.558001] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:37:29 2019][1664623.564311] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:37:29 2019][1664623.571746] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:37:29 2019][1664623.578664] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:37:29 2019][1664623.586879] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:37:29 2019][1664623.593275] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:37:29 2019][1664623.599498] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:37:29 2019][1664623.606240] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:37:29 2019][1664623.612117] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:37:29 2019][1664623.619242] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:37:29 2019][1664623.627128] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:37:29 2019][1664623.633628] [] kthread+0xd1/0xe0 [Wed Feb 6 00:37:29 2019][1664623.638721] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:37:29 2019][1664623.645370] [] 0xffffffffffffffff [Wed Feb 6 00:37:29 2019][1664623.650565] LustreError: dumping log to /tmp/lustre-log.1549442425.127135 [Wed Feb 6 00:37:37 2019][1664631.702055] Pid: 126631, comm: mdt02_002 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:37:37 2019][1664631.712054] Call Trace: [Wed Feb 6 00:37:37 2019][1664631.714700] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:37:37 2019][1664631.721819] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:37:37 2019][1664631.729098] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:37:37 2019][1664631.735937] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:37:37 2019][1664631.742670] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:37:37 2019][1664631.749066] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:37:37 2019][1664631.755377] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:37:37 2019][1664631.762813] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:37:37 2019][1664631.769728] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:37:37 2019][1664631.777945] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:37:37 2019][1664631.784339] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:37:38 2019][1664631.790563] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:37:38 2019][1664631.797296] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:37:38 2019][1664631.803166] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:37:38 2019][1664631.810288] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:37:38 2019][1664631.818176] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:37:38 2019][1664631.824676] [] kthread+0xd1/0xe0 [Wed Feb 6 00:37:38 2019][1664631.829762] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:37:38 2019][1664631.836412] [] 0xffffffffffffffff [Wed Feb 6 00:37:38 2019][1664631.841604] LustreError: dumping log to /tmp/lustre-log.1549442433.126631 [Wed Feb 6 00:37:44 2019][1664637.846105] LNet: Service thread pid 128538 was inactive for 664.46s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:37:44 2019][1664637.859224] LNet: Skipped 2 previous similar messages [Wed Feb 6 00:37:44 2019][1664637.864456] LustreError: dumping log to /tmp/lustre-log.1549442439.128538 [Wed Feb 6 00:37:50 2019][1664643.990152] LustreError: dumping log to /tmp/lustre-log.1549442446.126632 [Wed Feb 6 00:37:52 2019][1664646.682190] Lustre: 17831:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:37:52 2019][1664646.682190] req@ffffa06597331200 x1624697509350544/t0(0) o36->302c6952-5b6c-1588-cbd4-2c54f063f559@10.9.103.37@o2ib4:498/0 lens 520/2888 e 2 to 0 dl 1549442453 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:37:52 2019][1664646.711351] Lustre: 17831:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [Wed Feb 6 00:37:53 2019][1664647.044815] Lustre: fir-MDT0003: Client e97cf1e5-446b-1854-d0f2-74936bdf8d40 (at 10.9.102.9@o2ib4) reconnecting [Wed Feb 6 00:37:53 2019][1664647.055073] Lustre: Skipped 7 previous similar messages [Wed Feb 6 00:37:56 2019][1664650.134205] LustreError: dumping log to /tmp/lustre-log.1549442452.127254 [Wed Feb 6 00:38:06 2019][1664660.374286] LustreError: dumping log to /tmp/lustre-log.1549442462.129795 [Wed Feb 6 00:38:10 2019][1664664.470322] LustreError: dumping log to /tmp/lustre-log.1549442466.14718 [Wed Feb 6 00:38:14 2019][1664668.566355] LustreError: dumping log to /tmp/lustre-log.1549442470.127131 [Wed Feb 6 00:38:16 2019][1664670.614366] LustreError: dumping log to /tmp/lustre-log.1549442472.17823 [Wed Feb 6 00:38:17 2019][1664671.659371] LNetError: 129249:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.115.2@tcp from [Wed Feb 6 00:38:17 2019][1664671.670155] LNetError: 129249:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 122629310 previous similar messages [Wed Feb 6 00:38:18 2019][1664672.663386] LustreError: dumping log to /tmp/lustre-log.1549442474.127390 [Wed Feb 6 00:38:19 2019][1664672.988390] LustreError: 127216:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549442175, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa061d49a2f40/0x8c1de5393fb7252e lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 108 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351805118a7 expref: -99 pid: 127216 timeout: 0 lvb_type: 0 [Wed Feb 6 00:38:19 2019][1664673.028904] LustreError: 127216:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 21 previous similar messages [Wed Feb 6 00:38:20 2019][1664674.711400] LustreError: dumping log to /tmp/lustre-log.1549442476.17889 [Wed Feb 6 00:38:22 2019][1664676.758425] LustreError: dumping log to /tmp/lustre-log.1549442478.129244 [Wed Feb 6 00:38:27 2019][1664680.854460] LustreError: dumping log to /tmp/lustre-log.1549442482.17890 [Wed Feb 6 00:38:29 2019][1664682.902475] LustreError: dumping log to /tmp/lustre-log.1549442484.127206 [Wed Feb 6 00:38:31 2019][1664685.681479] Lustre: 126630:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549442487/real 1549442487] req@ffffa086da268600 x1623667886716816/t0(0) o104->fir-MDT0003@10.10.115.2@tcp:15/16 lens 296/224 e 0 to 1 dl 1549442498 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:38:31 2019][1664685.681482] Lustre: 129249:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549442487/real 1549442487] req@ffffa052fecacb00 x1623667886758080/t0(0) o104->fir-MDT0003@10.10.115.2@tcp:15/16 lens 296/224 e 0 to 1 dl 1549442498 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:38:32 2019][1664685.681486] Lustre: 129249:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 122685200 previous similar messages [Wed Feb 6 00:38:32 2019][1664685.747700] Lustre: 126630:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10769 previous similar messages [Wed Feb 6 00:38:33 2019][1664686.998498] LustreError: dumping log to /tmp/lustre-log.1549442489.17903 [Wed Feb 6 00:38:43 2019][1664697.239583] LustreError: dumping log to /tmp/lustre-log.1549442499.17935 [Wed Feb 6 00:38:45 2019][1664699.286604] LustreError: dumping log to /tmp/lustre-log.1549442501.17953 [Wed Feb 6 00:38:49 2019][1664703.383631] LustreError: dumping log to /tmp/lustre-log.1549442505.127210 [Wed Feb 6 00:38:53 2019][1664707.478666] LustreError: dumping log to /tmp/lustre-log.1549442509.17804 [Wed Feb 6 00:38:58 2019][1664712.518316] Lustre: fir-MDT0003: Client 13784174-095e-5af3-8ad0-9ac564324f6f (at 10.9.105.54@o2ib4) reconnecting [Wed Feb 6 00:38:58 2019][1664712.528660] Lustre: Skipped 10 previous similar messages [Wed Feb 6 00:38:59 2019][1664712.982722] Lustre: 129279:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:38:59 2019][1664712.982722] req@ffffa05381d4e000 x1624699627978496/t0(0) o36->469c4428-ebc2-c887-175b-661c91237c7b@10.9.105.31@o2ib4:565/0 lens 520/2888 e 2 to 0 dl 1549442520 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:38:59 2019][1664713.011966] Lustre: 129279:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 11 previous similar messages [Wed Feb 6 00:39:01 2019][1664715.670735] LustreError: dumping log to /tmp/lustre-log.1549442517.129805 [Wed Feb 6 00:39:02 2019][1664716.255503] LNet: Service thread pid 127248 completed after 744.74s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 00:39:02 2019][1664716.271926] LNet: Skipped 12 previous similar messages [Wed Feb 6 00:39:02 2019][1664716.291913] Lustre: 127133:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffffa07153f15a00 x1624699318547888/t0(0) o36->d17746b1-99a9-d3fc-f1a0-b877f22b2279@10.9.105.57@o2ib4:562/0 lens 520/424 e 2 to 0 dl 1549442517 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 00:39:26 2019][1664740.265941] Lustre: fir-MDT0000-osp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:39:28 2019][1664742.263993] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.114.11@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa0739e32da00/0x8c1de5393fbb92a2 lrc: 3/0,0 mode: PR/PR res: [0x280000d67:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.114.11@tcp remote: 0xdbb126caab60e721 expref: 11 pid: 17838 timeout: 0 lvb_type: 0 [Wed Feb 6 00:39:28 2019][1664742.301302] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages [Wed Feb 6 00:39:28 2019][1664742.311639] LustreError: 17851:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa0545da3d400 ns: mdt-fir-MDT0003_UUID lock: ffffa0626d675580/0x8c1de5393fbb92b7 lrc: 1/0,0 mode: EX/EX res: [0x280000d67:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.114.11@tcp remote: 0xdbb126caab60e72f expref: 4 pid: 17851 timeout: 0 lvb_type: 3 [Wed Feb 6 00:39:28 2019][1664742.346011] LustreError: 17851:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message [Wed Feb 6 00:41:34 2019][1664868.055974] Lustre: 126633:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:41:34 2019][1664868.055974] req@ffffa086d145e300 x1624672975756720/t0(0) o36->8fed3aec-f5ba-00e3-ce44-384855c15f1f@10.9.102.4@o2ib4:720/0 lens 528/2888 e 1 to 0 dl 1549442675 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:41:34 2019][1664868.085128] Lustre: 126633:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Feb 6 00:41:40 2019][1664874.681151] Lustre: fir-MDT0003: Client 8fed3aec-f5ba-00e3-ce44-384855c15f1f (at 10.9.102.4@o2ib4) reconnecting [Wed Feb 6 00:41:56 2019][1664890.703143] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:42:01 2019][1664895.265188] LustreError: 128085:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.114.11@tcp) failed to reply to blocking AST (req@ffffa05a2c471500 x1623667886910448 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa0637ed53840/0x8c1de5393fc1876e lrc: 4/0,0 mode: PR/PR res: [0x280000d67:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.114.11@tcp remote: 0xdbb126caab60e744 expref: 10 pid: 129796 timeout: 1665030 lvb_type: 0 [Wed Feb 6 00:42:01 2019][1664895.265199] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.114.10@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:42:01 2019][1664895.265201] LustreError: Skipped 5 previous similar messages [Wed Feb 6 00:42:01 2019][1664895.326561] LustreError: 128085:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 4 previous similar messages [Wed Feb 6 00:42:01 2019][1664895.336956] LustreError: 128085:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa071c0e77800 ns: mdt-fir-MDT0003_UUID lock: ffffa08a11728d80/0x8c1de5393fc187f3 lrc: 3/0,0 mode: PW/PW res: [0x280000d67:0x1:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x50200000000000 nid: 10.10.114.11@tcp remote: 0xdbb126caab60e752 expref: 3 pid: 128085 timeout: 0 lvb_type: 0 [Wed Feb 6 00:42:01 2019][1664895.371590] LustreError: 128085:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message [Wed Feb 6 00:44:29 2019][1665043.054842] Lustre: fir-MDT0003: Connection restored to 15318f71-542c-f7c8-985f-d2c8b12268bf (at 10.8.3.28@o2ib6) [Wed Feb 6 00:44:29 2019][1665043.065280] Lustre: Skipped 66 previous similar messages [Wed Feb 6 00:44:32 2019][1665046.266431] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.115.2@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa088b5b0f2c0/0x8c1de5393fc18a07 lrc: 3/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 26 type: IBT flags: 0x60200400000020 nid: 10.10.115.2@tcp remote: 0xb8e1c27d85cfe9f4 expref: 11 pid: 17999 timeout: 0 lvb_type: 0 [Wed Feb 6 00:44:32 2019][1665046.303734] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages [Wed Feb 6 00:45:00 2019][1665073.927299] Lustre: fir-MDT0003: haven't heard from client 0e69550c-a059-5e4e-973b-b3107042840e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa06fbfbda400, cur 1549442876 expire 1549442726 last 1549442649 [Wed Feb 6 00:45:00 2019][1665073.949187] Lustre: Skipped 1 previous similar message [Wed Feb 6 00:46:07 2019][1665141.658179] Lustre: 128085:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-126), not sending early reply [Wed Feb 6 00:46:07 2019][1665141.658179] req@ffffa052258fd400 x1624701444395488/t0(0) o36->a392de15-7860-bd51-2003-63fafc3fc666@10.9.107.51@o2ib4:238/0 lens 520/2888 e 0 to 0 dl 1549442948 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:46:07 2019][1665141.687687] Lustre: 128085:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 20 previous similar messages [Wed Feb 6 00:46:14 2019][1665147.898825] Lustre: fir-MDT0003: Client a392de15-7860-bd51-2003-63fafc3fc666 (at 10.9.107.51@o2ib4) reconnecting [Wed Feb 6 00:46:14 2019][1665147.909171] Lustre: Skipped 20 previous similar messages [Wed Feb 6 00:47:03 2019][1665197.267717] LustreError: 129228:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa07af2d2e000 ns: mdt-fir-MDT0003_UUID lock: ffffa069f374e9c0/0x8c1de5393fc5e473 lrc: 3/0,0 mode: PW/PW res: [0x280000d67:0x2:0x0].0x0 bits 0x40/0x0 rrc: 3 type: IBT flags: 0x50200000000000 nid: 10.10.114.11@tcp remote: 0xdbb126caab60e775 expref: 7 pid: 129228 timeout: 0 lvb_type: 0 [Wed Feb 6 00:47:03 2019][1665197.302335] LustreError: 129228:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message [Wed Feb 6 00:47:56 2019][1665250.155039] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:47:56 2019][1665250.171204] LustreError: 18178:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549442752, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa050f39906c0/0x8c1de5393fc5abcb lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 121 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518054b8c1 expref: -99 pid: 18178 timeout: 0 lvb_type: 0 [Wed Feb 6 00:47:56 2019][1665250.211539] LustreError: 18178:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 118 previous similar messages [Wed Feb 6 00:48:17 2019][1665271.664211] LNetError: 17823:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.115.2@tcp from [Wed Feb 6 00:48:17 2019][1665271.674905] LNetError: 17823:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 96400250 previous similar messages [Wed Feb 6 00:48:31 2019][1665285.479323] Lustre: fir-MDT0000-osp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:48:31 2019][1665285.686323] Lustre: 17823:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549443087/real 1549443087] req@ffffa0661c677500 x1623667887150096/t0(0) o104->fir-MDT0003@10.10.115.2@tcp:15/16 lens 296/224 e 0 to 1 dl 1549443098 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:48:31 2019][1665285.714011] Lustre: 17823:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 96009785 previous similar messages [Wed Feb 6 00:51:36 2019][1665470.269821] LustreError: 17952:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.15.5@tcp) failed to reply to blocking AST (req@ffffa056e19efb00 x1623667887202128 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffffa05d6f9eba80/0x8c1de5393fc5e7d0 lrc: 4/0,0 mode: PR/PR res: [0x240000406:0x4f2:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349bf67 expref: 10 pid: 129371 timeout: 1665605 lvb_type: 0 [Wed Feb 6 00:51:36 2019][1665470.312410] LustreError: 17952:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 6 previous similar messages [Wed Feb 6 00:51:36 2019][1665470.322671] LustreError: 138-a: fir-MDT0001: A client on nid 10.10.15.5@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:51:36 2019][1665470.335185] LustreError: Skipped 8 previous similar messages [Wed Feb 6 00:52:05 2019][1665499.270108] LustreError: 129796:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa0665c73b900 x1623667887238816/t0(0) o104->fir-MDT0003@10.10.114.11@tcp:15/16 lens 296/224 e 0 to 1 dl 1549443312 ref 1 fl Rpc:eX/2/ffffffff rc -5/-1 [Wed Feb 6 00:53:09 2019][1665563.469576] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:55:10 2019][1665684.271567] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.115.2@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa08ac0644800/0x8c1de5393fc91c3c lrc: 3/0,0 mode: PR/PR res: [0x280000d7d:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.115.2@tcp remote: 0xb8e1c27d85cfea64 expref: 11 pid: 18154 timeout: 0 lvb_type: 0 [Wed Feb 6 00:55:10 2019][1665684.308702] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 6 previous similar messages [Wed Feb 6 00:55:10 2019][1665684.351296] Lustre: fir-MDT0003: Connection restored to 64452b1e-c992-0b39-ceb8-0d9660977672 (at 10.10.115.2@tcp) [Wed Feb 6 00:55:10 2019][1665684.361730] Lustre: Skipped 133 previous similar messages [Wed Feb 6 00:55:26 2019][1665700.254687] Lustre: 129246:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 00:55:26 2019][1665700.254687] req@ffffa078b8e5aa00 x1624669508062512/t0(0) o36->64c2e10f-595e-bb8e-efbd-0544992f523c@10.9.101.49@o2ib4:42/0 lens 520/2888 e 0 to 0 dl 1549443507 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:55:26 2019][1665700.284105] Lustre: 129246:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 101 previous similar messages [Wed Feb 6 00:55:32 2019][1665706.229070] Lustre: fir-MDT0003: Client 64c2e10f-595e-bb8e-efbd-0544992f523c (at 10.9.101.49@o2ib4) reconnecting [Wed Feb 6 00:55:32 2019][1665706.239419] Lustre: Skipped 102 previous similar messages [Wed Feb 6 00:56:36 2019][1665770.361248] Lustre: fir-MDT0000-osp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:57:03 2019][1665797.023455] LNet: Service thread pid 17870 was inactive for 1204.57s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:57:03 2019][1665797.040651] LNet: Skipped 4 previous similar messages [Wed Feb 6 00:57:03 2019][1665797.045886] Pid: 17870, comm: mdt00_028 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:57:03 2019][1665797.055797] Call Trace: [Wed Feb 6 00:57:03 2019][1665797.058441] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:57:03 2019][1665797.065555] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:57:03 2019][1665797.072836] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:57:03 2019][1665797.079683] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:57:03 2019][1665797.086417] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:57:03 2019][1665797.092814] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:57:03 2019][1665797.099113] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:57:03 2019][1665797.106549] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:57:03 2019][1665797.113466] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:57:03 2019][1665797.121680] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:57:03 2019][1665797.128078] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:57:03 2019][1665797.134291] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:57:03 2019][1665797.141024] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:57:03 2019][1665797.146901] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:57:03 2019][1665797.154017] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:57:03 2019][1665797.161903] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:57:03 2019][1665797.168403] [] kthread+0xd1/0xe0 [Wed Feb 6 00:57:03 2019][1665797.173493] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:57:03 2019][1665797.180138] [] 0xffffffffffffffff [Wed Feb 6 00:57:03 2019][1665797.185333] LustreError: dumping log to /tmp/lustre-log.1549443599.17870 [Wed Feb 6 00:57:11 2019][1665805.215513] LNet: Service thread pid 127317 was inactive for 1202.14s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:57:11 2019][1665805.232792] Pid: 127317, comm: mdt03_008 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:57:11 2019][1665805.242794] Call Trace: [Wed Feb 6 00:57:11 2019][1665805.245436] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:57:11 2019][1665805.252540] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:57:11 2019][1665805.259820] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:57:11 2019][1665805.266668] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:57:11 2019][1665805.273428] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:57:11 2019][1665805.279824] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:57:11 2019][1665805.286133] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:57:11 2019][1665805.293567] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:57:11 2019][1665805.300485] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:57:11 2019][1665805.308701] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:57:11 2019][1665805.315096] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:57:11 2019][1665805.321320] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:57:11 2019][1665805.328052] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:57:11 2019][1665805.333922] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:57:11 2019][1665805.341053] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:57:11 2019][1665805.348941] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:57:11 2019][1665805.355442] [] kthread+0xd1/0xe0 [Wed Feb 6 00:57:11 2019][1665805.360527] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:57:11 2019][1665805.367175] [] 0xffffffffffffffff [Wed Feb 6 00:57:11 2019][1665805.372369] LustreError: dumping log to /tmp/lustre-log.1549443607.127317 [Wed Feb 6 00:57:15 2019][1665809.311546] Pid: 127199, comm: mdt02_006 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:57:15 2019][1665809.321542] Call Trace: [Wed Feb 6 00:57:15 2019][1665809.324178] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:57:15 2019][1665809.331300] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:57:15 2019][1665809.338571] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:57:15 2019][1665809.345408] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:57:15 2019][1665809.352143] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:57:15 2019][1665809.358540] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:57:15 2019][1665809.364848] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:57:15 2019][1665809.372284] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:57:15 2019][1665809.379191] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:57:15 2019][1665809.387407] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:57:15 2019][1665809.393803] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:57:15 2019][1665809.400026] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:57:15 2019][1665809.406760] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:57:15 2019][1665809.412636] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:57:15 2019][1665809.419765] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:57:15 2019][1665809.427664] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:57:15 2019][1665809.434165] [] kthread+0xd1/0xe0 [Wed Feb 6 00:57:15 2019][1665809.439244] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:57:15 2019][1665809.445890] [] 0xffffffffffffffff [Wed Feb 6 00:57:15 2019][1665809.451086] LustreError: dumping log to /tmp/lustre-log.1549443611.127199 [Wed Feb 6 00:57:15 2019][1665809.539884] Pid: 17838, comm: mdt02_029 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:57:15 2019][1665809.549800] Call Trace: [Wed Feb 6 00:57:15 2019][1665809.552437] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:57:15 2019][1665809.559557] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:57:15 2019][1665809.566836] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:57:15 2019][1665809.573682] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:57:15 2019][1665809.580425] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:57:15 2019][1665809.586821] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:57:15 2019][1665809.593132] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:57:15 2019][1665809.600567] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:57:15 2019][1665809.607485] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:57:15 2019][1665809.615699] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:57:15 2019][1665809.622102] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:57:15 2019][1665809.628318] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:57:15 2019][1665809.635052] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:57:15 2019][1665809.640919] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:57:15 2019][1665809.648045] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:57:15 2019][1665809.655931] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:57:15 2019][1665809.662434] [] kthread+0xd1/0xe0 [Wed Feb 6 00:57:15 2019][1665809.667517] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:57:15 2019][1665809.674165] [] 0xffffffffffffffff [Wed Feb 6 00:57:19 2019][1665813.407581] Pid: 17907, comm: mdt03_031 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:57:19 2019][1665813.417494] Call Trace: [Wed Feb 6 00:57:19 2019][1665813.420134] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:57:19 2019][1665813.427253] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:57:19 2019][1665813.434525] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:57:19 2019][1665813.441363] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:57:19 2019][1665813.448097] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:57:19 2019][1665813.454494] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:57:19 2019][1665813.460801] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:57:19 2019][1665813.468238] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:57:19 2019][1665813.475145] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:57:19 2019][1665813.483378] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:57:19 2019][1665813.489775] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:57:19 2019][1665813.495998] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:57:19 2019][1665813.502740] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:57:19 2019][1665813.508617] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:57:19 2019][1665813.515731] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:57:19 2019][1665813.523618] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:57:19 2019][1665813.530119] [] kthread+0xd1/0xe0 [Wed Feb 6 00:57:19 2019][1665813.535206] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:57:19 2019][1665813.541854] [] 0xffffffffffffffff [Wed Feb 6 00:57:19 2019][1665813.547062] LustreError: dumping log to /tmp/lustre-log.1549443615.17907 [Wed Feb 6 00:57:31 2019][1665825.695680] LNet: Service thread pid 17932 was inactive for 1203.05s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:57:31 2019][1665825.708800] LNet: Skipped 27 previous similar messages [Wed Feb 6 00:57:31 2019][1665825.714114] LustreError: dumping log to /tmp/lustre-log.1549443627.17932 [Wed Feb 6 00:57:41 2019][1665835.272864] LustreError: 17970:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa086d7323000 ns: mdt-fir-MDT0003_UUID lock: ffffa078cc3e0b40/0x8c1de5393fc9b6ac lrc: 1/0,0 mode: --/PW res: [0x280000d67:0x2:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x54a01000000000 nid: 10.10.114.11@tcp remote: 0xdbb126caab60e7de expref: 5 pid: 17970 timeout: 0 lvb_type: 0 [Wed Feb 6 00:57:41 2019][1665835.307305] LustreError: 17970:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 6 previous similar messages [Wed Feb 6 00:57:48 2019][1665842.079818] LustreError: dumping log to /tmp/lustre-log.1549443644.127195 [Wed Feb 6 00:58:17 2019][1665871.669049] LNetError: 17970:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.115.2@tcp from [Wed Feb 6 00:58:17 2019][1665871.679740] LNetError: 17970:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 122023875 previous similar messages [Wed Feb 6 00:58:29 2019][1665883.040167] LustreError: dumping log to /tmp/lustre-log.1549443685.129248 [Wed Feb 6 00:58:31 2019][1665885.691164] Lustre: 17970:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549443687/real 1549443687] req@ffffa053d699ef00 x1623667887442192/t0(0) o104->fir-MDT0003@10.10.115.2@tcp:15/16 lens 296/224 e 0 to 1 dl 1549443698 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:58:31 2019][1665885.718849] Lustre: 17970:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 122164324 previous similar messages [Wed Feb 6 00:58:33 2019][1665887.136178] LustreError: dumping log to /tmp/lustre-log.1549443689.17831 [Wed Feb 6 00:58:37 2019][1665891.232212] LNet: Service thread pid 17994 was inactive for 1203.75s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:58:37 2019][1665891.245332] LNet: Skipped 6 previous similar messages [Wed Feb 6 00:58:37 2019][1665891.250569] LustreError: dumping log to /tmp/lustre-log.1549443693.17994 [Wed Feb 6 00:58:40 2019][1665893.971261] LustreError: 126629:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549443396, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa073b18d6c00/0x8c1de5393fc94b3d lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 132 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518065f243 expref: -99 pid: 126629 timeout: 0 lvb_type: 0 [Wed Feb 6 00:58:40 2019][1665894.011770] LustreError: 126629:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 37 previous similar messages [Wed Feb 6 00:58:45 2019][1665899.424275] LustreError: dumping log to /tmp/lustre-log.1549443701.127273 [Wed Feb 6 00:58:49 2019][1665903.520320] LustreError: dumping log to /tmp/lustre-log.1549443705.17811 [Wed Feb 6 00:59:06 2019][1665919.904469] LustreError: dumping log to /tmp/lustre-log.1549443721.17930 [Wed Feb 6 00:59:10 2019][1665924.000477] LustreError: dumping log to /tmp/lustre-log.1549443726.128267 [Wed Feb 6 00:59:22 2019][1665936.288576] LustreError: dumping log to /tmp/lustre-log.1549443738.14718 [Wed Feb 6 00:59:33 2019][1665947.256461] LNet: Service thread pid 17870 completed after 1354.80s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 00:59:33 2019][1665947.272880] LNet: Skipped 17 previous similar messages [Wed Feb 6 00:59:38 2019][1665952.672711] LustreError: dumping log to /tmp/lustre-log.1549443754.17968 [Wed Feb 6 00:59:43 2019][1665956.768739] LustreError: dumping log to /tmp/lustre-log.1549443758.17935 [Wed Feb 6 00:59:47 2019][1665960.864770] LustreError: dumping log to /tmp/lustre-log.1549443762.17829 [Wed Feb 6 00:59:55 2019][1665969.056919] LustreError: dumping log to /tmp/lustre-log.1549443771.129250 [Wed Feb 6 00:59:59 2019][1665973.152870] LustreError: dumping log to /tmp/lustre-log.1549443775.126630 [Wed Feb 6 01:00:03 2019][1665977.248904] LustreError: dumping log to /tmp/lustre-log.1549443779.127254 [Wed Feb 6 01:00:07 2019][1665981.344978] LustreError: dumping log to /tmp/lustre-log.1549443783.129594 [Wed Feb 6 01:00:19 2019][1665993.633045] LustreError: dumping log to /tmp/lustre-log.1549443795.18164 [Wed Feb 6 01:00:23 2019][1665997.729066] LustreError: dumping log to /tmp/lustre-log.1549443799.127198 [Wed Feb 6 01:00:40 2019][1666014.113201] LustreError: dumping log to /tmp/lustre-log.1549443816.129244 [Wed Feb 6 01:02:03 2019][1666097.274878] LustreError: 17932:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.3@tcp) failed to reply to blocking AST (req@ffffa04ee8832400 x1623667887483888 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffffa08a512fa640/0x8c1de5393fc989d4 lrc: 4/0,0 mode: PR/PR res: [0x240000406:0x4f2:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.113.3@tcp remote: 0x4d977b31312bdcf3 expref: 10 pid: 17850 timeout: 1666232 lvb_type: 0 [Wed Feb 6 01:02:03 2019][1666097.317553] LustreError: 17932:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 10 previous similar messages [Wed Feb 6 01:02:03 2019][1666097.327902] LustreError: 138-a: fir-MDT0001: A client on nid 10.10.113.3@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 01:02:03 2019][1666097.340519] LustreError: Skipped 10 previous similar messages [Wed Feb 6 01:02:03 2019][1666097.346769] LNet: Service thread pid 17932 completed after 1474.70s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:02:59 2019][1666153.378322] LNet: Service thread pid 18178 was inactive for 1203.21s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 01:02:59 2019][1666153.395518] LNet: Skipped 3 previous similar messages [Wed Feb 6 01:02:59 2019][1666153.400753] Pid: 18178, comm: mdt02_044 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:02:59 2019][1666153.410684] Call Trace: [Wed Feb 6 01:02:59 2019][1666153.413321] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:02:59 2019][1666153.420439] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:02:59 2019][1666153.427725] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:02:59 2019][1666153.434566] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:02:59 2019][1666153.441321] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:02:59 2019][1666153.447704] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:02:59 2019][1666153.454020] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:02:59 2019][1666153.461460] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:02:59 2019][1666153.468383] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:02:59 2019][1666153.476593] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:02:59 2019][1666153.483005] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:02:59 2019][1666153.489228] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:02:59 2019][1666153.495976] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:02:59 2019][1666153.501847] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:02:59 2019][1666153.508977] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:02:59 2019][1666153.516865] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:02:59 2019][1666153.523380] [] kthread+0xd1/0xe0 [Wed Feb 6 01:02:59 2019][1666153.528472] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:02:59 2019][1666153.535123] [] 0xffffffffffffffff [Wed Feb 6 01:02:59 2019][1666153.540319] LustreError: dumping log to /tmp/lustre-log.1549443955.18178 [Wed Feb 6 01:03:11 2019][1666165.666419] Pid: 127133, comm: mdt02_003 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:03:11 2019][1666165.676419] Call Trace: [Wed Feb 6 01:03:11 2019][1666165.679058] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:03:11 2019][1666165.686184] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:03:11 2019][1666165.693487] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:03:11 2019][1666165.700338] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:03:11 2019][1666165.707096] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:03:11 2019][1666165.713495] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:03:11 2019][1666165.719818] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:03:11 2019][1666165.727266] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:03:11 2019][1666165.734197] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:03:11 2019][1666165.742415] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:03:11 2019][1666165.748827] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:03:11 2019][1666165.755050] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:03:11 2019][1666165.761808] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:03:11 2019][1666165.767688] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:03:11 2019][1666165.774835] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:03:11 2019][1666165.782724] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:03:11 2019][1666165.789234] [] kthread+0xd1/0xe0 [Wed Feb 6 01:03:11 2019][1666165.794319] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:03:12 2019][1666165.800985] [] 0xffffffffffffffff [Wed Feb 6 01:03:12 2019][1666165.806192] LustreError: dumping log to /tmp/lustre-log.1549443967.127133 [Wed Feb 6 01:03:28 2019][1666182.051559] Pid: 127250, comm: mdt00_009 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:03:28 2019][1666182.061556] Call Trace: [Wed Feb 6 01:03:28 2019][1666182.064197] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:03:28 2019][1666182.071313] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:03:28 2019][1666182.078585] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:03:28 2019][1666182.085434] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:03:28 2019][1666182.092166] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:03:28 2019][1666182.098553] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:03:28 2019][1666182.104864] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:03:28 2019][1666182.112300] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:03:28 2019][1666182.119205] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:03:28 2019][1666182.127422] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:03:28 2019][1666182.133809] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:03:28 2019][1666182.140034] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:03:28 2019][1666182.146777] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:03:28 2019][1666182.152642] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:03:28 2019][1666182.159768] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:03:28 2019][1666182.167654] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:03:28 2019][1666182.174154] [] kthread+0xd1/0xe0 [Wed Feb 6 01:03:28 2019][1666182.179241] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:03:28 2019][1666182.185889] [] 0xffffffffffffffff [Wed Feb 6 01:03:28 2019][1666182.191084] LustreError: dumping log to /tmp/lustre-log.1549443984.127250 [Wed Feb 6 01:03:32 2019][1666186.146620] LNet: Service thread pid 127970 was inactive for 1200.65s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 01:03:32 2019][1666186.163903] LNet: Skipped 2 previous similar messages [Wed Feb 6 01:03:32 2019][1666186.169139] Pid: 127970, comm: mdt00_010 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:03:32 2019][1666186.179138] Call Trace: [Wed Feb 6 01:03:32 2019][1666186.181777] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:03:32 2019][1666186.188895] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:03:32 2019][1666186.196167] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:03:32 2019][1666186.203013] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:03:32 2019][1666186.209746] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:03:32 2019][1666186.216135] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:03:32 2019][1666186.222444] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:03:32 2019][1666186.229879] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:03:32 2019][1666186.236787] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:03:32 2019][1666186.245004] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:03:32 2019][1666186.251399] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:03:32 2019][1666186.257622] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:03:32 2019][1666186.264357] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:03:32 2019][1666186.270224] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:03:32 2019][1666186.277350] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:03:32 2019][1666186.285235] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:03:32 2019][1666186.291734] [] kthread+0xd1/0xe0 [Wed Feb 6 01:03:32 2019][1666186.296822] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:03:32 2019][1666186.303469] [] 0xffffffffffffffff [Wed Feb 6 01:03:32 2019][1666186.308663] LustreError: dumping log to /tmp/lustre-log.1549443988.127970 [Wed Feb 6 01:04:33 2019][1666247.270081] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 01:04:33 2019][1666247.286235] Lustre: Skipped 1 previous similar message [Wed Feb 6 01:04:34 2019][1666248.276463] LNet: Service thread pid 17934 completed after 1625.47s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:04:34 2019][1666248.292878] LNet: Skipped 7 previous similar messages [Wed Feb 6 01:04:46 2019][1666259.875191] LNet: Service thread pid 129371 was inactive for 1201.90s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 01:04:46 2019][1666259.892478] Pid: 129371, comm: mdt01_029 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:04:46 2019][1666259.902494] Call Trace: [Wed Feb 6 01:04:46 2019][1666259.905137] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:04:46 2019][1666259.912282] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:04:46 2019][1666259.919563] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:04:46 2019][1666259.926424] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:04:46 2019][1666259.933172] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:04:46 2019][1666259.939599] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:04:46 2019][1666259.945912] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:04:46 2019][1666259.953376] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:04:46 2019][1666259.960296] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:04:46 2019][1666259.968529] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:04:46 2019][1666259.974926] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:04:46 2019][1666259.981165] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:04:46 2019][1666259.987908] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:04:46 2019][1666259.993785] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:04:46 2019][1666260.000924] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:04:46 2019][1666260.008829] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:04:46 2019][1666260.015345] [] kthread+0xd1/0xe0 [Wed Feb 6 01:04:46 2019][1666260.020436] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:04:46 2019][1666260.027096] [] 0xffffffffffffffff [Wed Feb 6 01:04:46 2019][1666260.032296] LustreError: dumping log to /tmp/lustre-log.1549444062.129371 [Wed Feb 6 01:05:39 2019][1666313.123612] LNet: Service thread pid 18223 was inactive for 1203.05s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 01:05:39 2019][1666313.136733] LNet: Skipped 62 previous similar messages [Wed Feb 6 01:05:39 2019][1666313.142057] LustreError: dumping log to /tmp/lustre-log.1549444115.18223 [Wed Feb 6 01:06:10 2019][1666344.355874] Lustre: 18267:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 01:06:10 2019][1666344.355874] req@ffffa069ac240900 x1624705132658080/t0(0) o36->5b8aac1d-db3a-957a-49fa-fd5d736eb555@10.8.27.13@o2ib6:686/0 lens 520/2888 e 0 to 0 dl 1549444151 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 01:06:10 2019][1666344.385204] Lustre: 18267:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 37 previous similar messages [Wed Feb 6 01:06:16 2019][1666350.028770] Lustre: fir-MDT0003: Client 5b8aac1d-db3a-957a-49fa-fd5d736eb555 (at 10.8.27.13@o2ib6) reconnecting [Wed Feb 6 01:06:16 2019][1666350.039035] Lustre: Skipped 88 previous similar messages [Wed Feb 6 01:06:16 2019][1666350.044549] Lustre: fir-MDT0003: Connection restored to 5b8aac1d-db3a-957a-49fa-fd5d736eb555 (at 10.8.27.13@o2ib6) [Wed Feb 6 01:06:16 2019][1666350.055089] Lustre: Skipped 97 previous similar messages [Wed Feb 6 01:06:24 2019][1666358.179975] LustreError: dumping log to /tmp/lustre-log.1549444160.129229 [Wed Feb 6 01:06:32 2019][1666366.372042] LustreError: dumping log to /tmp/lustre-log.1549444168.18262 [Wed Feb 6 01:06:36 2019][1666370.468075] LustreError: dumping log to /tmp/lustre-log.1549444172.17894 [Wed Feb 6 01:07:05 2019][1666399.263837] LNet: Service thread pid 18178 completed after 1449.09s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:07:05 2019][1666399.280274] LNet: Skipped 59 previous similar messages [Wed Feb 6 01:07:54 2019][1666448.277725] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.8@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa08a21a02f40/0x8c1de5393fca2eaf lrc: 3/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 11 type: IBT flags: 0x60200400000020 nid: 10.10.112.8@tcp remote: 0x8ea3e4d27143d567 expref: 9 pid: 18165 timeout: 0 lvb_type: 0 [Wed Feb 6 01:07:54 2019][1666448.314945] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 7 previous similar messages [Wed Feb 6 01:08:10 2019][1666464.676838] LNet: Service thread pid 17958 was inactive for 1201.19s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 01:08:10 2019][1666464.694034] Pid: 17958, comm: mdt01_042 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:08:10 2019][1666464.703949] Call Trace: [Wed Feb 6 01:08:10 2019][1666464.706596] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:08:10 2019][1666464.713722] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:08:10 2019][1666464.721001] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:08:10 2019][1666464.727848] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:08:10 2019][1666464.734581] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:08:10 2019][1666464.740978] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:08:10 2019][1666464.747289] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:08:10 2019][1666464.754724] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:08:10 2019][1666464.761640] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:08:10 2019][1666464.769856] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:08:10 2019][1666464.776243] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:08:10 2019][1666464.782466] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:08:11 2019][1666464.789200] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:08:11 2019][1666464.795068] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:08:11 2019][1666464.802193] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:08:11 2019][1666464.810080] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:08:11 2019][1666464.816578] [] kthread+0xd1/0xe0 [Wed Feb 6 01:08:11 2019][1666464.821665] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:08:11 2019][1666464.828314] [] 0xffffffffffffffff [Wed Feb 6 01:08:11 2019][1666464.833517] LustreError: dumping log to /tmp/lustre-log.1549444266.17958 [Wed Feb 6 01:08:14 2019][1666468.772875] Pid: 126625, comm: mdt00_002 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:08:14 2019][1666468.782873] Call Trace: [Wed Feb 6 01:08:14 2019][1666468.785514] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:08:15 2019][1666468.792642] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:08:15 2019][1666468.799912] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:08:15 2019][1666468.806751] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:08:15 2019][1666468.813483] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:08:15 2019][1666468.819871] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:08:15 2019][1666468.826172] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:08:15 2019][1666468.833610] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:08:15 2019][1666468.840515] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:08:15 2019][1666468.848732] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:08:15 2019][1666468.855128] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:08:15 2019][1666468.861341] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:08:15 2019][1666468.868076] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:08:15 2019][1666468.873944] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:08:15 2019][1666468.881060] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:08:15 2019][1666468.888946] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:08:15 2019][1666468.895446] [] kthread+0xd1/0xe0 [Wed Feb 6 01:08:15 2019][1666468.900534] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:08:15 2019][1666468.907181] [] 0xffffffffffffffff [Wed Feb 6 01:08:15 2019][1666468.912374] LustreError: dumping log to /tmp/lustre-log.1549444270.126625 [Wed Feb 6 01:08:17 2019][1666471.673883] LNetError: 17953:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.15.6@tcp from [Wed Feb 6 01:08:17 2019][1666471.684492] LNetError: 17953:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 76252778 previous similar messages [Wed Feb 6 01:08:31 2019][1666485.695995] Lustre: 17953:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549444287/real 1549444287] req@ffffa05811398900 x1623667887705376/t0(0) o104->fir-MDT0003@10.10.15.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549444298 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 01:08:31 2019][1666485.723587] Lustre: 17953:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 76266640 previous similar messages [Wed Feb 6 01:09:08 2019][1666522.021299] Pid: 17853, comm: mdt03_026 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:09:08 2019][1666522.031213] Call Trace: [Wed Feb 6 01:09:08 2019][1666522.033855] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.040982] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.048296] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.055142] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:09:08 2019][1666522.061891] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:09:08 2019][1666522.068290] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:09:08 2019][1666522.074614] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:09:08 2019][1666522.082051] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:09:08 2019][1666522.088974] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:09:08 2019][1666522.097194] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:09:08 2019][1666522.103603] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:09:08 2019][1666522.109829] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:09:08 2019][1666522.116577] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:09:08 2019][1666522.122449] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.129588] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.137477] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.143977] [] kthread+0xd1/0xe0 [Wed Feb 6 01:09:08 2019][1666522.149067] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:09:08 2019][1666522.155727] [] 0xffffffffffffffff [Wed Feb 6 01:09:08 2019][1666522.160922] LustreError: dumping log to /tmp/lustre-log.1549444324.17853 [Wed Feb 6 01:09:08 2019][1666522.191938] Pid: 17828, comm: mdt03_022 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:09:08 2019][1666522.201846] Call Trace: [Wed Feb 6 01:09:08 2019][1666522.204481] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.211594] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.218882] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.225731] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:09:08 2019][1666522.232465] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:09:08 2019][1666522.238852] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:09:08 2019][1666522.245168] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:09:08 2019][1666522.252607] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:09:08 2019][1666522.259535] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:09:08 2019][1666522.267756] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:09:08 2019][1666522.274158] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:09:08 2019][1666522.280385] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:09:08 2019][1666522.287131] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:09:08 2019][1666522.293002] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.300117] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.308005] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.314519] [] kthread+0xd1/0xe0 [Wed Feb 6 01:09:08 2019][1666522.319610] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:09:08 2019][1666522.326271] [] 0xffffffffffffffff [Wed Feb 6 01:09:08 2019][1666522.331459] Pid: 17904, comm: mdt00_037 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:09:08 2019][1666522.341379] Call Trace: [Wed Feb 6 01:09:08 2019][1666522.344012] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.351113] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.358401] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.365238] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:09:08 2019][1666522.371972] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:09:08 2019][1666522.378361] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:09:08 2019][1666522.384677] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:09:08 2019][1666522.392107] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:09:08 2019][1666522.399027] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:09:08 2019][1666522.407247] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:09:08 2019][1666522.413650] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:09:08 2019][1666522.419876] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:09:08 2019][1666522.426622] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:09:08 2019][1666522.432502] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.439624] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.447514] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:09:08 2019][1666522.454028] [] kthread+0xd1/0xe0 [Wed Feb 6 01:09:08 2019][1666522.459110] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:09:08 2019][1666522.465763] [] 0xffffffffffffffff [Wed Feb 6 01:09:16 2019][1666530.213373] LustreError: dumping log to /tmp/lustre-log.1549444332.129587 [Wed Feb 6 01:09:28 2019][1666542.501461] LustreError: dumping log to /tmp/lustre-log.1549444344.18265 [Wed Feb 6 01:09:34 2019][1666548.304510] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 01:09:34 2019][1666548.305520] LustreError: 129280:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549444050, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0001 lock: ffffa06386c36e40/0x8c1de5393fcb4b24 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 14 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351807f0f37 expref: -99 pid: 129280 timeout: 0 lvb_type: 0 [Wed Feb 6 01:09:34 2019][1666548.305523] LustreError: 129280:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 36 previous similar messages [Wed Feb 6 01:09:34 2019][1666548.372313] Lustre: Skipped 1 previous similar message [Wed Feb 6 01:09:36 2019][1666550.265769] LNet: Service thread pid 127250 completed after 1570.50s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:09:36 2019][1666550.282295] LNet: Skipped 1 previous similar message [Wed Feb 6 01:09:36 2019][1666550.693531] LustreError: dumping log to /tmp/lustre-log.1549444352.17834 [Wed Feb 6 01:09:54 2019][1666568.265278] Lustre: 129371:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:755s); client may timeout. req@ffffa064201f3900 x1624705184361472/t0(0) o36->361df7fd-56ca-4920-f3c9-3d9bba670835@10.10.15.5@tcp:150/0 lens 520/424 e 0 to 0 dl 1549443615 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 01:09:54 2019][1666568.267461] LNet: Service thread pid 18223 completed after 1458.20s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:11:39 2019][1666673.574514] LNet: Service thread pid 847 was inactive for 1202.30s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 01:11:39 2019][1666673.587458] LNet: Skipped 13 previous similar messages [Wed Feb 6 01:11:39 2019][1666673.592778] LustreError: dumping log to /tmp/lustre-log.1549444475.847 [Wed Feb 6 01:11:47 2019][1666681.766593] LustreError: dumping log to /tmp/lustre-log.1549444483.17992 [Wed Feb 6 01:12:28 2019][1666722.267590] LNet: Service thread pid 847 completed after 1250.99s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:12:28 2019][1666722.283848] LNet: Skipped 20 previous similar messages [Wed Feb 6 01:13:42 2019][1666796.455506] LNet: Service thread pid 126629 was inactive for 1202.47s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 01:13:42 2019][1666796.472789] LNet: Skipped 4 previous similar messages [Wed Feb 6 01:13:42 2019][1666796.478017] Pid: 126629, comm: mdt02_000 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:13:42 2019][1666796.488015] Call Trace: [Wed Feb 6 01:13:42 2019][1666796.490648] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:13:42 2019][1666796.497764] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:13:42 2019][1666796.505033] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:13:42 2019][1666796.511872] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:13:42 2019][1666796.518609] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:13:42 2019][1666796.524994] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:13:42 2019][1666796.531302] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:13:42 2019][1666796.538740] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:13:42 2019][1666796.545653] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:13:42 2019][1666796.553862] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:13:42 2019][1666796.560251] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:13:42 2019][1666796.566465] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:13:42 2019][1666796.573198] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:13:42 2019][1666796.579070] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:13:42 2019][1666796.586191] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:13:42 2019][1666796.594077] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:13:42 2019][1666796.600581] [] kthread+0xd1/0xe0 [Wed Feb 6 01:13:42 2019][1666796.605660] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:13:42 2019][1666796.612304] [] 0xffffffffffffffff [Wed Feb 6 01:13:42 2019][1666796.617500] LustreError: dumping log to /tmp/lustre-log.1549444598.126629 [Wed Feb 6 01:14:11 2019][1666825.127747] Pid: 127208, comm: mdt01_008 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:14:11 2019][1666825.137744] Call Trace: [Wed Feb 6 01:14:11 2019][1666825.140378] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.147493] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.154770] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.161619] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:14:11 2019][1666825.168351] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:14:11 2019][1666825.174749] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:14:11 2019][1666825.181050] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:14:11 2019][1666825.188485] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:14:11 2019][1666825.195392] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:14:11 2019][1666825.203609] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:14:11 2019][1666825.210005] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:14:11 2019][1666825.216227] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:14:11 2019][1666825.222960] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:14:11 2019][1666825.228837] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.235953] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.243840] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.250339] [] kthread+0xd1/0xe0 [Wed Feb 6 01:14:11 2019][1666825.255428] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:14:11 2019][1666825.262076] [] 0xffffffffffffffff [Wed Feb 6 01:14:11 2019][1666825.267267] LustreError: dumping log to /tmp/lustre-log.1549444627.127208 [Wed Feb 6 01:14:11 2019][1666825.357086] Pid: 129267, comm: mdt00_020 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:14:11 2019][1666825.367084] Call Trace: [Wed Feb 6 01:14:11 2019][1666825.369721] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.376841] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.384112] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.390957] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:14:11 2019][1666825.397700] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:14:11 2019][1666825.404097] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:14:11 2019][1666825.410397] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:14:11 2019][1666825.417834] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:14:11 2019][1666825.424741] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:14:11 2019][1666825.432958] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:14:11 2019][1666825.439354] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:14:11 2019][1666825.445569] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:14:11 2019][1666825.452312] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:14:11 2019][1666825.458179] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.465302] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.473190] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.479688] [] kthread+0xd1/0xe0 [Wed Feb 6 01:14:11 2019][1666825.484779] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:14:11 2019][1666825.491423] [] 0xffffffffffffffff [Wed Feb 6 01:14:11 2019][1666825.496620] Pid: 129252, comm: mdt01_022 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:14:11 2019][1666825.506619] Call Trace: [Wed Feb 6 01:14:11 2019][1666825.509250] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.516349] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.523629] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.530467] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:14:11 2019][1666825.537202] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:14:11 2019][1666825.543588] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:14:11 2019][1666825.549891] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:14:11 2019][1666825.557317] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:14:11 2019][1666825.564225] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:14:11 2019][1666825.572440] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:14:11 2019][1666825.578836] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:14:11 2019][1666825.585059] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:14:11 2019][1666825.591795] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:14:11 2019][1666825.597670] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.604777] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.612665] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.619162] [] kthread+0xd1/0xe0 [Wed Feb 6 01:14:11 2019][1666825.624242] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:14:11 2019][1666825.630882] [] 0xffffffffffffffff [Wed Feb 6 01:14:11 2019][1666825.636058] Pid: 18044, comm: mdt01_053 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:14:11 2019][1666825.645965] Call Trace: [Wed Feb 6 01:14:11 2019][1666825.648593] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.655695] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.662966] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.669803] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:14:11 2019][1666825.676537] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:14:11 2019][1666825.682925] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:14:11 2019][1666825.689226] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:14:11 2019][1666825.696652] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:14:11 2019][1666825.703562] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:14:11 2019][1666825.711777] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:14:11 2019][1666825.718173] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:14:11 2019][1666825.724396] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:14:11 2019][1666825.731130] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:14:11 2019][1666825.737006] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.744113] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.751998] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:14:11 2019][1666825.758501] [] kthread+0xd1/0xe0 [Wed Feb 6 01:14:11 2019][1666825.763578] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:14:11 2019][1666825.770224] [] 0xffffffffffffffff [Wed Feb 6 01:14:15 2019][1666829.223772] LustreError: dumping log to /tmp/lustre-log.1549444631.17997 [Wed Feb 6 01:14:23 2019][1666837.415838] LustreError: dumping log to /tmp/lustre-log.1549444639.17850 [Wed Feb 6 01:14:59 2019][1666873.417623] LNet: Service thread pid 17992 completed after 1391.79s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:14:59 2019][1666873.434039] LNet: Skipped 3 previous similar messages [Wed Feb 6 01:15:25 2019][1666898.856330] LustreError: dumping log to /tmp/lustre-log.1549444700.17903 [Wed Feb 6 01:15:27 2019][1666901.281366] LustreError: 17860:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.15.5@tcp) failed to reply to blocking AST (req@ffffa053c58c4b00 x1623667887834096 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa04fa1a269c0/0x8c1de5393fcf48ad lrc: 4/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 24 type: IBT flags: 0x60200400000020 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349c0be expref: 15 pid: 17953 timeout: 1667036 lvb_type: 0 [Wed Feb 6 01:15:27 2019][1666901.323872] LustreError: 17860:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages [Wed Feb 6 01:15:27 2019][1666901.334137] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.15.5@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 01:15:27 2019][1666901.346648] LustreError: Skipped 2 previous similar messages [Wed Feb 6 01:15:27 2019][1666901.352858] LustreError: 17916:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa07b11227800 ns: mdt-fir-MDT0003_UUID lock: ffffa05b0d43ee40/0x8c1de5393fcfdff8 lrc: 1/0,0 mode: EX/EX res: [0x280000dc7:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349c0f6 expref: 4 pid: 17916 timeout: 0 lvb_type: 3 [Wed Feb 6 01:15:27 2019][1666901.387047] LustreError: 17916:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 2 previous similar messages [Wed Feb 6 01:15:29 2019][1666902.952369] LustreError: dumping log to /tmp/lustre-log.1549444705.129262 [Wed Feb 6 01:15:37 2019][1666911.144431] LustreError: dumping log to /tmp/lustre-log.1549444713.18272 [Wed Feb 6 01:16:10 2019][1666943.912695] LustreError: dumping log to /tmp/lustre-log.1549444745.17915 [Wed Feb 6 01:16:18 2019][1666952.104776] LustreError: dumping log to /tmp/lustre-log.1549444754.129228 [Wed Feb 6 01:17:04 2019][1666998.185159] Lustre: 17983:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 01:17:04 2019][1666998.185159] req@ffffa05360842400 x1624705119911664/t0(0) o36->95d24266-18da-6c0e-7cc6-3ed980208315@10.8.27.1@o2ib6:585/0 lens 528/2888 e 0 to 0 dl 1549444805 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 01:17:04 2019][1666998.214406] Lustre: 17983:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 36 previous similar messages [Wed Feb 6 01:17:10 2019][1667004.361970] Lustre: fir-MDT0003: Client 582c0bf9-0c65-cb8a-6846-32e5a6d43ca1 (at 10.9.107.43@o2ib4) reconnecting [Wed Feb 6 01:17:10 2019][1667004.362632] Lustre: fir-MDT0001: Connection restored to ed540f5a-5df6-5998-8f5c-40181564f690 (at 10.9.106.50@o2ib4) [Wed Feb 6 01:17:10 2019][1667004.362634] Lustre: Skipped 47 previous similar messages [Wed Feb 6 01:17:10 2019][1667004.388404] Lustre: Skipped 47 previous similar messages [Wed Feb 6 01:17:30 2019][1667024.271291] LNet: Service thread pid 18044 completed after 1401.99s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:17:30 2019][1667024.287718] LNet: Skipped 9 previous similar messages [Wed Feb 6 01:18:17 2019][1667071.678718] LNetError: 17870:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.15.5@tcp from [Wed Feb 6 01:18:17 2019][1667071.689323] LNetError: 17870:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 101233820 previous similar messages [Wed Feb 6 01:18:29 2019][1667083.282857] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 150s: evicting client at 10.10.15.5@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa0573db5fbc0/0x8c1de5393fe9ae7e lrc: 3/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 19 type: IBT flags: 0x60200400000020 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349c112 expref: 13 pid: 17931 timeout: 0 lvb_type: 0 [Wed Feb 6 01:18:29 2019][1667083.319999] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages [Wed Feb 6 01:18:31 2019][1667085.700840] Lustre: 17850:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549444887/real 1549444887] req@ffffa08ab2782a00 x1623667887965232/t0(0) o104->fir-MDT0001@10.10.114.15@tcp:15/16 lens 296/224 e 0 to 1 dl 1549444898 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 01:18:31 2019][1667085.728613] Lustre: 17850:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 102866935 previous similar messages [Wed Feb 6 01:19:34 2019][1667148.714344] Pid: 17909, comm: mdt02_037 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:19:34 2019][1667148.724257] Call Trace: [Wed Feb 6 01:19:34 2019][1667148.726896] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:19:34 2019][1667148.734012] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:19:34 2019][1667148.741312] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:19:34 2019][1667148.748156] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:19:34 2019][1667148.754915] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:19:34 2019][1667148.761313] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:19:34 2019][1667148.767628] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:19:34 2019][1667148.775067] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:19:34 2019][1667148.781998] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:19:34 2019][1667148.790218] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:19:34 2019][1667148.796629] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:19:34 2019][1667148.802855] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:19:34 2019][1667148.809601] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:19:35 2019][1667148.815481] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667148.822618] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667148.830508] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667148.837025] [] kthread+0xd1/0xe0 [Wed Feb 6 01:19:35 2019][1667148.842115] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:19:35 2019][1667148.848776] [] 0xffffffffffffffff [Wed Feb 6 01:19:35 2019][1667148.853971] LustreError: dumping log to /tmp/lustre-log.1549444950.17909 [Wed Feb 6 01:19:35 2019][1667149.112353] Pid: 17907, comm: mdt03_031 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:19:35 2019][1667149.122260] Call Trace: [Wed Feb 6 01:19:35 2019][1667149.124902] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.132019] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.139306] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.146154] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:19:35 2019][1667149.152900] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:19:35 2019][1667149.159302] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:19:35 2019][1667149.165625] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:19:35 2019][1667149.173063] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:19:35 2019][1667149.179998] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:19:35 2019][1667149.188215] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:19:35 2019][1667149.194623] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:19:35 2019][1667149.200850] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:19:35 2019][1667149.207597] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:19:35 2019][1667149.213468] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.220606] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.228496] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.235014] [] kthread+0xd1/0xe0 [Wed Feb 6 01:19:35 2019][1667149.240107] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:19:35 2019][1667149.246771] [] 0xffffffffffffffff [Wed Feb 6 01:19:35 2019][1667149.251969] Pid: 127206, comm: mdt01_007 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:19:35 2019][1667149.261974] Call Trace: [Wed Feb 6 01:19:35 2019][1667149.264610] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.271723] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.279008] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.285861] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:19:35 2019][1667149.292597] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:19:35 2019][1667149.298996] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:19:35 2019][1667149.305293] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:19:35 2019][1667149.312733] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:19:35 2019][1667149.319645] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:19:35 2019][1667149.327865] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:19:35 2019][1667149.334265] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:19:35 2019][1667149.340503] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:19:35 2019][1667149.347246] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:19:35 2019][1667149.353139] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.360249] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.368152] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.374653] [] kthread+0xd1/0xe0 [Wed Feb 6 01:19:35 2019][1667149.379745] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:19:35 2019][1667149.386398] [] 0xffffffffffffffff [Wed Feb 6 01:19:35 2019][1667149.391577] Pid: 17838, comm: mdt02_029 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:19:35 2019][1667149.401486] Call Trace: [Wed Feb 6 01:19:35 2019][1667149.404119] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.411217] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.418508] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.425345] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:19:35 2019][1667149.432092] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:19:35 2019][1667149.438483] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:19:35 2019][1667149.444799] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:19:35 2019][1667149.452239] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:19:35 2019][1667149.459158] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:19:35 2019][1667149.467373] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:19:35 2019][1667149.473772] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:19:35 2019][1667149.479997] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:19:35 2019][1667149.486746] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:19:35 2019][1667149.492626] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.499748] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.507637] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:19:35 2019][1667149.514155] [] kthread+0xd1/0xe0 [Wed Feb 6 01:19:35 2019][1667149.519233] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:19:35 2019][1667149.525886] [] 0xffffffffffffffff [Wed Feb 6 01:19:36 2019][1667150.712363] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 01:19:36 2019][1667150.728514] Lustre: Skipped 1 previous similar message [Wed Feb 6 01:19:36 2019][1667150.733851] LustreError: 17875:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549444652, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa05310324c80/0x8c1de5393fd02c0a lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 137 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351812c0ed2 expref: -99 pid: 17875 timeout: 0 lvb_type: 0 [Wed Feb 6 01:19:37 2019][1667150.774180] LustreError: 17875:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 97 previous similar messages [Wed Feb 6 01:19:59 2019][1667173.290545] Pid: 18166, comm: mdt03_043 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:19:59 2019][1667173.300455] Call Trace: [Wed Feb 6 01:19:59 2019][1667173.303095] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:19:59 2019][1667173.310205] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:19:59 2019][1667173.317476] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:19:59 2019][1667173.324314] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:19:59 2019][1667173.331048] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:19:59 2019][1667173.337445] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:19:59 2019][1667173.343745] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:19:59 2019][1667173.351179] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:19:59 2019][1667173.358087] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:19:59 2019][1667173.366304] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:19:59 2019][1667173.372700] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:19:59 2019][1667173.378921] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:19:59 2019][1667173.385658] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:19:59 2019][1667173.391524] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:19:59 2019][1667173.398639] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:19:59 2019][1667173.406526] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:19:59 2019][1667173.413027] [] kthread+0xd1/0xe0 [Wed Feb 6 01:19:59 2019][1667173.418113] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:19:59 2019][1667173.424761] [] 0xffffffffffffffff [Wed Feb 6 01:19:59 2019][1667173.429954] LustreError: dumping log to /tmp/lustre-log.1549444975.18166 [Wed Feb 6 01:20:01 2019][1667175.283716] Lustre: 17850:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:787s); client may timeout. req@ffffa08a4f658f00 x1624701947463776/t0(0) o36->7646fd6e-8364-5414-6410-5e73509a82fd@10.10.113.3@tcp:725/0 lens 520/424 e 0 to 0 dl 1549444190 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 01:20:01 2019][1667175.285689] LNet: Service thread pid 17903 completed after 1479.64s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:20:01 2019][1667175.285691] LNet: Skipped 4 previous similar messages [Wed Feb 6 01:20:15 2019][1667189.674675] LNet: Service thread pid 17885 was inactive for 1200.92s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 01:20:15 2019][1667189.687800] LNet: Skipped 26 previous similar messages [Wed Feb 6 01:20:15 2019][1667189.693122] LustreError: dumping log to /tmp/lustre-log.1549444991.17885 [Wed Feb 6 01:21:21 2019][1667255.212204] LustreError: dumping log to /tmp/lustre-log.1549445057.18269 [Wed Feb 6 01:24:38 2019][1667451.820794] LNet: Service thread pid 17829 was inactive for 1202.42s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 01:24:38 2019][1667451.837987] LNet: Skipped 9 previous similar messages [Wed Feb 6 01:24:38 2019][1667451.843223] Pid: 17829, comm: mdt03_023 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:24:38 2019][1667451.853134] Call Trace: [Wed Feb 6 01:24:38 2019][1667451.855770] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667451.862881] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667451.870163] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667451.877009] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:24:38 2019][1667451.883749] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:24:38 2019][1667451.890147] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:24:38 2019][1667451.896467] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:24:38 2019][1667451.903903] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:24:38 2019][1667451.910819] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:24:38 2019][1667451.919034] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:24:38 2019][1667451.925430] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:24:38 2019][1667451.931652] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:24:38 2019][1667451.938388] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:24:38 2019][1667451.944263] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667451.951390] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667451.959289] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667451.965791] [] kthread+0xd1/0xe0 [Wed Feb 6 01:24:38 2019][1667451.970878] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:24:38 2019][1667451.977525] [] 0xffffffffffffffff [Wed Feb 6 01:24:38 2019][1667451.982721] LustreError: dumping log to /tmp/lustre-log.1549445254.17829 [Wed Feb 6 01:24:38 2019][1667452.184525] Pid: 17911, comm: mdt00_039 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:24:38 2019][1667452.194440] Call Trace: [Wed Feb 6 01:24:38 2019][1667452.197081] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.204198] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.211479] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.218342] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:24:38 2019][1667452.225076] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:24:38 2019][1667452.231464] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:24:38 2019][1667452.237763] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:24:38 2019][1667452.245202] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:24:38 2019][1667452.252116] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:24:38 2019][1667452.260333] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:24:38 2019][1667452.266729] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:24:38 2019][1667452.272942] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:24:38 2019][1667452.279677] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:24:38 2019][1667452.285568] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.292694] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.300581] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.307081] [] kthread+0xd1/0xe0 [Wed Feb 6 01:24:38 2019][1667452.312170] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:24:38 2019][1667452.318816] [] 0xffffffffffffffff [Wed Feb 6 01:24:38 2019][1667452.324014] Pid: 129242, comm: mdt01_018 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:24:38 2019][1667452.334010] Call Trace: [Wed Feb 6 01:24:38 2019][1667452.336643] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.343751] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.351047] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.357895] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:24:38 2019][1667452.364629] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:24:38 2019][1667452.371025] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:24:38 2019][1667452.377327] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:24:38 2019][1667452.384752] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:24:38 2019][1667452.391669] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:24:38 2019][1667452.399887] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:24:38 2019][1667452.406282] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:24:38 2019][1667452.412503] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:24:38 2019][1667452.419262] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:24:38 2019][1667452.425141] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.432255] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.440142] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.446643] [] kthread+0xd1/0xe0 [Wed Feb 6 01:24:38 2019][1667452.451728] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:24:38 2019][1667452.458377] [] 0xffffffffffffffff [Wed Feb 6 01:24:38 2019][1667452.463557] Pid: 129227, comm: mdt01_013 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:24:38 2019][1667452.473555] Call Trace: [Wed Feb 6 01:24:38 2019][1667452.476187] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.483295] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.490582] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.497430] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:24:38 2019][1667452.504164] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:24:38 2019][1667452.510552] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:24:38 2019][1667452.516851] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:24:38 2019][1667452.524287] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:24:38 2019][1667452.531205] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:24:38 2019][1667452.539413] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:24:38 2019][1667452.545799] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:24:38 2019][1667452.552037] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:24:38 2019][1667452.558787] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:24:38 2019][1667452.564668] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.571787] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.579677] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.586178] [] kthread+0xd1/0xe0 [Wed Feb 6 01:24:38 2019][1667452.591266] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:24:38 2019][1667452.597913] [] 0xffffffffffffffff [Wed Feb 6 01:24:38 2019][1667452.603098] Pid: 18001, comm: mdt00_055 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:24:38 2019][1667452.613012] Call Trace: [Wed Feb 6 01:24:38 2019][1667452.615647] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.622753] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.630034] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.636880] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:24:38 2019][1667452.643621] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:24:38 2019][1667452.650018] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:24:38 2019][1667452.656318] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:24:38 2019][1667452.663754] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:24:38 2019][1667452.670661] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:24:38 2019][1667452.678877] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:24:38 2019][1667452.685292] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:24:38 2019][1667452.691514] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:24:38 2019][1667452.698258] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:24:38 2019][1667452.704131] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.711259] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.719145] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:24:38 2019][1667452.725644] [] kthread+0xd1/0xe0 [Wed Feb 6 01:24:38 2019][1667452.730731] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:24:38 2019][1667452.737377] [] 0xffffffffffffffff [Wed Feb 6 01:24:42 2019][1667455.916826] LustreError: dumping log to /tmp/lustre-log.1549445257.17935 [Wed Feb 6 01:24:46 2019][1667460.012860] LustreError: dumping log to /tmp/lustre-log.1549445262.129797 [Wed Feb 6 01:24:54 2019][1667468.204925] LustreError: dumping log to /tmp/lustre-log.1549445270.17804 [Wed Feb 6 01:25:03 2019][1667477.396823] LNet: Service thread pid 129227 completed after 1229.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:25:03 2019][1667477.413325] LNet: Skipped 29 previous similar messages [Wed Feb 6 01:26:05 2019][1667539.286509] LustreError: 18269:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.15.5@tcp) failed to reply to blocking AST (req@ffffa056c9529200 x1623667888184000 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa0678cdfb600/0x8c1de5393ff94971 lrc: 4/0,0 mode: PR/PR res: [0x280000dc7:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349c269 expref: 10 pid: 127206 timeout: 1667674 lvb_type: 0 [Wed Feb 6 01:26:05 2019][1667539.286518] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.15.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 01:26:05 2019][1667539.286520] LustreError: Skipped 8 previous similar messages [Wed Feb 6 01:26:05 2019][1667539.286623] LustreError: 17915:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa0681e5f7400 ns: mdt-fir-MDT0003_UUID lock: ffffa0523a34f2c0/0x8c1de5393ff9491d lrc: 5/0,0 mode: PW/PW res: [0x280000dc8:0x1:0x0].0x0 bits 0x40/0x0 rrc: 3 type: IBT flags: 0x50200000000000 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7c429 expref: 4 pid: 17915 timeout: 0 lvb_type: 0 [Wed Feb 6 01:26:05 2019][1667539.286625] LustreError: 17915:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 6 previous similar messages [Wed Feb 6 01:26:05 2019][1667539.391805] LustreError: 18269:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 9 previous similar messages [Wed Feb 6 01:27:07 2019][1667601.327011] Lustre: 129280:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/-151), not sending early reply [Wed Feb 6 01:27:07 2019][1667601.327011] req@ffffa077ba322d00 x1624705102835440/t0(0) o36->44c34e5e-d358-e5f1-f032-e5118620e81b@10.8.24.9@o2ib6:432/0 lens 520/2888 e 0 to 0 dl 1549445407 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 01:27:07 2019][1667601.356345] Lustre: 129280:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 103 previous similar messages [Wed Feb 6 01:27:12 2019][1667606.769902] Lustre: fir-MDT0003: Client 44c34e5e-d358-e5f1-f032-e5118620e81b (at 10.8.24.9@o2ib6) reconnecting [Wed Feb 6 01:27:12 2019][1667606.780100] Lustre: Skipped 87 previous similar messages [Wed Feb 6 01:27:12 2019][1667606.785617] Lustre: fir-MDT0003: Connection restored to 44c34e5e-d358-e5f1-f032-e5118620e81b (at 10.8.24.9@o2ib6) [Wed Feb 6 01:27:13 2019][1667606.796053] Lustre: Skipped 116 previous similar messages [Wed Feb 6 01:28:17 2019][1667671.683555] LNetError: 2937:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.15.6@tcp from [Wed Feb 6 01:28:17 2019][1667671.694072] LNetError: 2937:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 125451823 previous similar messages [Wed Feb 6 01:28:31 2019][1667685.705669] Lustre: 129374:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549445487/real 1549445487] req@ffffa062774a6000 x1623667888307952/t0(0) o104->fir-MDT0003@10.10.15.5@tcp:15/16 lens 296/224 e 0 to 1 dl 1549445498 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 01:28:31 2019][1667685.733351] Lustre: 129374:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 125365293 previous similar messages [Wed Feb 6 01:28:36 2019][1667690.287742] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.15.6@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa07388d269c0/0x8c1de5393ffc2dc6 lrc: 3/0,0 mode: PR/PR res: [0x280000dc8:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7c43e expref: 11 pid: 18269 timeout: 0 lvb_type: 0 [Wed Feb 6 01:28:36 2019][1667690.324699] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 8 previous similar messages [Wed Feb 6 01:29:38 2019][1667752.766216] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 01:29:38 2019][1667752.782367] Lustre: Skipped 3 previous similar messages [Wed Feb 6 01:29:38 2019][1667752.787789] LustreError: 129372:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549445254, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa089c6e9c800/0x8c1de5393ffa1ad7 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 141 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351812cf8a3 expref: -99 pid: 129372 timeout: 0 lvb_type: 0 [Wed Feb 6 01:29:39 2019][1667752.828293] LustreError: 129372:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 65 previous similar messages [Wed Feb 6 01:30:05 2019][1667779.299703] Lustre: 129253:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:151s); client may timeout. req@ffffa064710c4500 x1624702092187472/t0(0) o36->05683605-3c7a-66e7-65c2-872123284fcd@10.10.114.15@tcp:455/0 lens 520/424 e 0 to 0 dl 1549445430 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 01:36:09 2019][1668143.291375] LustreError: 127197:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.15.5@tcp) failed to reply to blocking AST (req@ffffa07750ad8900 x1623667888557408 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa0605847aac0/0x8c1de5394000b581 lrc: 4/0,0 mode: PR/PR res: [0x280000dc7:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349c2f5 expref: 10 pid: 18907 timeout: 1668278 lvb_type: 0 [Wed Feb 6 01:36:09 2019][1668143.333789] LustreError: 127197:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 8 previous similar messages [Wed Feb 6 01:36:09 2019][1668143.344145] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.15.5@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 01:36:09 2019][1668143.356661] LustreError: Skipped 9 previous similar messages [Wed Feb 6 01:36:09 2019][1668143.362608] LustreError: 127197:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa0753bdb1c00 ns: mdt-fir-MDT0003_UUID lock: ffffa062916aaac0/0x8c1de5394000b59d lrc: 1/0,0 mode: --/PW res: [0x280000dc7:0x2:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x54a01000000000 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349c303 expref: 4 pid: 127197 timeout: 0 lvb_type: 0 [Wed Feb 6 01:36:09 2019][1668143.397063] LustreError: 127197:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 8 previous similar messages [Wed Feb 6 01:37:18 2019][1668212.423432] Lustre: fir-MDT0001: Connection restored to b29100f0-7f9b-f4f6-2c85-7505f2641dbf (at 10.8.6.22@o2ib6) [Wed Feb 6 01:37:18 2019][1668212.433865] Lustre: Skipped 70 previous similar messages [Wed Feb 6 01:38:17 2019][1668271.688392] LNetError: 18001:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.15.5@tcp from [Wed Feb 6 01:38:17 2019][1668271.699005] LNetError: 18001:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 125620890 previous similar messages [Wed Feb 6 01:38:31 2019][1668285.710502] Lustre: 18224:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549446087/real 1549446087] req@ffffa074b9ccec00 x1623667888688368/t0(0) o104->fir-MDT0003@10.10.15.5@tcp:15/16 lens 296/224 e 0 to 1 dl 1549446098 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 01:38:31 2019][1668285.738101] Lustre: 18224:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 125110748 previous similar messages [Wed Feb 6 01:39:04 2019][1668318.292805] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.15.5@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa0674e8d0fc0/0x8c1de539400629fa lrc: 3/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 24 type: IBT flags: 0x60200400000020 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349c373 expref: 15 pid: 127066 timeout: 0 lvb_type: 0 [Wed Feb 6 01:39:04 2019][1668318.330018] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 9 previous similar messages [Wed Feb 6 01:40:07 2019][1668381.428285] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 01:40:07 2019][1668381.433299] LustreError: 17860:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549445883, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0001 lock: ffffa08a17fc0000/0x8c1de53940014fab lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 7 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935181b5cd36 expref: -99 pid: 17860 timeout: 0 lvb_type: 0 [Wed Feb 6 01:40:07 2019][1668381.433302] LustreError: 17860:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 136 previous similar messages [Wed Feb 6 01:40:07 2019][1668381.495811] Lustre: Skipped 3 previous similar messages [Wed Feb 6 01:40:44 2019][1668417.972910] Lustre: fir-MDT0003: haven't heard from client 7f648de4-ccea-3fb5-cc96-0c4479d5ecdb (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa058c2479800, cur 1549446220 expire 1549446070 last 1549445993 [Wed Feb 6 01:40:44 2019][1668417.994799] Lustre: Skipped 1 previous similar message [Wed Feb 6 01:46:38 2019][1668772.296439] LustreError: 129266:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.15.6@tcp) failed to reply to blocking AST (req@ffffa056c9528000 x1623667889236240 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa067e91e9200/0x8c1de5394008855f lrc: 4/0,0 mode: CR/CR res: [0x280000dd7:0x1:0x0].0x0 bits 0x9/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7c548 expref: 14 pid: 17838 timeout: 1668907 lvb_type: 0 [Wed Feb 6 01:46:38 2019][1668772.338762] LustreError: 129266:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages [Wed Feb 6 01:46:38 2019][1668772.349115] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.15.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 01:46:38 2019][1668772.361640] LustreError: Skipped 3 previous similar messages [Wed Feb 6 01:46:38 2019][1668772.367823] LustreError: 129266:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa08a9773e400 ns: mdt-fir-MDT0003_UUID lock: ffffa07a4f871f80/0x8c1de5394018c4fc lrc: 1/0,0 mode: EX/EX res: [0x280000dd7:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7c5cd expref: 4 pid: 129266 timeout: 0 lvb_type: 3 [Wed Feb 6 01:46:38 2019][1668772.402204] LustreError: 129266:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 2 previous similar messages [Wed Feb 6 01:47:19 2019][1668813.171483] Lustre: fir-MDT0001: Connection restored to bd4172ee-b21a-171f-590d-07ab0a2614bb (at 10.8.12.25@o2ib6) [Wed Feb 6 01:47:19 2019][1668813.181999] Lustre: Skipped 259 previous similar messages [Wed Feb 6 01:48:17 2019][1668871.693234] LNetError: 19083:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.15.6@tcp from [Wed Feb 6 01:48:17 2019][1668871.703841] LNetError: 19083:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 99525656 previous similar messages [Wed Feb 6 01:48:31 2019][1668885.715345] Lustre: 17834:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549446687/real 1549446687] req@ffffa07436f70000 x1623667889466224/t0(0) o104->fir-MDT0003@10.10.15.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549446698 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 01:48:31 2019][1668885.742944] Lustre: 17834:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 99554998 previous similar messages [Wed Feb 6 01:49:10 2019][1668924.297684] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.15.6@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa0641a6b18c0/0x8c1de539401f6659 lrc: 3/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 27 type: IBT flags: 0x60200400000020 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7c5e9 expref: 13 pid: 17815 timeout: 0 lvb_type: 0 [Wed Feb 6 01:49:10 2019][1668924.334814] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages [Wed Feb 6 01:50:08 2019][1668981.903131] LustreError: 129243:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549446483, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa062ef2ebcc0/0x8c1de539401b7660 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 137 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935182116966 expref: -99 pid: 129243 timeout: 0 lvb_type: 0 [Wed Feb 6 01:50:08 2019][1668981.943644] LustreError: 129243:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 180 previous similar messages [Wed Feb 6 01:50:11 2019][1668985.462153] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 01:50:11 2019][1668985.478309] Lustre: Skipped 2 previous similar messages [Wed Feb 6 01:52:39 2019][1669133.242370] Lustre: 18271:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 01:52:39 2019][1669133.242370] req@ffffa089c4703600 x1624700571785888/t0(0) o36->9f0fc7f1-3017-a377-d5ca-3f45fac7d96c@10.9.106.2@o2ib4:455/0 lens 528/2888 e 0 to 0 dl 1549446940 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 01:52:39 2019][1669133.271697] Lustre: 18271:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 31 previous similar messages [Wed Feb 6 01:52:45 2019][1669139.467235] Lustre: fir-MDT0003: Client 9f0fc7f1-3017-a377-d5ca-3f45fac7d96c (at 10.9.106.2@o2ib4) reconnecting [Wed Feb 6 01:52:45 2019][1669139.477497] Lustre: Skipped 19 previous similar messages [Wed Feb 6 01:54:05 2019][1669219.387059] Lustre: 19059:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 01:54:05 2019][1669219.387059] req@ffffa064f0267500 x1624672917332176/t0(0) o36->d405c92c-c32a-78c2-b321-55f4db7bb864@10.9.102.14@o2ib4:541/0 lens 536/2888 e 0 to 0 dl 1549447026 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 01:54:05 2019][1669219.416475] Lustre: 19059:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 76 previous similar messages [Wed Feb 6 01:54:11 2019][1669225.361535] Lustre: fir-MDT0003: Client d405c92c-c32a-78c2-b321-55f4db7bb864 (at 10.9.102.14@o2ib4) reconnecting [Wed Feb 6 01:54:11 2019][1669225.371882] Lustre: Skipped 37 previous similar messages [Wed Feb 6 01:55:52 2019][1669325.960019] Lustre: fir-MDT0003: haven't heard from client b1ae38e6-4e9c-51ec-0d30-af325c63f3ef (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa05865ec8c00, cur 1549447128 expire 1549446978 last 1549446901 [Wed Feb 6 01:55:52 2019][1669325.981806] Lustre: Skipped 1 previous similar message [Wed Feb 6 01:56:43 2019][1669377.301323] LustreError: 18262:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.15.6@tcp) failed to reply to blocking AST (req@ffffa078e0088600 x1623667890005824 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa053acb03f00/0x8c1de539402bcd18 lrc: 4/0,0 mode: PR/PR res: [0x280000dd7:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7c71d expref: 10 pid: 129258 timeout: 1669512 lvb_type: 0 [Wed Feb 6 01:56:43 2019][1669377.301328] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.15.5@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 01:56:43 2019][1669377.301330] LustreError: Skipped 7 previous similar messages [Wed Feb 6 01:56:43 2019][1669377.301441] LustreError: 19097:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa067bd1c4000 ns: mdt-fir-MDT0003_UUID lock: ffffa06383f4a640/0x8c1de539402bcbeb lrc: 3/0,0 mode: PW/PW res: [0x280000dd5:0x1:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x50200000000000 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349c509 expref: 3 pid: 19097 timeout: 0 lvb_type: 0 [Wed Feb 6 01:56:43 2019][1669377.301442] LustreError: 19097:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 4 previous similar messages [Wed Feb 6 01:56:43 2019][1669377.406591] LustreError: 18262:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 8 previous similar messages [Wed Feb 6 01:57:30 2019][1669424.169966] Lustre: fir-MDT0001: Connection restored to b1ae38e6-4e9c-51ec-0d30-af325c63f3ef (at 10.8.4.15@o2ib6) [Wed Feb 6 01:57:30 2019][1669424.180403] Lustre: Skipped 167 previous similar messages [Wed Feb 6 01:57:42 2019][1669435.836800] Lustre: 19097:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 01:57:42 2019][1669435.836800] req@ffffa06242bbc500 x1624699178553840/t0(0) o36->c3afa774-0e39-3459-0af2-ddf264214c5b@10.9.104.54@o2ib4:2/0 lens 520/2888 e 0 to 0 dl 1549447242 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 01:57:42 2019][1669435.866046] Lustre: 19097:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 21 previous similar messages [Wed Feb 6 01:58:17 2019][1669471.698068] LNetError: 17975:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.15.5@tcp from [Wed Feb 6 01:58:17 2019][1669471.708676] LNetError: 17975:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 98035181 previous similar messages [Wed Feb 6 01:58:31 2019][1669485.720183] Lustre: 18262:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549447287/real 1549447287] req@ffffa078e0088600 x1623667890236912/t0(0) o104->fir-MDT0003@10.10.15.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549447298 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 01:58:31 2019][1669485.747777] Lustre: 18262:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 98536094 previous similar messages [Wed Feb 6 01:58:54 2019][1669508.115893] Lustre: fir-MDT0003: Client 79d8a0e0-7e23-2184-06c2-32c9e0b34341 (at 10.9.106.24@o2ib4) reconnecting [Wed Feb 6 01:58:54 2019][1669508.126238] Lustre: Skipped 13 previous similar messages [Wed Feb 6 01:59:14 2019][1669528.302563] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.15.6@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa06383f4f500/0x8c1de5394035c181 lrc: 3/0,0 mode: PR/PR res: [0x280000dd7:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7c740 expref: 11 pid: 19097 timeout: 0 lvb_type: 0 [Wed Feb 6 01:59:14 2019][1669528.339523] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 4 previous similar messages [Wed Feb 6 02:00:15 2019][1669589.334028] LustreError: 17963:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549447091, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa089f3b1a880/0x8c1de539402d53f2 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 132 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935182294d36 expref: -99 pid: 17963 timeout: 0 lvb_type: 0 [Wed Feb 6 02:00:15 2019][1669589.374360] LustreError: 17963:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 63 previous similar messages [Wed Feb 6 02:02:46 2019][1669740.322241] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 02:02:46 2019][1669740.338396] Lustre: Skipped 2 previous similar messages [Wed Feb 6 02:06:47 2019][1669981.306189] LustreError: 17975:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.15.6@tcp) failed to reply to blocking AST (req@ffffa061b49fb600 x1623667890973344 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa071ded08480/0x8c1de5394058510a lrc: 4/0,0 mode: PR/PR res: [0x280000dd7:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7c7a9 expref: 10 pid: 127297 timeout: 1670116 lvb_type: 0 [Wed Feb 6 02:06:47 2019][1669981.348608] LustreError: 17975:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 6 previous similar messages [Wed Feb 6 02:06:47 2019][1669981.358875] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.15.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 02:06:47 2019][1669981.371385] LustreError: Skipped 7 previous similar messages [Wed Feb 6 02:06:47 2019][1669981.377275] LustreError: 17910:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa05aaed35d00 x1623667891264352/t0(0) o104->fir-MDT0003@10.10.15.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549447794 ref 1 fl Rpc:eX/2/ffffffff rc -5/-1 [Wed Feb 6 02:06:47 2019][1669981.377329] LustreError: 17975:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa062a5258c00 ns: mdt-fir-MDT0003_UUID lock: ffffa04b87247080/0x8c1de53940585142 lrc: 1/0,0 mode: --/PW res: [0x280000dd7:0x2:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x54a01000000000 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7c7b7 expref: 4 pid: 17975 timeout: 0 lvb_type: 0 [Wed Feb 6 02:06:47 2019][1669981.377330] LustreError: 17975:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 7 previous similar messages [Wed Feb 6 02:08:17 2019][1670071.702905] LNetError: 126633:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.15.6@tcp from [Wed Feb 6 02:08:17 2019][1670071.713594] LNetError: 126633:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 101885486 previous similar messages [Wed Feb 6 02:08:31 2019][1670085.725031] Lustre: 127501:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549447887/real 1549447887] req@ffffa056c952da00 x1623667891271120/t0(0) o104->fir-MDT0003@10.10.15.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549447898 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 02:08:31 2019][1670085.752740] Lustre: 127501:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 101358375 previous similar messages [Wed Feb 6 02:09:18 2019][1670132.307424] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.15.6@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa06a7db445c0/0x8c1de53940695969 lrc: 3/0,0 mode: PR/PR res: [0x280000dd7:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7c7cc expref: 11 pid: 17973 timeout: 0 lvb_type: 0 [Wed Feb 6 02:09:18 2019][1670132.344410] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 6 previous similar messages [Wed Feb 6 02:09:18 2019][1670132.354953] Lustre: fir-MDT0003: Connection restored to ffafe5be-09a9-59b7-d26e-f32fde550fcb (at 10.10.15.6@tcp) [Wed Feb 6 02:09:18 2019][1670132.365321] Lustre: Skipped 21 previous similar messages [Wed Feb 6 02:12:50 2019][1670344.510126] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 02:12:50 2019][1670344.526293] LustreError: 17963:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549447846, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa08a2b741b00/0x8c1de539406e7543 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 120 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518255b766 expref: -99 pid: 17963 timeout: 0 lvb_type: 0 [Wed Feb 6 02:12:50 2019][1670344.566650] LustreError: 17963:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 25 previous similar messages [Wed Feb 6 02:15:32 2019][1670506.437417] LNet: Service thread pid 17999 was inactive for 302.74s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 02:15:32 2019][1670506.454522] LNet: Skipped 4 previous similar messages [Wed Feb 6 02:15:32 2019][1670506.459751] Pid: 17999, comm: mdt02_042 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:15:32 2019][1670506.469664] Call Trace: [Wed Feb 6 02:15:32 2019][1670506.472306] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:15:32 2019][1670506.479436] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:15:32 2019][1670506.486717] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:15:32 2019][1670506.493579] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:15:32 2019][1670506.500314] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:15:32 2019][1670506.506728] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:15:32 2019][1670506.513029] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:15:32 2019][1670506.520482] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:15:32 2019][1670506.527406] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:15:32 2019][1670506.535654] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:15:32 2019][1670506.542046] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:15:32 2019][1670506.548260] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:15:32 2019][1670506.554993] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:15:32 2019][1670506.560876] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:15:32 2019][1670506.568004] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:15:32 2019][1670506.575905] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:15:32 2019][1670506.582408] [] kthread+0xd1/0xe0 [Wed Feb 6 02:15:32 2019][1670506.587508] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:15:32 2019][1670506.594160] [] 0xffffffffffffffff [Wed Feb 6 02:15:32 2019][1670506.599382] LustreError: dumping log to /tmp/lustre-log.1549448308.17999 [Wed Feb 6 02:15:33 2019][1670507.300825] Pid: 129258, comm: mdt00_017 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:15:33 2019][1670507.310846] Call Trace: [Wed Feb 6 02:15:33 2019][1670507.313489] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:15:33 2019][1670507.320618] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:15:33 2019][1670507.327915] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:15:33 2019][1670507.334755] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:15:33 2019][1670507.341504] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:15:33 2019][1670507.347891] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:15:33 2019][1670507.354216] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:15:33 2019][1670507.361669] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:15:33 2019][1670507.368596] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:15:33 2019][1670507.376827] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:15:33 2019][1670507.383242] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:15:33 2019][1670507.389458] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:15:33 2019][1670507.396200] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:15:33 2019][1670507.402078] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:15:33 2019][1670507.409218] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:15:33 2019][1670507.417105] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:15:33 2019][1670507.423622] [] kthread+0xd1/0xe0 [Wed Feb 6 02:15:33 2019][1670507.428727] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:15:33 2019][1670507.435399] [] 0xffffffffffffffff [Wed Feb 6 02:15:36 2019][1670510.533453] Pid: 17978, comm: mdt01_047 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:15:36 2019][1670510.543366] Call Trace: [Wed Feb 6 02:15:36 2019][1670510.546011] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:15:36 2019][1670510.553132] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:15:36 2019][1670510.560428] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:15:36 2019][1670510.567275] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:15:36 2019][1670510.574032] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:15:36 2019][1670510.580432] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:15:36 2019][1670510.586758] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:15:36 2019][1670510.594196] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:15:36 2019][1670510.601136] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:15:36 2019][1670510.609353] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:15:36 2019][1670510.615767] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:15:36 2019][1670510.621989] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:15:36 2019][1670510.628748] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:15:36 2019][1670510.634626] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:15:36 2019][1670510.641763] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:15:36 2019][1670510.649654] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:15:36 2019][1670510.656200] [] kthread+0xd1/0xe0 [Wed Feb 6 02:15:36 2019][1670510.661294] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:15:36 2019][1670510.667955] [] 0xffffffffffffffff [Wed Feb 6 02:15:36 2019][1670510.673162] LustreError: dumping log to /tmp/lustre-log.1549448312.17978 [Wed Feb 6 02:15:36 2019][1670510.680618] Pid: 17832, comm: mdt01_040 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:15:36 2019][1670510.690557] Call Trace: [Wed Feb 6 02:15:36 2019][1670510.693192] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:15:36 2019][1670510.700293] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:15:36 2019][1670510.707581] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:15:36 2019][1670510.714419] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:15:36 2019][1670510.721168] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:15:36 2019][1670510.727558] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:15:36 2019][1670510.733876] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:15:36 2019][1670510.741305] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:15:36 2019][1670510.748226] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:15:36 2019][1670510.756446] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:15:36 2019][1670510.762845] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:15:36 2019][1670510.769072] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:15:36 2019][1670510.775820] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:15:36 2019][1670510.781690] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:15:36 2019][1670510.788826] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:15:36 2019][1670510.796739] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:15:36 2019][1670510.803247] [] kthread+0xd1/0xe0 [Wed Feb 6 02:15:36 2019][1670510.808324] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:15:36 2019][1670510.814978] [] 0xffffffffffffffff [Wed Feb 6 02:15:36 2019][1670510.820158] Pid: 17805, comm: mdt03_015 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:15:36 2019][1670510.830085] Call Trace: [Wed Feb 6 02:15:36 2019][1670510.832719] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:15:37 2019][1670510.839833] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:15:37 2019][1670510.847132] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:15:37 2019][1670510.853973] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:15:37 2019][1670510.860705] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:15:37 2019][1670510.867092] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:15:37 2019][1670510.873393] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:15:37 2019][1670510.880853] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:15:37 2019][1670510.887771] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:15:37 2019][1670510.895980] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:15:37 2019][1670510.902383] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:15:37 2019][1670510.908606] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:15:37 2019][1670510.915358] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:15:37 2019][1670510.921244] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:15:37 2019][1670510.928367] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:15:37 2019][1670510.936256] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:15:37 2019][1670510.942769] [] kthread+0xd1/0xe0 [Wed Feb 6 02:15:37 2019][1670510.947850] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:15:37 2019][1670510.954511] [] 0xffffffffffffffff [Wed Feb 6 02:15:43 2019][1670517.701522] LNet: Service thread pid 127198 was inactive for 302.96s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 02:15:43 2019][1670517.714641] LNet: Skipped 10 previous similar messages [Wed Feb 6 02:15:43 2019][1670517.719963] LustreError: dumping log to /tmp/lustre-log.1549448319.127198 [Wed Feb 6 02:15:56 2019][1670529.989609] LustreError: dumping log to /tmp/lustre-log.1549448332.18272 [Wed Feb 6 02:16:15 2019][1670549.445779] LustreError: dumping log to /tmp/lustre-log.1549448351.129796 [Wed Feb 6 02:16:27 2019][1670561.733868] LustreError: dumping log to /tmp/lustre-log.1549448363.17830 [Wed Feb 6 02:16:39 2019][1670572.997956] LustreError: dumping log to /tmp/lustre-log.1549448375.17900 [Wed Feb 6 02:16:41 2019][1670575.045977] LustreError: dumping log to /tmp/lustre-log.1549448377.126632 [Wed Feb 6 02:16:44 2019][1670578.118000] LustreError: dumping log to /tmp/lustre-log.1549448380.17889 [Wed Feb 6 02:16:46 2019][1670580.166012] LustreError: dumping log to /tmp/lustre-log.1549448382.127214 [Wed Feb 6 02:16:59 2019][1670593.478123] LNet: Service thread pid 17958 was inactive for 302.75s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 02:16:59 2019][1670593.491161] LNet: Skipped 8 previous similar messages [Wed Feb 6 02:16:59 2019][1670593.496393] LustreError: dumping log to /tmp/lustre-log.1549448395.17958 [Wed Feb 6 02:17:16 2019][1670609.862254] LustreError: dumping log to /tmp/lustre-log.1549448411.129595 [Wed Feb 6 02:17:17 2019][1670610.886264] LustreError: dumping log to /tmp/lustre-log.1549448412.129280 [Wed Feb 6 02:17:21 2019][1670614.982295] LustreError: dumping log to /tmp/lustre-log.1549448417.17890 [Wed Feb 6 02:17:23 2019][1670617.030317] LustreError: dumping log to /tmp/lustre-log.1549448419.847 [Wed Feb 6 02:17:24 2019][1670618.054318] LustreError: dumping log to /tmp/lustre-log.1549448420.129264 [Wed Feb 6 02:17:40 2019][1670634.438451] LustreError: dumping log to /tmp/lustre-log.1549448436.127205 [Wed Feb 6 02:17:48 2019][1670642.630523] LustreError: dumping log to /tmp/lustre-log.1549448444.126631 [Wed Feb 6 02:17:52 2019][1670646.726552] LustreError: dumping log to /tmp/lustre-log.1549448448.126625 [Wed Feb 6 02:18:05 2019][1670659.014648] LustreError: dumping log to /tmp/lustre-log.1549448461.14718 [Wed Feb 6 02:19:39 2019][1670753.223408] LNet: Service thread pid 129244 was inactive for 557.13s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 02:19:39 2019][1670753.236530] LNet: Skipped 15 previous similar messages [Wed Feb 6 02:19:39 2019][1670753.241853] LustreError: dumping log to /tmp/lustre-log.1549448555.129244 [Wed Feb 6 02:20:16 2019][1670790.471725] Lustre: 17885:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 02:20:16 2019][1670790.471725] req@ffffa08a4c6f2400 x1624671894697920/t0(0) o36->72ef71e7-ecde-a7f2-d85d-452a22011f5b@10.9.101.14@o2ib4:602/0 lens 520/2888 e 13 to 0 dl 1549448597 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 02:20:16 2019][1670790.500973] Lustre: 17885:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 13 previous similar messages [Wed Feb 6 02:20:22 2019][1670796.444456] Lustre: fir-MDT0003: Client 2608d790-9724-41e2-f5f3-2c19b5d91363 (at 10.9.105.69@o2ib4) reconnecting [Wed Feb 6 02:20:22 2019][1670796.450728] Lustre: fir-MDT0003: Connection restored to 282817f5-6ae5-e49a-959f-04d16934f700 (at 10.9.101.12@o2ib4) [Wed Feb 6 02:20:22 2019][1670796.450731] Lustre: Skipped 8 previous similar messages [Wed Feb 6 02:20:22 2019][1670796.470826] Lustre: Skipped 7 previous similar messages [Wed Feb 6 02:20:25 2019][1670799.326830] LNet: Service thread pid 14718 completed after 603.93s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 02:20:25 2019][1670799.343161] LNet: Skipped 4 previous similar messages [Wed Feb 6 02:22:47 2019][1670941.704941] Lustre: 17870:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 02:22:47 2019][1670941.704941] req@ffffa053f67a6600 x1624673156519152/t0(0) o36->4882d594-a454-41bf-1d8d-94b894128777@10.9.102.7@o2ib4:753/0 lens 536/2888 e 2 to 0 dl 1549448748 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 02:22:47 2019][1670941.734008] Lustre: 17870:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 13 previous similar messages [Wed Feb 6 02:22:53 2019][1670947.432357] Lustre: fir-MDT0003: Client 95d24266-18da-6c0e-7cc6-3ed980208315 (at 10.8.27.1@o2ib6) reconnecting [Wed Feb 6 02:22:53 2019][1670947.442529] Lustre: Skipped 15 previous similar messages [Wed Feb 6 02:22:54 2019][1670948.351982] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 02:22:54 2019][1670948.368144] Lustre: Skipped 3 previous similar messages [Wed Feb 6 02:22:54 2019][1670948.373566] LustreError: 17975:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549448450, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa056bf537980/0x8c1de539409bb28b lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 126 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518283a35a expref: -99 pid: 17975 timeout: 0 lvb_type: 0 [Wed Feb 6 02:22:54 2019][1670948.413917] LustreError: 17975:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 138 previous similar messages [Wed Feb 6 02:23:00 2019][1670953.929027] LNet: Service thread pid 129245 was inactive for 604.48s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 02:23:00 2019][1670953.946227] LNet: Skipped 4 previous similar messages [Wed Feb 6 02:23:00 2019][1670953.951457] Pid: 129245, comm: mdt00_014 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:23:00 2019][1670953.961459] Call Trace: [Wed Feb 6 02:23:00 2019][1670953.964102] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:23:00 2019][1670953.971231] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:23:00 2019][1670953.978512] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:23:00 2019][1670953.985377] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:23:00 2019][1670953.992111] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:23:00 2019][1670953.998512] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:23:00 2019][1670954.004839] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:23:00 2019][1670954.012294] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:23:00 2019][1670954.019212] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:23:00 2019][1670954.027457] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:23:00 2019][1670954.033851] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:23:00 2019][1670954.040088] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:23:00 2019][1670954.046823] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:23:00 2019][1670954.052705] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:23:00 2019][1670954.059835] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:23:00 2019][1670954.067736] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:23:00 2019][1670954.074237] [] kthread+0xd1/0xe0 [Wed Feb 6 02:23:00 2019][1670954.079339] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:23:00 2019][1670954.085989] [] 0xffffffffffffffff [Wed Feb 6 02:23:00 2019][1670954.091214] LustreError: dumping log to /tmp/lustre-log.1549448756.129245 [Wed Feb 6 02:23:02 2019][1670955.977046] Pid: 17970, comm: mdt00_051 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:23:02 2019][1670955.986959] Call Trace: [Wed Feb 6 02:23:02 2019][1670955.989605] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670955.996740] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.004040] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.010892] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:23:02 2019][1670956.017650] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:23:02 2019][1670956.024050] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:23:02 2019][1670956.030373] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:23:02 2019][1670956.037828] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:23:02 2019][1670956.044759] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:23:02 2019][1670956.052980] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:23:02 2019][1670956.059390] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:23:02 2019][1670956.065615] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:23:02 2019][1670956.072372] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:23:02 2019][1670956.078252] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.085389] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.093282] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.099804] [] kthread+0xd1/0xe0 [Wed Feb 6 02:23:02 2019][1670956.104894] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:23:02 2019][1670956.111571] [] 0xffffffffffffffff [Wed Feb 6 02:23:02 2019][1670956.116769] LustreError: dumping log to /tmp/lustre-log.1549448758.17970 [Wed Feb 6 02:23:02 2019][1670956.124227] Pid: 18154, comm: mdt03_041 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:23:02 2019][1670956.134166] Call Trace: [Wed Feb 6 02:23:02 2019][1670956.136802] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.143911] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.151194] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.158038] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:23:02 2019][1670956.164789] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:23:02 2019][1670956.171191] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:23:02 2019][1670956.177508] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:23:02 2019][1670956.184948] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:23:02 2019][1670956.191869] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:23:02 2019][1670956.200088] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:23:02 2019][1670956.206492] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:23:02 2019][1670956.212716] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:23:02 2019][1670956.219463] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:23:02 2019][1670956.225337] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.232477] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.240376] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.246880] [] kthread+0xd1/0xe0 [Wed Feb 6 02:23:02 2019][1670956.251957] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:23:02 2019][1670956.258631] [] 0xffffffffffffffff [Wed Feb 6 02:23:02 2019][1670956.263809] Pid: 126628, comm: mdt01_002 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:23:02 2019][1670956.273824] Call Trace: [Wed Feb 6 02:23:02 2019][1670956.276460] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.283583] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.290871] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.297728] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:23:02 2019][1670956.304477] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:23:02 2019][1670956.310883] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:23:02 2019][1670956.317183] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:23:02 2019][1670956.324627] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:23:02 2019][1670956.331539] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:23:02 2019][1670956.339759] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:23:02 2019][1670956.346150] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:23:02 2019][1670956.352398] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:23:02 2019][1670956.359131] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:23:02 2019][1670956.365018] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.372137] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.380048] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:23:02 2019][1670956.386553] [] kthread+0xd1/0xe0 [Wed Feb 6 02:23:02 2019][1670956.391656] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:23:02 2019][1670956.398297] [] 0xffffffffffffffff [Wed Feb 6 02:23:04 2019][1670958.025060] Pid: 126626, comm: mdt01_000 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:23:04 2019][1670958.035058] Call Trace: [Wed Feb 6 02:23:04 2019][1670958.037702] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:23:04 2019][1670958.044831] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:23:04 2019][1670958.052131] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:23:04 2019][1670958.058992] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:23:04 2019][1670958.065754] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:23:04 2019][1670958.072150] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:23:04 2019][1670958.078474] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:23:04 2019][1670958.085911] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:23:04 2019][1670958.092842] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:23:04 2019][1670958.101063] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:23:04 2019][1670958.107471] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:23:04 2019][1670958.113698] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:23:04 2019][1670958.120448] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:23:04 2019][1670958.126342] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:23:04 2019][1670958.133481] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:23:04 2019][1670958.141371] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:23:04 2019][1670958.147888] [] kthread+0xd1/0xe0 [Wed Feb 6 02:23:04 2019][1670958.152976] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:23:04 2019][1670958.159639] [] 0xffffffffffffffff [Wed Feb 6 02:23:04 2019][1670958.164833] LustreError: dumping log to /tmp/lustre-log.1549448760.126626 [Wed Feb 6 02:23:06 2019][1670960.073085] LustreError: dumping log to /tmp/lustre-log.1549448762.17910 [Wed Feb 6 02:23:12 2019][1670966.217132] LustreError: dumping log to /tmp/lustre-log.1549448768.17853 [Wed Feb 6 02:23:14 2019][1670968.265144] LustreError: dumping log to /tmp/lustre-log.1549448770.129372 [Wed Feb 6 02:23:16 2019][1670970.313159] LustreError: dumping log to /tmp/lustre-log.1549448772.129243 [Wed Feb 6 02:23:22 2019][1670976.457232] LustreError: dumping log to /tmp/lustre-log.1549448778.129228 [Wed Feb 6 02:23:34 2019][1670988.745312] LustreError: dumping log to /tmp/lustre-log.1549448790.129250 [Wed Feb 6 02:23:39 2019][1670992.841339] LustreError: dumping log to /tmp/lustre-log.1549448794.17968 [Wed Feb 6 02:23:41 2019][1670994.889355] LustreError: dumping log to /tmp/lustre-log.1549448796.18566 [Wed Feb 6 02:24:01 2019][1671015.369524] LustreError: dumping log to /tmp/lustre-log.1549448817.129248 [Wed Feb 6 02:24:04 2019][1671018.057557] Lustre: 127501:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 02:24:04 2019][1671018.057557] req@ffffa07602a02400 x1624699659673728/t0(0) o36->f34aebb6-59fc-4cef-28dc-df15ded97223@10.9.105.62@o2ib4:75/0 lens 520/2888 e 1 to 0 dl 1549448825 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 02:24:04 2019][1671018.086715] Lustre: 127501:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 41 previous similar messages [Wed Feb 6 02:24:07 2019][1671021.513571] LustreError: dumping log to /tmp/lustre-log.1549448823.17957 [Wed Feb 6 02:24:10 2019][1671024.564057] Lustre: fir-MDT0003: Client f34aebb6-59fc-4cef-28dc-df15ded97223 (at 10.9.105.62@o2ib4) reconnecting [Wed Feb 6 02:24:10 2019][1671024.574405] Lustre: Skipped 37 previous similar messages [Wed Feb 6 02:24:17 2019][1671031.753655] LustreError: dumping log to /tmp/lustre-log.1549448833.17834 [Wed Feb 6 02:24:26 2019][1671039.945722] LustreError: dumping log to /tmp/lustre-log.1549448841.129806 [Wed Feb 6 02:24:34 2019][1671048.137787] LustreError: dumping log to /tmp/lustre-log.1549448850.17940 [Wed Feb 6 02:24:46 2019][1671060.425890] LNet: Service thread pid 127273 was inactive for 636.91s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 02:24:46 2019][1671060.439035] LNet: Skipped 20 previous similar messages [Wed Feb 6 02:24:46 2019][1671060.444371] LustreError: dumping log to /tmp/lustre-log.1549448862.127273 [Wed Feb 6 02:25:02 2019][1671076.810019] LustreError: dumping log to /tmp/lustre-log.1549448878.17935 [Wed Feb 6 02:25:07 2019][1671080.906050] LustreError: dumping log to /tmp/lustre-log.1549448882.18260 [Wed Feb 6 02:25:09 2019][1671082.954072] LustreError: dumping log to /tmp/lustre-log.1549448884.129252 [Wed Feb 6 02:25:11 2019][1671085.002087] LustreError: dumping log to /tmp/lustre-log.1549448887.126633 [Wed Feb 6 02:25:15 2019][1671089.098151] LustreError: dumping log to /tmp/lustre-log.1549448891.17976 [Wed Feb 6 02:25:17 2019][1671091.146147] LustreError: dumping log to /tmp/lustre-log.1549448893.17997 [Wed Feb 6 02:25:27 2019][1671101.449841] LNet: Service thread pid 129245 completed after 752.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 02:25:27 2019][1671101.466261] LNet: Skipped 44 previous similar messages [Wed Feb 6 02:26:00 2019][1671134.154484] LustreError: dumping log to /tmp/lustre-log.1549448936.129253 [Wed Feb 6 02:26:06 2019][1671140.298534] LustreError: dumping log to /tmp/lustre-log.1549448942.127248 [Wed Feb 6 02:26:53 2019][1671187.146922] Lustre: 126624:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 02:26:53 2019][1671187.146922] req@ffffa051d6bfbc00 x1624699695601120/t0(0) o36->86d44073-2a86-f18b-4a0f-e98051cdbb2e@10.9.105.51@o2ib4:244/0 lens 520/2888 e 1 to 0 dl 1549448994 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 02:26:53 2019][1671187.176185] Lustre: 126624:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 22 previous similar messages [Wed Feb 6 02:26:59 2019][1671193.332830] Lustre: fir-MDT0003: Client 86d44073-2a86-f18b-4a0f-e98051cdbb2e (at 10.9.105.51@o2ib4) reconnecting [Wed Feb 6 02:26:59 2019][1671193.343180] Lustre: Skipped 22 previous similar messages [Wed Feb 6 02:27:03 2019][1671197.643000] LustreError: dumping log to /tmp/lustre-log.1549448999.128085 [Wed Feb 6 02:27:05 2019][1671199.691009] LustreError: dumping log to /tmp/lustre-log.1549449001.127247 [Wed Feb 6 02:28:07 2019][1671261.335298] LNet: Service thread pid 127212 completed after 763.36s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 02:28:07 2019][1671261.351744] LNet: Skipped 13 previous similar messages [Wed Feb 6 02:32:55 2019][1671549.389842] Lustre: 127501:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 02:32:55 2019][1671549.389842] req@ffffa0769baff200 x1624699331352128/t0(0) o36->4c6307e0-e5e3-3295-9e75-ac7ce5c5822c@10.9.104.62@o2ib4:606/0 lens 520/2888 e 0 to 0 dl 1549449356 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 02:32:55 2019][1671549.419346] Lustre: 127501:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 11 previous similar messages [Wed Feb 6 02:33:01 2019][1671555.424831] Lustre: fir-MDT0003: Client 94c2dba8-b225-c91a-753d-91a7f0495a0f (at 10.9.101.8@o2ib4) reconnecting [Wed Feb 6 02:33:01 2019][1671555.428920] Lustre: fir-MDT0003: Connection restored to 5d84d21d-4aed-2e66-e01b-378e9d302a9a (at 10.9.102.26@o2ib4) [Wed Feb 6 02:33:01 2019][1671555.428922] Lustre: Skipped 95 previous similar messages [Wed Feb 6 02:33:01 2019][1671555.451185] Lustre: Skipped 19 previous similar messages [Wed Feb 6 02:33:07 2019][1671561.351925] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 02:33:07 2019][1671561.368081] Lustre: Skipped 3 previous similar messages [Wed Feb 6 02:33:07 2019][1671561.373503] LustreError: 129253:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549449063, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa063615d72c0/0x8c1de53940b5ac61 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 130 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935182b70819 expref: -99 pid: 129253 timeout: 0 lvb_type: 0 [Wed Feb 6 02:33:07 2019][1671561.414014] LustreError: 129253:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 122 previous similar messages [Wed Feb 6 02:35:40 2019][1671714.418963] Lustre: 127198:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:1s); client may timeout. req@ffffa07432385d00 x1624697981967232/t8856612055(0) o36->d5be83ac-c435-a05d-4965-bc79fa913f84@10.9.103.4@o2ib4:10/0 lens 544/424 e 0 to 0 dl 1549449515 ref 1 fl Complete:/0/0 rc 0/0 [Wed Feb 6 02:35:59 2019][1671733.496379] LNetError: 17855:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.19.4@tcp from [Wed Feb 6 02:35:59 2019][1671733.506992] LNetError: 17855:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 9228799 previous similar messages [Wed Feb 6 02:35:59 2019][1671733.518153] Lustre: 17855:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549449535/real 1549449535] req@ffffa07ad8601e00 x1623667894005872/t0(0) o104->fir-MDT0003@10.10.19.4@tcp:15/16 lens 296/224 e 0 to 1 dl 1549449546 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 [Wed Feb 6 02:35:59 2019][1671733.545766] Lustre: 17855:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7088923 previous similar messages [Wed Feb 6 02:37:14 2019][1671808.496906] LNetError: 17855:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.19.4@tcp from [Wed Feb 6 02:37:14 2019][1671808.507510] LNetError: 17855:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 10554061 previous similar messages [Wed Feb 6 02:37:14 2019][1671808.518909] Lustre: 17855:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549449610/real 1549449610] req@ffffa07ad8601e00 x1623667894005872/t0(0) o104->fir-MDT0003@10.10.19.4@tcp:15/16 lens 296/224 e 0 to 1 dl 1549449621 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 02:37:14 2019][1671808.546520] Lustre: 17855:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10554083 previous similar messages [Wed Feb 6 02:38:30 2019][1671884.321534] LustreError: 17855:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.19.4@tcp) failed to reply to blocking AST (req@ffffa07ad8601e00 x1623667894005872 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa0602a047080/0x8c1de53940c68429 lrc: 4/0,0 mode: PR/PR res: [0x280000de5:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a76a6a expref: 15 pid: 17978 timeout: 1672019 lvb_type: 0 [Wed Feb 6 02:38:30 2019][1671884.363862] LustreError: 17855:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages [Wed Feb 6 02:38:30 2019][1671884.374132] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.19.4@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 02:38:30 2019][1671884.386664] LustreError: Skipped 2 previous similar messages [Wed Feb 6 02:38:30 2019][1671884.392533] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.19.4@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa0602a047080/0x8c1de53940c68429 lrc: 3/0,0 mode: PR/PR res: [0x280000de5:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a76a6a expref: 16 pid: 17978 timeout: 0 lvb_type: 0 [Wed Feb 6 02:38:30 2019][1671884.429849] LustreError: 17855:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa057746f6c00 ns: mdt-fir-MDT0003_UUID lock: ffffa07a7f20bcc0/0x8c1de53940c69d5f lrc: 1/0,0 mode: EX/EX res: [0x280000de5:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.19.4@tcp remote: 0x844b529590a76a9b expref: 4 pid: 17855 timeout: 0 lvb_type: 3 [Wed Feb 6 02:38:30 2019][1671884.464051] LustreError: 17855:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message [Wed Feb 6 02:39:44 2019][1671958.498122] LNetError: 127970:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.19.4@tcp from [Wed Feb 6 02:39:44 2019][1671958.508818] LNetError: 127970:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 19818737 previous similar messages [Wed Feb 6 02:39:44 2019][1671958.520139] Lustre: 127970:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549449760/real 1549449760] req@ffffa055a22f4800 x1623667894293056/t0(0) o104->fir-MDT0003@10.10.19.4@tcp:15/16 lens 296/224 e 0 to 1 dl 1549449771 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 02:39:44 2019][1671958.547842] Lustre: 127970:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 19818715 previous similar messages [Wed Feb 6 02:41:10 2019][1672044.322828] LustreError: 127970:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.19.4@tcp) failed to reply to blocking AST (req@ffffa055a22f4800 x1623667894293056 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa05388475a00/0x8c1de53940d6c2ca lrc: 4/0,0 mode: PR/PR res: [0x280000de5:0x1:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x60000400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a76acc expref: 12 pid: 127970 timeout: 1672179 lvb_type: 0 [Wed Feb 6 02:41:10 2019][1672044.365343] LustreError: 127970:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message [Wed Feb 6 02:41:10 2019][1672044.375611] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.19.4@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 02:41:10 2019][1672044.388140] LustreError: Skipped 1 previous similar message [Wed Feb 6 02:41:10 2019][1672044.393916] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.19.4@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa05388475a00/0x8c1de53940d6c2ca lrc: 3/0,0 mode: PR/PR res: [0x280000de5:0x1:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x60000400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a76acc expref: 13 pid: 127970 timeout: 0 lvb_type: 0 [Wed Feb 6 02:41:10 2019][1672044.431052] LustreError: 127970:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa0702be7dc00 ns: mdt-fir-MDT0003_UUID lock: ffffa053884721c0/0x8c1de53940d6c325 lrc: 5/0,0 mode: PW/PW res: [0x280000de5:0x1:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT flags: 0x50201000000000 nid: 10.10.19.4@tcp remote: 0x844b529590a76bb3 expref: 6 pid: 127970 timeout: 0 lvb_type: 0 [Wed Feb 6 02:43:11 2019][1672165.473810] LustreError: 18044:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549449667, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa05e4c2133c0/0x8c1de53940d2a455 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 117 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935182eb9c01 expref: -99 pid: 18044 timeout: 0 lvb_type: 0 [Wed Feb 6 02:43:11 2019][1672165.514147] LustreError: 18044:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 72 previous similar messages [Wed Feb 6 02:43:43 2019][1672197.324052] LustreError: 127970:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.19.4@tcp) failed to reply to blocking AST (req@ffffa05b05613600 x1623667894566192 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa077cfa09680/0x8c1de53940de9200 lrc: 4/0,0 mode: PR/PR res: [0x280000de5:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a76bc8 expref: 10 pid: 129266 timeout: 1672332 lvb_type: 0 [Wed Feb 6 02:43:43 2019][1672197.366569] LustreError: 127970:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message [Wed Feb 6 02:43:43 2019][1672197.376828] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.19.4@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 02:43:43 2019][1672197.389352] LustreError: Skipped 1 previous similar message [Wed Feb 6 02:43:43 2019][1672197.395124] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.19.4@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa077cfa09680/0x8c1de53940de9200 lrc: 3/0,0 mode: PR/PR res: [0x280000de5:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a76bc8 expref: 11 pid: 129266 timeout: 0 lvb_type: 0 [Wed Feb 6 02:43:43 2019][1672197.432294] LustreError: 127970:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa0659de36c00 ns: mdt-fir-MDT0003_UUID lock: ffffa04f9d3de540/0x8c1de53940de921c lrc: 1/0,0 mode: --/PW res: [0x280000de5:0x1:0x0].0x0 bits 0x40/0x0 rrc: 3 type: IBT flags: 0x54a01000000000 nid: 10.10.19.4@tcp remote: 0x844b529590a76bd6 expref: 8 pid: 127970 timeout: 0 lvb_type: 0 [Wed Feb 6 02:43:43 2019][1672197.495765] Lustre: fir-MDT0003: Connection restored to 04a6727c-f0a5-409b-d667-2e88045cb817 (at 10.10.19.4@tcp) [Wed Feb 6 02:43:43 2019][1672197.506116] Lustre: Skipped 116 previous similar messages [Wed Feb 6 02:44:44 2019][1672258.500539] LNetError: 129252:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.19.4@tcp from [Wed Feb 6 02:44:44 2019][1672258.511235] LNetError: 129252:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 41047653 previous similar messages [Wed Feb 6 02:44:44 2019][1672258.522545] Lustre: 129252:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549450060/real 1549450060] req@ffffa0657c975d00 x1623667894845904/t0(0) o104->fir-MDT0003@10.10.19.4@tcp:15/16 lens 296/224 e 0 to 1 dl 1549450071 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 02:44:44 2019][1672258.550229] Lustre: 129252:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 41047653 previous similar messages [Wed Feb 6 02:45:42 2019][1672316.444009] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 02:45:42 2019][1672316.460161] Lustre: Skipped 2 previous similar messages [Wed Feb 6 02:48:45 2019][1672499.326492] LustreError: 17935:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.19.4@tcp) failed to reply to blocking AST (req@ffffa08a4c6f5a00 x1623667895136960 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa05fb2f01680/0x8c1de53940f6edb0 lrc: 4/0,0 mode: PR/PR res: [0x280000de5:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a76c0e expref: 10 pid: 129252 timeout: 1672634 lvb_type: 0 [Wed Feb 6 02:48:45 2019][1672499.368907] LustreError: 17935:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message [Wed Feb 6 02:48:45 2019][1672499.379083] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.19.4@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 02:48:45 2019][1672499.391611] LustreError: Skipped 1 previous similar message [Wed Feb 6 02:48:45 2019][1672499.397387] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.19.4@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa05fb2f01680/0x8c1de53940f6edb0 lrc: 3/0,0 mode: PR/PR res: [0x280000de5:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a76c0e expref: 11 pid: 129252 timeout: 0 lvb_type: 0 [Wed Feb 6 02:48:45 2019][1672499.434453] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [Wed Feb 6 02:48:45 2019][1672499.444498] LustreError: 17935:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa0763e9eb400 ns: mdt-fir-MDT0003_UUID lock: ffffa05527a11d40/0x8c1de53940f6ede1 lrc: 3/0,0 mode: PW/PW res: [0x280000de5:0x1:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x50200000000000 nid: 10.10.19.4@tcp remote: 0x844b529590a76c1c expref: 3 pid: 17935 timeout: 0 lvb_type: 0 [Wed Feb 6 02:48:45 2019][1672499.478796] LustreError: 17935:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message [Wed Feb 6 02:53:13 2019][1672767.141650] LustreError: 129805:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549450269, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa0787e727500/0x8c1de53941061b51 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 106 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518335b9c0 expref: -99 pid: 129805 timeout: 0 lvb_type: 0 [Wed Feb 6 02:53:13 2019][1672767.182175] LustreError: 129805:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 104 previous similar messages [Wed Feb 6 02:53:47 2019][1672801.358243] Lustre: fir-MDT0003: Connection restored to 04a6727c-f0a5-409b-d667-2e88045cb817 (at 10.10.19.4@tcp) [Wed Feb 6 02:53:47 2019][1672801.368597] Lustre: Skipped 7 previous similar messages [Wed Feb 6 02:54:44 2019][1672858.505378] LNetError: 19141:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.19.4@tcp from [Wed Feb 6 02:54:44 2019][1672858.515984] LNetError: 19141:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 81755325 previous similar messages [Wed Feb 6 02:54:44 2019][1672858.527377] Lustre: 19141:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549450660/real 1549450660] req@ffffa0611a43ec00 x1623667895797328/t0(0) o104->fir-MDT0003@10.10.19.4@tcp:15/16 lens 296/224 e 0 to 1 dl 1549450671 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 02:54:44 2019][1672858.554987] Lustre: 19141:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 81755345 previous similar messages [Wed Feb 6 02:55:47 2019][1672921.370888] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 02:55:47 2019][1672921.387047] Lustre: Skipped 3 previous similar messages [Wed Feb 6 03:03:20 2019][1673374.375556] LustreError: 17919:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549450876, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa0557877d100/0x8c1de539411ff29e lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 105 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351837bd327 expref: -99 pid: 17919 timeout: 0 lvb_type: 0 [Wed Feb 6 03:03:20 2019][1673374.415899] LustreError: 17919:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 120 previous similar messages [Wed Feb 6 03:08:30 2019][1673684.838043] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 03:08:30 2019][1673684.854200] Lustre: Skipped 3 previous similar messages [Wed Feb 6 03:08:30 2019][1673684.876213] Lustre: fir-MDT0000-osp-MDT0003: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [Wed Feb 6 03:08:31 2019][1673684.886043] Lustre: Skipped 9 previous similar messages [Wed Feb 6 03:13:25 2019][1673979.129422] LustreError: 127250:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549451481, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa04b592c3cc0/0x8c1de53941480ba7 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 86 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935183ead409 expref: -99 pid: 127250 timeout: 0 lvb_type: 0 [Wed Feb 6 03:13:25 2019][1673979.169844] LustreError: 127250:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 141 previous similar messages [Wed Feb 6 03:15:42 2019][1674115.999609] Lustre: fir-MDT0003: haven't heard from client 8dd25978-9a6e-9f02-c6e4-dd789624ff48 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa06256042000, cur 1549451918 expire 1549451768 last 1549451691 [Wed Feb 6 03:15:42 2019][1674116.021483] Lustre: Skipped 1 previous similar message [Wed Feb 6 03:28:59 2019][1674913.264993] Lustre: fir-MDT0003: Connection restored to cef0c972-4cf2-c1ae-1fb9-c404ed41cb0f (at 10.8.8.22@o2ib6) [Wed Feb 6 03:28:59 2019][1674913.275433] Lustre: Skipped 4 previous similar messages [Wed Feb 6 03:33:32 2019][1675186.830153] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 03:33:32 2019][1675186.846314] Lustre: Skipped 3 previous similar messages [Wed Feb 6 03:33:32 2019][1675186.851733] LustreError: 17854:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549452688, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa069ede998c0/0x8c1de53942014098 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0rrc: 67 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351852d1b28 expref: -99 pid: 17854 timeout: 0 lvb_type: 0 [Wed Feb 6 03:33:32 2019][1675186.868345] Lustre: fir-MDT0000-osp-MDT0003: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [Wed Feb 6 03:33:32 2019][1675186.868346] Lustre: Skipped 2 previous similar messages [Wed Feb 6 03:33:33 2019][1675186.907218] LustreError: 17854:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 41 previous similar messages [Wed Feb 6 03:33:35 2019][1675189.227172] LNet: Service thread pid 129279 was inactive for 302.39s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 03:33:35 2019][1675189.244372] LNet: Skipped 4 previous similar messages [Wed Feb 6 03:33:35 2019][1675189.249602] Pid: 129279, comm: mdt02_021 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:33:35 2019][1675189.259616] Call Trace: [Wed Feb 6 03:33:35 2019][1675189.262253] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:33:35 2019][1675189.269360] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:33:35 2019][1675189.276647] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:33:35 2019][1675189.283486] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:33:35 2019][1675189.290236] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:33:35 2019][1675189.296623] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:33:35 2019][1675189.302947] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:33:35 2019][1675189.310402] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:33:35 2019][1675189.317326] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:33:35 2019][1675189.325545] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:33:35 2019][1675189.331955] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:33:35 2019][1675189.338182] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:33:35 2019][1675189.344931] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:33:35 2019][1675189.350800] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:33:35 2019][1675189.357932] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:33:35 2019][1675189.365820] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:33:35 2019][1675189.372337] [] kthread+0xd1/0xe0 [Wed Feb 6 03:33:35 2019][1675189.377441] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:33:35 2019][1675189.384102] [] 0xffffffffffffffff [Wed Feb 6 03:33:35 2019][1675189.389300] LustreError: dumping log to /tmp/lustre-log.1549452991.129279 [Wed Feb 6 03:33:35 2019][1675189.619298] Pid: 17854, comm: mdt03_027 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:33:35 2019][1675189.629214] Call Trace: [Wed Feb 6 03:33:35 2019][1675189.631862] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:33:35 2019][1675189.638968] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:33:35 2019][1675189.646259] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:33:35 2019][1675189.653096] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:33:35 2019][1675189.659856] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:33:35 2019][1675189.666261] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:33:35 2019][1675189.672578] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:33:35 2019][1675189.680016] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:33:35 2019][1675189.686939] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:33:35 2019][1675189.695158] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:33:35 2019][1675189.701560] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:33:35 2019][1675189.707783] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:33:35 2019][1675189.714533] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:33:35 2019][1675189.720403] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:33:35 2019][1675189.727535] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:33:35 2019][1675189.735437] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:33:35 2019][1675189.741956] [] kthread+0xd1/0xe0 [Wed Feb 6 03:33:35 2019][1675189.747045] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:33:35 2019][1675189.753697] [] 0xffffffffffffffff [Wed Feb 6 03:33:37 2019][1675191.275188] Pid: 17838, comm: mdt02_029 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:33:37 2019][1675191.285100] Call Trace: [Wed Feb 6 03:33:37 2019][1675191.287738] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:33:37 2019][1675191.294848] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:33:37 2019][1675191.302135] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:33:37 2019][1675191.308973] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:33:37 2019][1675191.315708] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:33:37 2019][1675191.322095] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:33:37 2019][1675191.328396] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:33:37 2019][1675191.335822] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:33:37 2019][1675191.342746] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:33:37 2019][1675191.350954] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:33:37 2019][1675191.357357] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:33:37 2019][1675191.363574] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:33:37 2019][1675191.370331] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:33:37 2019][1675191.376208] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:33:37 2019][1675191.383317] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:33:37 2019][1675191.391193] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:33:37 2019][1675191.397686] [] kthread+0xd1/0xe0 [Wed Feb 6 03:33:37 2019][1675191.402779] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:33:37 2019][1675191.409421] [] 0xffffffffffffffff [Wed Feb 6 03:33:37 2019][1675191.414615] LustreError: dumping log to /tmp/lustre-log.1549452993.17838 [Wed Feb 6 03:33:37 2019][1675191.422018] Pid: 129805, comm: mdt02_025 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:33:37 2019][1675191.432052] Call Trace: [Wed Feb 6 03:33:37 2019][1675191.434682] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:33:37 2019][1675191.441783] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:33:37 2019][1675191.449087] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:33:37 2019][1675191.455926] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:33:37 2019][1675191.462677] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:33:37 2019][1675191.469067] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:33:37 2019][1675191.475379] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:33:37 2019][1675191.482814] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:33:37 2019][1675191.489734] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:33:37 2019][1675191.497945] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:33:37 2019][1675191.504355] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:33:37 2019][1675191.510572] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:33:37 2019][1675191.517336] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:33:37 2019][1675191.523208] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:33:37 2019][1675191.530346] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:33:37 2019][1675191.538237] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:33:37 2019][1675191.544751] [] kthread+0xd1/0xe0 [Wed Feb 6 03:33:37 2019][1675191.549832] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:33:37 2019][1675191.556487] [] 0xffffffffffffffff [Wed Feb 6 03:33:38 2019][1675192.299196] Pid: 129373, comm: mdt01_031 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:33:38 2019][1675192.309193] Call Trace: [Wed Feb 6 03:33:38 2019][1675192.311839] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:33:38 2019][1675192.318969] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:33:38 2019][1675192.326279] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:33:38 2019][1675192.333127] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:33:38 2019][1675192.339887] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:33:38 2019][1675192.346287] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:33:38 2019][1675192.352626] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:33:38 2019][1675192.360075] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:33:38 2019][1675192.367005] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:33:38 2019][1675192.375223] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:33:38 2019][1675192.381634] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:33:38 2019][1675192.387875] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:33:38 2019][1675192.394645] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:33:38 2019][1675192.400523] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:33:38 2019][1675192.407675] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:33:38 2019][1675192.415575] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:33:38 2019][1675192.422091] [] kthread+0xd1/0xe0 [Wed Feb 6 03:33:38 2019][1675192.427179] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:33:38 2019][1675192.433842] [] 0xffffffffffffffff [Wed Feb 6 03:33:38 2019][1675192.439039] LustreError: dumping log to /tmp/lustre-log.1549452994.129373 [Wed Feb 6 03:33:42 2019][1675196.395236] LNet: Service thread pid 128085 was inactive for 302.26s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 03:33:42 2019][1675196.408364] LNet: Skipped 13 previous similar messages [Wed Feb 6 03:33:42 2019][1675196.413684] LustreError: dumping log to /tmp/lustre-log.1549452998.128085 [Wed Feb 6 03:33:43 2019][1675197.419239] LustreError: dumping log to /tmp/lustre-log.1549452999.129795 [Wed Feb 6 03:33:48 2019][1675202.539279] LustreError: dumping log to /tmp/lustre-log.1549453004.17839 [Wed Feb 6 03:33:49 2019][1675203.563289] LustreError: dumping log to /tmp/lustre-log.1549453005.127248 [Wed Feb 6 03:33:50 2019][1675204.587289] LustreError: dumping log to /tmp/lustre-log.1549453006.847 [Wed Feb 6 03:33:51 2019][1675205.611302] LustreError: dumping log to /tmp/lustre-log.1549453007.17831 [Wed Feb 6 03:33:52 2019][1675206.635308] LustreError: dumping log to /tmp/lustre-log.1549453008.17976 [Wed Feb 6 03:33:53 2019][1675207.659323] LustreError: dumping log to /tmp/lustre-log.1549453009.127327 [Wed Feb 6 03:33:54 2019][1675208.683331] LustreError: dumping log to /tmp/lustre-log.1549453010.17815 [Wed Feb 6 03:33:55 2019][1675209.707336] LustreError: dumping log to /tmp/lustre-log.1549453011.129266 [Wed Feb 6 03:33:56 2019][1675210.731348] LustreError: dumping log to /tmp/lustre-log.1549453012.127210 [Wed Feb 6 03:33:57 2019][1675211.755348] LustreError: dumping log to /tmp/lustre-log.1549453013.17991 [Wed Feb 6 03:33:59 2019][1675213.803369] LustreError: dumping log to /tmp/lustre-log.1549453015.17811 [Wed Feb 6 03:34:00 2019][1675214.827377] LustreError: dumping log to /tmp/lustre-log.1549453016.17958 [Wed Feb 6 03:34:02 2019][1675216.875395] LustreError: dumping log to /tmp/lustre-log.1549453018.18268 [Wed Feb 6 03:34:05 2019][1675218.923413] LustreError: dumping log to /tmp/lustre-log.1549453020.14718 [Wed Feb 6 03:34:06 2019][1675219.947423] LustreError: dumping log to /tmp/lustre-log.1549453021.17968 [Wed Feb 6 03:34:08 2019][1675221.995438] LustreError: dumping log to /tmp/lustre-log.1549453023.18272 [Wed Feb 6 03:34:13 2019][1675227.115482] LustreError: dumping log to /tmp/lustre-log.1549453029.129796 [Wed Feb 6 03:34:16 2019][1675230.187505] LustreError: dumping log to /tmp/lustre-log.1549453032.18262 [Wed Feb 6 03:34:18 2019][1675232.235520] LustreError: dumping log to /tmp/lustre-log.1549453034.17992 [Wed Feb 6 03:34:19 2019][1675233.259533] LustreError: dumping log to /tmp/lustre-log.1549453035.17885 [Wed Feb 6 03:34:20 2019][1675234.283532] LustreError: dumping log to /tmp/lustre-log.1549453036.127250 [Wed Feb 6 03:34:22 2019][1675236.331548] LustreError: dumping log to /tmp/lustre-log.1549453038.17931 [Wed Feb 6 03:34:24 2019][1675238.379568] LustreError: dumping log to /tmp/lustre-log.1549453040.19060 [Wed Feb 6 03:34:26 2019][1675240.427587] LustreError: dumping log to /tmp/lustre-log.1549453042.18335 [Wed Feb 6 03:34:27 2019][1675241.451595] LustreError: dumping log to /tmp/lustre-log.1549453043.19141 [Wed Feb 6 03:34:28 2019][1675242.475601] LustreError: dumping log to /tmp/lustre-log.1549453044.129371 [Wed Feb 6 03:34:29 2019][1675243.499611] LustreError: dumping log to /tmp/lustre-log.1549453045.17848 [Wed Feb 6 03:34:30 2019][1675244.523628] LustreError: dumping log to /tmp/lustre-log.1549453046.17983 [Wed Feb 6 03:34:33 2019][1675247.595639] LustreError: dumping log to /tmp/lustre-log.1549453049.127196 [Wed Feb 6 03:34:55 2019][1675269.099813] LustreError: dumping log to /tmp/lustre-log.1549453071.17850 [Wed Feb 6 03:35:05 2019][1675279.339900] LNet: Service thread pid 18261 was inactive for 302.62s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 03:35:05 2019][1675279.352937] LNet: Skipped 38 previous similar messages [Wed Feb 6 03:35:05 2019][1675279.358255] LustreError: dumping log to /tmp/lustre-log.1549453081.18261 [Wed Feb 6 03:35:06 2019][1675280.363911] LustreError: dumping log to /tmp/lustre-log.1549453082.129374 [Wed Feb 6 03:35:13 2019][1675287.531970] LustreError: dumping log to /tmp/lustre-log.1549453089.17984 [Wed Feb 6 03:35:14 2019][1675288.555972] LustreError: dumping log to /tmp/lustre-log.1549453090.126634 [Wed Feb 6 03:35:17 2019][1675291.627998] LustreError: dumping log to /tmp/lustre-log.1549453093.17875 [Wed Feb 6 03:35:24 2019][1675298.796056] LustreError: dumping log to /tmp/lustre-log.1549453100.17999 [Wed Feb 6 03:35:27 2019][1675301.868082] LustreError: dumping log to /tmp/lustre-log.1549453103.127205 [Wed Feb 6 03:35:28 2019][1675302.892092] LustreError: dumping log to /tmp/lustre-log.1549453104.129251 [Wed Feb 6 03:35:35 2019][1675309.036135] LustreError: dumping log to /tmp/lustre-log.1549453111.129253 [Wed Feb 6 03:35:36 2019][1675310.060146] LustreError: dumping log to /tmp/lustre-log.1549453112.17851 [Wed Feb 6 03:35:49 2019][1675323.372250] LustreError: dumping log to /tmp/lustre-log.1549453125.127206 [Wed Feb 6 03:36:20 2019][1675354.092515] LustreError: dumping log to /tmp/lustre-log.1549453156.17805 [Wed Feb 6 03:36:21 2019][1675355.116510] LustreError: dumping log to /tmp/lustre-log.1549453157.17853 [Wed Feb 6 03:36:27 2019][1675361.260555] LustreError: dumping log to /tmp/lustre-log.1549453163.126632 [Wed Feb 6 03:38:27 2019][1675481.233533] Lustre: 127273:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 03:38:27 2019][1675481.233533] req@ffffa07b0e657500 x1624700971642976/t0(0) o36->dc527ec4-db62-14c0-459e-fcb26c78037c@10.9.107.8@o2ib4:8/0 lens 536/2888 e 17 to 0 dl 1549453288 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 03:38:27 2019][1675481.262602] Lustre: 127273:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 116 previous similar messages [Wed Feb 6 03:38:34 2019][1675488.892189] Lustre: fir-MDT0003: Client dc527ec4-db62-14c0-459e-fcb26c78037c (at 10.9.107.8@o2ib4) reconnecting [Wed Feb 6 03:38:34 2019][1675488.902452] Lustre: Skipped 98 previous similar messages [Wed Feb 6 03:38:34 2019][1675488.907965] Lustre: fir-MDT0003: Connection restored to dc527ec4-db62-14c0-459e-fcb26c78037c (at 10.9.107.8@o2ib4) [Wed Feb 6 03:38:35 2019][1675488.918504] Lustre: Skipped 2 previous similar messages [Wed Feb 6 03:39:04 2019][1675517.960823] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 03:39:04 2019][1675517.976975] Lustre: Skipped 1 previous similar message [Wed Feb 6 03:39:21 2019][1675535.341966] LNet: Service thread pid 129244 was inactive for 439.42s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 03:39:21 2019][1675535.359160] LNet: Skipped 4 previous similar messages [Wed Feb 6 03:39:21 2019][1675535.364398] Pid: 129244, comm: mdt01_020 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:39:21 2019][1675535.374415] Call Trace: [Wed Feb 6 03:39:21 2019][1675535.377058] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:39:21 2019][1675535.384188] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:39:21 2019][1675535.391483] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:39:21 2019][1675535.398338] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:39:21 2019][1675535.405084] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:39:21 2019][1675535.411495] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:39:21 2019][1675535.417808] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:39:21 2019][1675535.425292] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:39:21 2019][1675535.432210] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:39:21 2019][1675535.440443] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:39:21 2019][1675535.446841] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:39:21 2019][1675535.453076] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:39:21 2019][1675535.459814] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:39:21 2019][1675535.465705] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:39:21 2019][1675535.472833] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:39:21 2019][1675535.480746] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:39:21 2019][1675535.487244] [] kthread+0xd1/0xe0 [Wed Feb 6 03:39:21 2019][1675535.492347] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:39:21 2019][1675535.498997] [] 0xffffffffffffffff [Wed Feb 6 03:39:21 2019][1675535.504206] LustreError: dumping log to /tmp/lustre-log.1549453337.129244 [Wed Feb 6 03:39:47 2019][1675561.518190] Lustre: 18186:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 03:39:47 2019][1675561.518190] req@ffffa089a6ef0600 x1624702235363488/t0(0) o36->c2c0536b-b17c-9118-170c-d010e3c3b183@10.9.107.15@o2ib4:88/0 lens 536/2888 e 5 to 0 dl 1549453368 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 03:39:47 2019][1675561.547263] Lustre: 18186:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 42 previous similar messages [Wed Feb 6 03:39:54 2019][1675568.050577] Lustre: fir-MDT0003: Client c2c0536b-b17c-9118-170c-d010e3c3b183 (at 10.9.107.15@o2ib4) reconnecting [Wed Feb 6 03:39:54 2019][1675568.060930] Lustre: Skipped 42 previous similar messages [Wed Feb 6 03:40:22 2019][1675596.782463] LNet: Service thread pid 127131 was inactive for 488.74s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 03:40:22 2019][1675596.799658] Pid: 127131, comm: mdt01_004 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:40:22 2019][1675596.809656] Call Trace: [Wed Feb 6 03:40:22 2019][1675596.812301] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:40:22 2019][1675596.819429] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:40:22 2019][1675596.826725] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:40:22 2019][1675596.833573] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:40:22 2019][1675596.840330] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:40:22 2019][1675596.846729] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:40:22 2019][1675596.853055] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:40:22 2019][1675596.860494] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:40:22 2019][1675596.867430] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:40:22 2019][1675596.875648] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:40:22 2019][1675596.882043] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:40:22 2019][1675596.888285] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:40:22 2019][1675596.895041] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:40:22 2019][1675596.900922] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:40:23 2019][1675596.908061] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:40:23 2019][1675596.915953] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:40:23 2019][1675596.922487] [] kthread+0xd1/0xe0 [Wed Feb 6 03:40:23 2019][1675596.927573] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:40:23 2019][1675596.934233] [] 0xffffffffffffffff [Wed Feb 6 03:40:23 2019][1675596.939432] LustreError: dumping log to /tmp/lustre-log.1549453398.127131 [Wed Feb 6 03:40:29 2019][1675602.926505] Pid: 129230, comm: mdt01_016 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:40:29 2019][1675602.936504] Call Trace: [Wed Feb 6 03:40:29 2019][1675602.939144] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:40:29 2019][1675602.946258] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:40:29 2019][1675602.953555] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:40:29 2019][1675602.960402] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:40:29 2019][1675602.967149] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:40:29 2019][1675602.973549] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:40:29 2019][1675602.979875] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:40:29 2019][1675602.987312] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:40:29 2019][1675602.994227] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:40:29 2019][1675603.002446] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:40:29 2019][1675603.008860] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:40:29 2019][1675603.015083] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:40:29 2019][1675603.021849] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:40:29 2019][1675603.027727] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:40:29 2019][1675603.034868] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:40:29 2019][1675603.042754] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:40:29 2019][1675603.049271] [] kthread+0xd1/0xe0 [Wed Feb 6 03:40:29 2019][1675603.054360] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:40:29 2019][1675603.061023] [] 0xffffffffffffffff [Wed Feb 6 03:40:29 2019][1675603.066220] LustreError: dumping log to /tmp/lustre-log.1549453405.129230 [Wed Feb 6 03:40:33 2019][1675607.022544] Pid: 127132, comm: mdt03_004 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:40:33 2019][1675607.032544] Call Trace: [Wed Feb 6 03:40:33 2019][1675607.035184] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:40:33 2019][1675607.042314] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:40:33 2019][1675607.049632] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:40:33 2019][1675607.056483] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:40:33 2019][1675607.063240] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:40:33 2019][1675607.069640] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:40:33 2019][1675607.075961] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:40:33 2019][1675607.083403] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:40:33 2019][1675607.090333] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:40:33 2019][1675607.098551] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:40:33 2019][1675607.104960] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:40:33 2019][1675607.111187] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:40:33 2019][1675607.117949] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:40:33 2019][1675607.123824] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:40:33 2019][1675607.130938] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:40:33 2019][1675607.138835] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:40:33 2019][1675607.145335] [] kthread+0xd1/0xe0 [Wed Feb 6 03:40:33 2019][1675607.150422] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:40:33 2019][1675607.157070] [] 0xffffffffffffffff [Wed Feb 6 03:40:33 2019][1675607.162271] LustreError: dumping log to /tmp/lustre-log.1549453409.127132 [Wed Feb 6 03:41:46 2019][1675680.751137] LNet: Service thread pid 17840 was inactive for 538.98s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 03:41:46 2019][1675680.768243] LNet: Skipped 2 previous similar messages [Wed Feb 6 03:41:46 2019][1675680.773478] Pid: 17840, comm: mdt02_031 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:41:46 2019][1675680.783407] Call Trace: [Wed Feb 6 03:41:46 2019][1675680.786049] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:41:46 2019][1675680.793173] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:41:46 2019][1675680.800467] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:41:46 2019][1675680.807316] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:41:46 2019][1675680.814074] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:41:46 2019][1675680.820471] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:41:46 2019][1675680.826798] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:41:46 2019][1675680.834235] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:41:46 2019][1675680.841156] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:41:46 2019][1675680.849378] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:41:46 2019][1675680.855785] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:41:46 2019][1675680.862012] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:41:46 2019][1675680.868771] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:41:46 2019][1675680.874641] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:41:46 2019][1675680.881778] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:41:46 2019][1675680.889667] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:41:46 2019][1675680.896196] [] kthread+0xd1/0xe0 [Wed Feb 6 03:41:47 2019][1675680.901282] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:41:47 2019][1675680.907942] [] 0xffffffffffffffff [Wed Feb 6 03:41:47 2019][1675680.913139] LustreError: dumping log to /tmp/lustre-log.1549453482.17840 [Wed Feb 6 03:41:52 2019][1675686.895188] LNet: Service thread pid 17967 was inactive for 537.39s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 03:41:53 2019][1675686.908229] LNet: Skipped 17 previous similar messages [Wed Feb 6 03:41:53 2019][1675686.913551] LustreError: dumping log to /tmp/lustre-log.1549453488.17967 [Wed Feb 6 03:42:18 2019][1675712.751406] Lustre: 18186:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 03:42:18 2019][1675712.751406] req@ffffa08a00e21800 x1624700562891072/t0(0) o36->032aae30-e439-a130-1f18-efe924baca21@10.9.106.70@o2ib4:239/0 lens 536/2888 e 2 to 0 dl 1549453519 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 03:42:18 2019][1675712.780566] Lustre: 18186:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 21 previous similar messages [Wed Feb 6 03:42:24 2019][1675718.416674] Lustre: fir-MDT0003: Client 032aae30-e439-a130-1f18-efe924baca21 (at 10.9.106.70@o2ib4) reconnecting [Wed Feb 6 03:42:24 2019][1675718.427023] Lustre: Skipped 21 previous similar messages [Wed Feb 6 03:43:37 2019][1675791.230033] LustreError: 129255:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549453293, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa07458d67740/0x8c1de5394219ae7e lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 86 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935186bb1a5f expref: -99 pid: 129255 timeout: 0 lvb_type: 0 [Wed Feb 6 03:43:37 2019][1675791.270457] LustreError: 129255:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 76 previous similar messages [Wed Feb 6 03:44:05 2019][1675819.018052] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 03:44:05 2019][1675819.028665] Lustre: Skipped 67 previous similar messages [Wed Feb 6 03:44:22 2019][1675836.482394] Lustre: fir-MDT0000-osp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 03:45:32 2019][1675906.032949] LNet: Service thread pid 17828 was inactive for 688.06s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 03:45:32 2019][1675906.050059] Pid: 17828, comm: mdt03_022 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:45:32 2019][1675906.059971] Call Trace: [Wed Feb 6 03:45:32 2019][1675906.062609] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:45:32 2019][1675906.069731] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:45:32 2019][1675906.077028] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:45:32 2019][1675906.083876] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:45:32 2019][1675906.090633] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:45:32 2019][1675906.097031] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:45:32 2019][1675906.103355] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:45:32 2019][1675906.110792] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:45:32 2019][1675906.117724] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:45:32 2019][1675906.125958] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:45:32 2019][1675906.132369] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:45:32 2019][1675906.138595] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:45:32 2019][1675906.145353] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:45:32 2019][1675906.151231] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:45:32 2019][1675906.158362] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:45:32 2019][1675906.166254] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:45:32 2019][1675906.172767] [] kthread+0xd1/0xe0 [Wed Feb 6 03:45:32 2019][1675906.177858] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:45:32 2019][1675906.184519] [] 0xffffffffffffffff [Wed Feb 6 03:45:32 2019][1675906.189738] LustreError: dumping log to /tmp/lustre-log.1549453708.17828 [Wed Feb 6 03:45:34 2019][1675908.080973] Pid: 127136, comm: mdt00_005 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:45:34 2019][1675908.090970] Call Trace: [Wed Feb 6 03:45:34 2019][1675908.093610] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:45:34 2019][1675908.100740] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:45:34 2019][1675908.108041] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:45:34 2019][1675908.114892] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:45:34 2019][1675908.121652] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:45:34 2019][1675908.128065] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:45:34 2019][1675908.134392] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:45:34 2019][1675908.141848] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:45:34 2019][1675908.148779] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:45:34 2019][1675908.156997] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:45:34 2019][1675908.163416] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:45:34 2019][1675908.169645] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:45:34 2019][1675908.176410] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:45:34 2019][1675908.182286] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:45:34 2019][1675908.189428] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:45:34 2019][1675908.197315] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:45:34 2019][1675908.203831] [] kthread+0xd1/0xe0 [Wed Feb 6 03:45:34 2019][1675908.208920] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:45:34 2019][1675908.215591] [] 0xffffffffffffffff [Wed Feb 6 03:45:34 2019][1675908.220794] LustreError: dumping log to /tmp/lustre-log.1549453710.127136 [Wed Feb 6 03:46:45 2019][1675979.761544] Pid: 17908, comm: mdt02_036 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:46:45 2019][1675979.771458] Call Trace: [Wed Feb 6 03:46:45 2019][1675979.774101] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:46:45 2019][1675979.781232] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:46:45 2019][1675979.788547] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:46:45 2019][1675979.795394] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:46:45 2019][1675979.802150] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:46:45 2019][1675979.808549] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:46:45 2019][1675979.814874] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:46:45 2019][1675979.822311] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:46:45 2019][1675979.829241] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:46:45 2019][1675979.837461] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:46:45 2019][1675979.843872] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:46:45 2019][1675979.850111] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:46:45 2019][1675979.856871] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:46:45 2019][1675979.862744] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:46:45 2019][1675979.869880] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:46:45 2019][1675979.877771] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:46:45 2019][1675979.884286] [] kthread+0xd1/0xe0 [Wed Feb 6 03:46:45 2019][1675979.889374] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:46:45 2019][1675979.896036] [] 0xffffffffffffffff [Wed Feb 6 03:46:45 2019][1675979.901234] LustreError: dumping log to /tmp/lustre-log.1549453781.17908 [Wed Feb 6 03:46:56 2019][1675990.001638] Pid: 127209, comm: mdt00_006 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:46:56 2019][1675990.011643] Call Trace: [Wed Feb 6 03:46:56 2019][1675990.014285] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:46:56 2019][1675990.021413] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:46:56 2019][1675990.028684] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:46:56 2019][1675990.035533] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:46:56 2019][1675990.042266] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:46:56 2019][1675990.048670] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:46:56 2019][1675990.054987] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:46:56 2019][1675990.062424] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:46:56 2019][1675990.069356] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:46:56 2019][1675990.077593] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:46:56 2019][1675990.084003] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:46:56 2019][1675990.090219] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:46:56 2019][1675990.096969] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:46:56 2019][1675990.102840] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:46:56 2019][1675990.109976] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:46:56 2019][1675990.117867] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:46:56 2019][1675990.124382] [] kthread+0xd1/0xe0 [Wed Feb 6 03:46:56 2019][1675990.129472] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:46:56 2019][1675990.136135] [] 0xffffffffffffffff [Wed Feb 6 03:46:56 2019][1675990.141345] LustreError: dumping log to /tmp/lustre-log.1549453792.127209 [Wed Feb 6 03:47:47 2019][1676041.202043] Pid: 127317, comm: mdt03_008 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:47:47 2019][1676041.212043] Call Trace: [Wed Feb 6 03:47:47 2019][1676041.214679] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:47:47 2019][1676041.221803] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:47:47 2019][1676041.229100] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:47:47 2019][1676041.235946] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:47:47 2019][1676041.242705] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:47:47 2019][1676041.249103] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:47:47 2019][1676041.255426] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:47:47 2019][1676041.262874] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:47:47 2019][1676041.269802] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:47:47 2019][1676041.278022] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:47:47 2019][1676041.284436] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:47:47 2019][1676041.290673] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:47:47 2019][1676041.297424] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:47:47 2019][1676041.303304] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:47:47 2019][1676041.310435] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:47:47 2019][1676041.318333] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:47:47 2019][1676041.324847] [] kthread+0xd1/0xe0 [Wed Feb 6 03:47:47 2019][1676041.329937] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:47:47 2019][1676041.336618] [] 0xffffffffffffffff [Wed Feb 6 03:47:47 2019][1676041.341814] LustreError: dumping log to /tmp/lustre-log.1549453843.127317 [Wed Feb 6 03:48:36 2019][1676089.977275] Lustre: fir-MDT0003: Client c50e9e63-bc69-ffb4-d9c5-0a1d77a8b849 (at 10.9.106.60@o2ib4) reconnecting [Wed Feb 6 03:48:36 2019][1676089.987622] Lustre: Skipped 10 previous similar messages [Wed Feb 6 03:48:50 2019][1676104.178562] Lustre: 129278:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-80), not sending early reply [Wed Feb 6 03:48:50 2019][1676104.178562] req@ffffa07474001e00 x1624705129513216/t0(0) o36->680b2f8b-c457-9b65-c1bc-58d27b9ae2fe@10.8.27.14@o2ib6:631/0 lens 520/2888 e 0 to 0 dl 1549453911 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 03:48:50 2019][1676104.207889] Lustre: 129278:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 10 previous similar messages [Wed Feb 6 03:49:57 2019][1676171.191094] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 03:49:57 2019][1676171.207250] Lustre: Skipped 1 previous similar message [Wed Feb 6 03:51:32 2019][1676266.483861] LNet: Service thread pid 17830 was inactive for 938.19s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 03:51:32 2019][1676266.500967] LNet: Skipped 4 previous similar messages [Wed Feb 6 03:51:32 2019][1676266.506206] Pid: 17830, comm: mdt02_027 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:51:32 2019][1676266.516131] Call Trace: [Wed Feb 6 03:51:32 2019][1676266.518775] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:51:32 2019][1676266.525903] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:51:32 2019][1676266.533198] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:51:32 2019][1676266.540048] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:51:32 2019][1676266.546809] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:51:32 2019][1676266.553219] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:51:32 2019][1676266.559546] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:51:32 2019][1676266.566992] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:51:32 2019][1676266.573916] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:51:32 2019][1676266.582135] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:51:32 2019][1676266.588536] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:51:32 2019][1676266.594761] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:51:32 2019][1676266.601512] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:51:32 2019][1676266.607381] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:51:32 2019][1676266.614535] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:51:32 2019][1676266.622425] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:51:32 2019][1676266.628941] [] kthread+0xd1/0xe0 [Wed Feb 6 03:51:32 2019][1676266.634031] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:51:32 2019][1676266.640695] [] 0xffffffffffffffff [Wed Feb 6 03:51:32 2019][1676266.645891] LustreError: dumping log to /tmp/lustre-log.1549454068.17830 [Wed Feb 6 03:52:01 2019][1676295.156094] Pid: 17889, comm: mdt00_032 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:52:01 2019][1676295.166016] Call Trace: [Wed Feb 6 03:52:01 2019][1676295.168660] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:52:01 2019][1676295.175789] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:52:01 2019][1676295.183093] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:52:01 2019][1676295.189955] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:52:01 2019][1676295.196714] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:52:01 2019][1676295.203114] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:52:01 2019][1676295.209444] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:52:01 2019][1676295.216898] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:52:01 2019][1676295.223819] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:52:01 2019][1676295.232059] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:52:01 2019][1676295.238455] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:52:01 2019][1676295.244695] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:52:01 2019][1676295.251439] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:52:01 2019][1676295.257353] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:52:01 2019][1676295.264481] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:52:01 2019][1676295.272384] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:52:01 2019][1676295.278886] [] kthread+0xd1/0xe0 [Wed Feb 6 03:52:01 2019][1676295.283988] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:52:01 2019][1676295.290638] [] 0xffffffffffffffff [Wed Feb 6 03:52:01 2019][1676295.295855] LustreError: dumping log to /tmp/lustre-log.1549454097.17889 [Wed Feb 6 03:52:54 2019][1676348.404524] Pid: 129246, comm: mdt02_013 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:52:54 2019][1676348.414525] Call Trace: [Wed Feb 6 03:52:54 2019][1676348.417171] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:52:54 2019][1676348.424298] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:52:54 2019][1676348.431584] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:52:54 2019][1676348.438425] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:52:54 2019][1676348.445174] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:52:54 2019][1676348.451563] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:52:54 2019][1676348.457880] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:52:54 2019][1676348.465334] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:52:54 2019][1676348.472257] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:52:54 2019][1676348.480477] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:52:54 2019][1676348.486879] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:52:54 2019][1676348.493094] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:52:54 2019][1676348.499843] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:52:54 2019][1676348.505714] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:52:54 2019][1676348.512844] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:52:54 2019][1676348.520734] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:52:54 2019][1676348.527249] [] kthread+0xd1/0xe0 [Wed Feb 6 03:52:54 2019][1676348.532353] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:52:54 2019][1676348.539017] [] 0xffffffffffffffff [Wed Feb 6 03:52:54 2019][1676348.544213] LustreError: dumping log to /tmp/lustre-log.1549454150.129246 [Wed Feb 6 03:54:06 2019][1676420.105202] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 03:54:06 2019][1676420.115809] Lustre: Skipped 85 previous similar messages [Wed Feb 6 03:54:14 2019][1676427.921165] LustreError: 17969:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549453929, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa08992e9af40/0x8c1de539421ef47b lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 96 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935187b8b694 expref: -99 pid: 17969 timeout: 0 lvb_type: 0 [Wed Feb 6 03:54:14 2019][1676427.961420] LustreError: 17969:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 16 previous similar messages [Wed Feb 6 03:55:42 2019][1676516.341874] Pid: 128267, comm: mdt02_012 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:55:42 2019][1676516.351873] Call Trace: [Wed Feb 6 03:55:42 2019][1676516.354516] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:55:42 2019][1676516.361638] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:55:42 2019][1676516.368934] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:55:42 2019][1676516.375790] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:55:42 2019][1676516.382533] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:55:42 2019][1676516.388943] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:55:42 2019][1676516.395256] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:55:42 2019][1676516.402708] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:55:42 2019][1676516.409617] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:55:42 2019][1676516.417849] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:55:42 2019][1676516.424247] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:55:42 2019][1676516.430476] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:55:42 2019][1676516.437221] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:55:42 2019][1676516.443113] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:55:42 2019][1676516.450239] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:55:42 2019][1676516.458140] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:55:42 2019][1676516.464642] [] kthread+0xd1/0xe0 [Wed Feb 6 03:55:42 2019][1676516.469744] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:55:42 2019][1676516.476395] [] 0xffffffffffffffff [Wed Feb 6 03:55:42 2019][1676516.481616] LustreError: dumping log to /tmp/lustre-log.1549454318.128267 [Wed Feb 6 03:58:09 2019][1676663.799061] Pid: 129276, comm: mdt00_021 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:58:09 2019][1676663.809063] Call Trace: [Wed Feb 6 03:58:09 2019][1676663.811701] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:58:09 2019][1676663.818830] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:58:09 2019][1676663.826129] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:58:09 2019][1676663.832974] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:58:09 2019][1676663.839723] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:58:09 2019][1676663.846123] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:58:09 2019][1676663.852437] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:58:09 2019][1676663.859893] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:58:09 2019][1676663.866822] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:58:09 2019][1676663.875045] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:58:09 2019][1676663.881462] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:58:09 2019][1676663.887686] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:58:09 2019][1676663.894435] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:58:09 2019][1676663.900317] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:58:09 2019][1676663.907463] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:58:09 2019][1676663.915353] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:58:09 2019][1676663.921867] [] kthread+0xd1/0xe0 [Wed Feb 6 03:58:09 2019][1676663.926957] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:58:10 2019][1676663.933636] [] 0xffffffffffffffff [Wed Feb 6 03:58:10 2019][1676663.938832] LustreError: dumping log to /tmp/lustre-log.1549454465.129276 [Wed Feb 6 03:58:37 2019][1676691.053011] Lustre: fir-MDT0003: Client dc527ec4-db62-14c0-459e-fcb26c78037c (at 10.9.107.8@o2ib4) reconnecting [Wed Feb 6 03:58:37 2019][1676691.063276] Lustre: Skipped 92 previous similar messages [Wed Feb 6 03:58:38 2019][1676692.471291] Pid: 129255, comm: mdt01_025 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:58:38 2019][1676692.481292] Call Trace: [Wed Feb 6 03:58:38 2019][1676692.483940] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:58:38 2019][1676692.491065] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:58:38 2019][1676692.498368] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:58:38 2019][1676692.505227] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:58:38 2019][1676692.511985] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:58:38 2019][1676692.518383] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:58:38 2019][1676692.524708] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:58:38 2019][1676692.532147] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:58:38 2019][1676692.539070] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:58:38 2019][1676692.547308] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:58:38 2019][1676692.553722] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:58:38 2019][1676692.559949] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:58:38 2019][1676692.566708] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:58:38 2019][1676692.572602] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:58:38 2019][1676692.579742] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:58:38 2019][1676692.587631] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:58:38 2019][1676692.594148] [] kthread+0xd1/0xe0 [Wed Feb 6 03:58:38 2019][1676692.599235] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:58:38 2019][1676692.605899] [] 0xffffffffffffffff [Wed Feb 6 03:58:38 2019][1676692.611095] LustreError: dumping log to /tmp/lustre-log.1549454494.129255 [Wed Feb 6 03:59:12 2019][1676726.007569] Lustre: 17907:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 03:59:12 2019][1676726.007569] req@ffffa08a456bad00 x1624699436454416/t0(0) o36->f6b72b81-0962-ffdd-f885-a85cb99d1184@10.9.104.66@o2ib4:497/0 lens 560/2888 e 0 to 0 dl 1549454532 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 03:59:12 2019][1676726.036987] Lustre: 17907:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 17 previous similar messages [Wed Feb 6 03:59:23 2019][1676737.527656] Pid: 129249, comm: mdt01_021 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:59:23 2019][1676737.537655] Call Trace: [Wed Feb 6 03:59:23 2019][1676737.540290] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:59:23 2019][1676737.547414] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:59:23 2019][1676737.554716] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:59:23 2019][1676737.561565] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:59:23 2019][1676737.568322] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:59:23 2019][1676737.574737] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:59:23 2019][1676737.581063] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:59:23 2019][1676737.588502] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:59:23 2019][1676737.595434] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:59:23 2019][1676737.603651] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:59:23 2019][1676737.610064] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:59:23 2019][1676737.616288] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:59:23 2019][1676737.623045] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:59:23 2019][1676737.628925] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:59:23 2019][1676737.636064] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:59:23 2019][1676737.643968] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:59:23 2019][1676737.650487] [] kthread+0xd1/0xe0 [Wed Feb 6 03:59:23 2019][1676737.655574] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:59:23 2019][1676737.662236] [] 0xffffffffffffffff [Wed Feb 6 03:59:23 2019][1676737.667435] LustreError: dumping log to /tmp/lustre-log.1549454539.129249 [Wed Feb 6 03:59:48 2019][1676762.103854] Pid: 127273, comm: mdt02_009 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:59:48 2019][1676762.113855] Call Trace: [Wed Feb 6 03:59:48 2019][1676762.116501] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:59:48 2019][1676762.123619] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:59:48 2019][1676762.130909] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:59:48 2019][1676762.137763] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:59:48 2019][1676762.144514] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:59:48 2019][1676762.150904] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:59:48 2019][1676762.157217] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:59:48 2019][1676762.164658] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:59:48 2019][1676762.171581] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:59:48 2019][1676762.179790] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:59:48 2019][1676762.186191] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:59:48 2019][1676762.192409] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:59:48 2019][1676762.199157] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:59:48 2019][1676762.205041] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:59:48 2019][1676762.212175] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:59:48 2019][1676762.220063] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:59:48 2019][1676762.226579] [] kthread+0xd1/0xe0 [Wed Feb 6 03:59:48 2019][1676762.231669] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:59:48 2019][1676762.238332] [] 0xffffffffffffffff [Wed Feb 6 03:59:48 2019][1676762.243529] LustreError: dumping log to /tmp/lustre-log.1549454564.127273 [Wed Feb 6 03:59:48 2019][1676762.251076] Pid: 129264, comm: mdt02_018 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:59:48 2019][1676762.261108] Call Trace: [Wed Feb 6 03:59:48 2019][1676762.263743] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:59:48 2019][1676762.270842] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:59:48 2019][1676762.278143] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:59:48 2019][1676762.284978] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:59:48 2019][1676762.291727] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:59:48 2019][1676762.298115] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:59:48 2019][1676762.304435] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:59:48 2019][1676762.311861] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:59:48 2019][1676762.318788] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:59:48 2019][1676762.327009] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:59:48 2019][1676762.333401] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:59:48 2019][1676762.339642] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:59:48 2019][1676762.346375] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:59:48 2019][1676762.352255] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:59:48 2019][1676762.359366] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:59:48 2019][1676762.367265] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:59:48 2019][1676762.373762] [] kthread+0xd1/0xe0 [Wed Feb 6 03:59:48 2019][1676762.378838] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:59:48 2019][1676762.385494] [] 0xffffffffffffffff [Wed Feb 6 03:59:52 2019][1676766.199887] LNet: Service thread pid 126631 was inactive for 1204.20s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 03:59:52 2019][1676766.213092] LustreError: dumping log to /tmp/lustre-log.1549454568.126631 [Wed Feb 6 04:01:51 2019][1676884.984843] LNet: Service thread pid 17806 was inactive for 1201.28s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 04:01:51 2019][1676884.997963] LustreError: dumping log to /tmp/lustre-log.1549454686.17806 [Wed Feb 6 04:02:03 2019][1676897.272946] LustreError: dumping log to /tmp/lustre-log.1549454699.17807 [Wed Feb 6 04:02:15 2019][1676909.561042] LustreError: dumping log to /tmp/lustre-log.1549454711.129247 [Wed Feb 6 04:03:12 2019][1676966.905506] LNet: Service thread pid 17952 was inactive for 1201.18s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 04:03:12 2019][1676966.922705] LNet: Skipped 8 previous similar messages [Wed Feb 6 04:03:13 2019][1676966.927938] Pid: 17952, comm: mdt00_048 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:03:13 2019][1676966.937867] Call Trace: [Wed Feb 6 04:03:13 2019][1676966.940501] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:03:13 2019][1676966.947615] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:03:13 2019][1676966.954901] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:03:13 2019][1676966.961743] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:03:13 2019][1676966.968494] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:03:13 2019][1676966.974889] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:03:13 2019][1676966.981206] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:03:13 2019][1676966.988643] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:03:13 2019][1676966.995551] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:03:13 2019][1676967.003758] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:03:13 2019][1676967.010159] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:03:13 2019][1676967.016379] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:03:13 2019][1676967.023125] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:03:13 2019][1676967.028996] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:03:13 2019][1676967.036127] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:03:13 2019][1676967.044017] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:03:13 2019][1676967.050547] [] kthread+0xd1/0xe0 [Wed Feb 6 04:03:13 2019][1676967.055637] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:03:13 2019][1676967.062301] [] 0xffffffffffffffff [Wed Feb 6 04:03:13 2019][1676967.067503] LustreError: dumping log to /tmp/lustre-log.1549454769.17952 [Wed Feb 6 04:03:13 2019][1676967.074955] Pid: 18186, comm: mdt03_045 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:03:13 2019][1676967.084913] Call Trace: [Wed Feb 6 04:03:13 2019][1676967.087545] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:03:13 2019][1676967.094648] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:03:13 2019][1676967.101935] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:03:13 2019][1676967.108774] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:03:13 2019][1676967.115524] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:03:13 2019][1676967.121914] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:03:13 2019][1676967.128232] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:03:13 2019][1676967.135669] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:03:13 2019][1676967.142574] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:03:13 2019][1676967.150782] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:03:13 2019][1676967.157169] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:03:13 2019][1676967.163384] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:03:13 2019][1676967.170119] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:03:13 2019][1676967.175984] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:03:13 2019][1676967.183093] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:03:13 2019][1676967.190979] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:03:13 2019][1676967.197479] [] kthread+0xd1/0xe0 [Wed Feb 6 04:03:13 2019][1676967.202566] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:03:13 2019][1676967.209214] [] 0xffffffffffffffff [Wed Feb 6 04:03:49 2019][1677003.769799] Pid: 17973, comm: mdt01_044 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:03:49 2019][1677003.779716] Call Trace: [Wed Feb 6 04:03:49 2019][1677003.782360] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:03:49 2019][1677003.789491] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:03:49 2019][1677003.796796] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:03:49 2019][1677003.803661] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:03:49 2019][1677003.810419] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:03:49 2019][1677003.816818] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:03:49 2019][1677003.823140] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:03:49 2019][1677003.830579] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:03:49 2019][1677003.837510] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:03:49 2019][1677003.845728] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:03:49 2019][1677003.852140] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:03:49 2019][1677003.858367] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:03:49 2019][1677003.865123] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:03:49 2019][1677003.871015] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:03:49 2019][1677003.878158] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:03:49 2019][1677003.886048] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:03:49 2019][1677003.892561] [] kthread+0xd1/0xe0 [Wed Feb 6 04:03:49 2019][1677003.897651] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:03:50 2019][1677003.904313] [] 0xffffffffffffffff [Wed Feb 6 04:03:50 2019][1677003.909509] LustreError: dumping log to /tmp/lustre-log.1549454805.17973 [Wed Feb 6 04:04:07 2019][1677021.193231] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 04:04:07 2019][1677021.203843] Lustre: Skipped 98 previous similar messages [Wed Feb 6 04:04:59 2019][1677073.402365] Pid: 129231, comm: mdt01_017 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:04:59 2019][1677073.412369] Call Trace: [Wed Feb 6 04:04:59 2019][1677073.415013] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:04:59 2019][1677073.422135] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:04:59 2019][1677073.429441] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:04:59 2019][1677073.436295] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:04:59 2019][1677073.443062] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:04:59 2019][1677073.449476] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:04:59 2019][1677073.455798] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:04:59 2019][1677073.463247] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:04:59 2019][1677073.470180] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:04:59 2019][1677073.478396] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:04:59 2019][1677073.484809] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:04:59 2019][1677073.491033] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:04:59 2019][1677073.497790] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:04:59 2019][1677073.503670] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:04:59 2019][1677073.510809] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:04:59 2019][1677073.518713] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:04:59 2019][1677073.525232] [] kthread+0xd1/0xe0 [Wed Feb 6 04:04:59 2019][1677073.530320] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:04:59 2019][1677073.536990] [] 0xffffffffffffffff [Wed Feb 6 04:04:59 2019][1677073.542195] LustreError: dumping log to /tmp/lustre-log.1549454875.129231 [Wed Feb 6 04:04:59 2019][1677073.549808] Pid: 129259, comm: mdt00_018 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:04:59 2019][1677073.559833] Call Trace: [Wed Feb 6 04:04:59 2019][1677073.562469] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:04:59 2019][1677073.569577] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:04:59 2019][1677073.576874] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:04:59 2019][1677073.583737] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:04:59 2019][1677073.590473] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:04:59 2019][1677073.596875] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:04:59 2019][1677073.603180] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:04:59 2019][1677073.610629] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:04:59 2019][1677073.617541] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:04:59 2019][1677073.625770] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:04:59 2019][1677073.632169] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:04:59 2019][1677073.638397] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:04:59 2019][1677073.645144] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:04:59 2019][1677073.651047] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:04:59 2019][1677073.658161] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:04:59 2019][1677073.666071] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:04:59 2019][1677073.672583] [] kthread+0xd1/0xe0 [Wed Feb 6 04:04:59 2019][1677073.677676] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:04:59 2019][1677073.684316] [] 0xffffffffffffffff [Wed Feb 6 04:04:59 2019][1677073.689510] LNet: Service thread pid 17822 was inactive for 1202.49s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 04:04:59 2019][1677073.702623] LNet: Skipped 2 previous similar messages [Wed Feb 6 04:05:36 2019][1677110.266662] LustreError: dumping log to /tmp/lustre-log.1549454912.129243 [Wed Feb 6 04:05:55 2019][1677129.714825] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 04:05:55 2019][1677129.730976] Lustre: Skipped 1 previous similar message [Wed Feb 6 04:05:55 2019][1677129.736313] LustreError: 18266:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549454631, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa0541574f740/0x8c1de5394223a1cf lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 102 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935188fa7d75 expref: -99 pid: 18266 timeout: 0 lvb_type: 0 [Wed Feb 6 04:05:55 2019][1677129.776688] LustreError: 18266:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 4 previous similar messages [Wed Feb 6 04:06:13 2019][1677147.130964] LustreError: dumping log to /tmp/lustre-log.1549454949.129260 [Wed Feb 6 04:06:41 2019][1677175.803209] LustreError: dumping log to /tmp/lustre-log.1549454977.18154 [Wed Feb 6 04:08:38 2019][1677292.114448] Lustre: fir-MDT0003: Client c50e9e63-bc69-ffb4-d9c5-0a1d77a8b849 (at 10.9.106.60@o2ib4) reconnecting [Wed Feb 6 04:08:38 2019][1677292.124799] Lustre: Skipped 93 previous similar messages [Wed Feb 6 04:09:17 2019][1677331.452488] Pid: 17969, comm: mdt03_038 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:09:17 2019][1677331.462401] Call Trace: [Wed Feb 6 04:09:17 2019][1677331.465045] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:09:17 2019][1677331.472165] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:09:17 2019][1677331.479465] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:09:17 2019][1677331.486311] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:09:17 2019][1677331.493068] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:09:17 2019][1677331.499467] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:09:17 2019][1677331.505793] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:09:17 2019][1677331.513240] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:09:17 2019][1677331.520172] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:09:17 2019][1677331.528405] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:09:17 2019][1677331.534816] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:09:17 2019][1677331.541041] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:09:17 2019][1677331.547798] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:09:17 2019][1677331.553678] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:09:17 2019][1677331.560807] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:09:17 2019][1677331.568707] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:09:17 2019][1677331.575220] [] kthread+0xd1/0xe0 [Wed Feb 6 04:09:17 2019][1677331.580310] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:09:17 2019][1677331.586973] [] 0xffffffffffffffff [Wed Feb 6 04:09:17 2019][1677331.592193] LustreError: dumping log to /tmp/lustre-log.1549455133.17969 [Wed Feb 6 04:11:20 2019][1677454.333434] Pid: 17930, comm: mdt02_038 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:11:20 2019][1677454.343354] Call Trace: [Wed Feb 6 04:11:20 2019][1677454.345995] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:11:20 2019][1677454.353117] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:11:20 2019][1677454.360398] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:11:20 2019][1677454.367244] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:11:20 2019][1677454.374002] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:11:20 2019][1677454.380402] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:11:20 2019][1677454.386727] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:11:20 2019][1677454.394162] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:11:20 2019][1677454.401098] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:11:20 2019][1677454.409312] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:11:20 2019][1677454.415731] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:11:20 2019][1677454.421951] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:11:20 2019][1677454.428696] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:11:20 2019][1677454.434576] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:11:20 2019][1677454.441705] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:11:20 2019][1677454.449597] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:11:20 2019][1677454.456112] [] kthread+0xd1/0xe0 [Wed Feb 6 04:11:20 2019][1677454.461202] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:11:20 2019][1677454.467864] [] 0xffffffffffffffff [Wed Feb 6 04:11:20 2019][1677454.473082] LustreError: dumping log to /tmp/lustre-log.1549455256.17930 [Wed Feb 6 04:11:20 2019][1677454.480547] Pid: 17934, comm: mdt03_033 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:11:20 2019][1677454.490476] Call Trace: [Wed Feb 6 04:11:20 2019][1677454.493109] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:11:20 2019][1677454.500208] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:11:20 2019][1677454.507496] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:11:20 2019][1677454.514335] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:11:20 2019][1677454.521084] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:11:20 2019][1677454.527475] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:11:20 2019][1677454.533789] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:11:20 2019][1677454.541235] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:11:20 2019][1677454.548159] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:11:20 2019][1677454.556369] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:11:20 2019][1677454.562772] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:11:20 2019][1677454.568988] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:11:20 2019][1677454.575737] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:11:20 2019][1677454.581608] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:11:20 2019][1677454.588728] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:11:20 2019][1677454.596619] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:11:20 2019][1677454.603132] [] kthread+0xd1/0xe0 [Wed Feb 6 04:11:20 2019][1677454.608230] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:11:20 2019][1677454.614883] [] 0xffffffffffffffff [Wed Feb 6 04:11:22 2019][1677456.074390] Lustre: fir-MDT0003: haven't heard from client c5acf83e-6f6d-4eb5-1fed-175307bafafe (at 10.8.10.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa07999a1dc00, cur 1549455258 expire 1549455108 last 1549455031 [Wed Feb 6 04:11:22 2019][1677456.096281] Lustre: Skipped 1 previous similar message [Wed Feb 6 04:13:25 2019][1677579.522449] Lustre: 129250:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 04:13:25 2019][1677579.522449] req@ffffa06feb710c00 x1624699291157552/t0(0) o36->01d5ada9-338b-fc1a-1541-7fe86bd87ddc@10.9.105.18@o2ib4:596/0 lens 560/2888 e 0 to 0 dl 1549455386 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 04:13:25 2019][1677579.551952] Lustre: 129250:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 5 previous similar messages [Wed Feb 6 04:14:08 2019][1677622.280319] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 04:14:08 2019][1677622.290931] Lustre: Skipped 100 previous similar messages [Wed Feb 6 04:14:45 2019][1677659.135108] LNet: Service thread pid 18269 was inactive for 1203.26s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 04:14:45 2019][1677659.152301] LNet: Skipped 7 previous similar messages [Wed Feb 6 04:14:45 2019][1677659.157533] Pid: 18269, comm: mdt02_052 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:14:45 2019][1677659.167462] Call Trace: [Wed Feb 6 04:14:45 2019][1677659.170097] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:14:45 2019][1677659.177220] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:14:45 2019][1677659.184515] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:14:45 2019][1677659.191364] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:14:45 2019][1677659.198120] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:14:45 2019][1677659.204519] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:14:45 2019][1677659.210859] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:14:45 2019][1677659.218299] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:14:45 2019][1677659.225222] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:14:45 2019][1677659.233439] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:14:45 2019][1677659.239843] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:14:45 2019][1677659.246069] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:14:45 2019][1677659.252824] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:14:45 2019][1677659.258695] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:14:45 2019][1677659.265835] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:14:45 2019][1677659.273740] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:14:45 2019][1677659.280270] [] kthread+0xd1/0xe0 [Wed Feb 6 04:14:45 2019][1677659.285355] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:14:45 2019][1677659.292031] [] 0xffffffffffffffff [Wed Feb 6 04:14:45 2019][1677659.297222] LustreError: dumping log to /tmp/lustre-log.1549455461.18269 [Wed Feb 6 04:15:54 2019][1677728.767661] Pid: 129587, comm: mdt00_023 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:15:54 2019][1677728.777661] Call Trace: [Wed Feb 6 04:15:54 2019][1677728.780295] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:15:54 2019][1677728.787417] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:15:54 2019][1677728.794734] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:15:54 2019][1677728.801582] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:15:54 2019][1677728.808338] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:15:54 2019][1677728.814736] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:15:54 2019][1677728.821061] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:15:54 2019][1677728.828499] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:15:54 2019][1677728.835429] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:15:54 2019][1677728.843648] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:15:54 2019][1677728.850071] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:15:54 2019][1677728.856294] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:15:54 2019][1677728.863041] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:15:54 2019][1677728.868923] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:15:54 2019][1677728.876061] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:15:54 2019][1677728.883952] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:15:54 2019][1677728.890458] [] kthread+0xd1/0xe0 [Wed Feb 6 04:15:54 2019][1677728.895544] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:15:54 2019][1677728.902209] [] 0xffffffffffffffff [Wed Feb 6 04:15:54 2019][1677728.907402] LustreError: dumping log to /tmp/lustre-log.1549455530.129587 [Wed Feb 6 04:18:39 2019][1677893.187060] Lustre: fir-MDT0003: Client dc527ec4-db62-14c0-459e-fcb26c78037c (at 10.9.107.8@o2ib4) reconnecting [Wed Feb 6 04:18:39 2019][1677893.197319] Lustre: Skipped 97 previous similar messages [Wed Feb 6 04:20:57 2019][1678031.874096] Pid: 18266, comm: mdt02_050 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:20:57 2019][1678031.884010] Call Trace: [Wed Feb 6 04:20:57 2019][1678031.886650] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:20:57 2019][1678031.893764] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:20:57 2019][1678031.901063] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:20:57 2019][1678031.907900] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:20:57 2019][1678031.914658] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:20:57 2019][1678031.921046] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:20:57 2019][1678031.927364] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:20:57 2019][1678031.934807] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:20:58 2019][1678031.941739] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:20:58 2019][1678031.949965] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:20:58 2019][1678031.956370] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:20:58 2019][1678031.962611] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:20:58 2019][1678031.969356] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:20:58 2019][1678031.975247] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:20:58 2019][1678031.982390] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:20:58 2019][1678031.990282] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:20:58 2019][1678031.996802] [] kthread+0xd1/0xe0 [Wed Feb 6 04:20:58 2019][1678032.001890] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:20:58 2019][1678032.008540] [] 0xffffffffffffffff [Wed Feb 6 04:20:58 2019][1678032.013749] LustreError: dumping log to /tmp/lustre-log.1549455833.18266 [Wed Feb 6 04:21:02 2019][1678035.970124] Pid: 129262, comm: mdt00_019 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:21:02 2019][1678035.980128] Call Trace: [Wed Feb 6 04:21:02 2019][1678035.982773] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:21:02 2019][1678035.989903] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:21:02 2019][1678035.997216] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:21:02 2019][1678036.004067] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:21:02 2019][1678036.010829] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:21:02 2019][1678036.017222] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:21:02 2019][1678036.023546] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:21:02 2019][1678036.030985] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:21:02 2019][1678036.037914] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:21:02 2019][1678036.046131] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:21:02 2019][1678036.052542] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:21:02 2019][1678036.058758] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:21:02 2019][1678036.065532] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:21:02 2019][1678036.071428] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:21:02 2019][1678036.078569] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:21:02 2019][1678036.086459] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:21:02 2019][1678036.092977] [] kthread+0xd1/0xe0 [Wed Feb 6 04:21:02 2019][1678036.098062] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:21:02 2019][1678036.104728] [] 0xffffffffffffffff [Wed Feb 6 04:21:02 2019][1678036.109921] LustreError: dumping log to /tmp/lustre-log.1549455838.129262 [Wed Feb 6 04:21:30 2019][1678064.642355] Pid: 18165, comm: mdt02_043 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:21:30 2019][1678064.652264] Call Trace: [Wed Feb 6 04:21:30 2019][1678064.654901] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:21:30 2019][1678064.662024] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:21:30 2019][1678064.669321] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:21:30 2019][1678064.676172] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:21:30 2019][1678064.682925] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:21:30 2019][1678064.689323] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:21:30 2019][1678064.695661] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:21:30 2019][1678064.703095] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:21:30 2019][1678064.710025] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:21:30 2019][1678064.718244] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:21:30 2019][1678064.724653] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:21:30 2019][1678064.730880] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:21:30 2019][1678064.737636] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:21:30 2019][1678064.743508] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:21:30 2019][1678064.750645] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:21:30 2019][1678064.758537] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:21:30 2019][1678064.765066] [] kthread+0xd1/0xe0 [Wed Feb 6 04:21:30 2019][1678064.770158] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:21:30 2019][1678064.776825] [] 0xffffffffffffffff [Wed Feb 6 04:21:30 2019][1678064.782016] LustreError: dumping log to /tmp/lustre-log.1549455866.18165 [Wed Feb 6 04:22:07 2019][1678101.506660] Pid: 127214, comm: mdt00_007 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:22:07 2019][1678101.516661] Call Trace: [Wed Feb 6 04:22:07 2019][1678101.519296] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:22:07 2019][1678101.526429] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:22:07 2019][1678101.533708] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:22:07 2019][1678101.540555] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:22:07 2019][1678101.547311] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:22:07 2019][1678101.553711] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:22:07 2019][1678101.560034] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:22:07 2019][1678101.567473] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:22:07 2019][1678101.574405] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:22:07 2019][1678101.582622] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:22:07 2019][1678101.589049] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:22:07 2019][1678101.595276] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:22:07 2019][1678101.602034] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:22:07 2019][1678101.607912] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:22:07 2019][1678101.615052] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:22:07 2019][1678101.622942] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:22:07 2019][1678101.629472] [] kthread+0xd1/0xe0 [Wed Feb 6 04:22:07 2019][1678101.634563] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:22:07 2019][1678101.641224] [] 0xffffffffffffffff [Wed Feb 6 04:22:07 2019][1678101.646423] LustreError: dumping log to /tmp/lustre-log.1549455903.127214 [Wed Feb 6 04:24:09 2019][1678223.366444] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 04:24:09 2019][1678223.377053] Lustre: Skipped 103 previous similar messages [Wed Feb 6 04:28:40 2019][1678494.254243] Lustre: fir-MDT0003: Client c50e9e63-bc69-ffb4-d9c5-0a1d77a8b849 (at 10.9.106.60@o2ib4) reconnecting [Wed Feb 6 04:28:40 2019][1678494.264595] Lustre: Skipped 95 previous similar messages [Wed Feb 6 04:34:10 2019][1678824.452500] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 04:34:10 2019][1678824.463111] Lustre: Skipped 96 previous similar messages [Wed Feb 6 04:38:41 2019][1679095.202683] Lustre: fir-MDT0003: Client 64d3667d-1c77-7867-b1f3-ec9c4a6035ad (at 10.9.104.56@o2ib4) reconnecting [Wed Feb 6 04:38:41 2019][1679095.213034] Lustre: Skipped 94 previous similar messages [Wed Feb 6 04:41:50 2019][1679284.236187] LNet: Service thread pid 126625 was inactive for 200.18s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 04:41:50 2019][1679284.253382] LNet: Skipped 5 previous similar messages [Wed Feb 6 04:41:50 2019][1679284.258617] Pid: 126625, comm: mdt00_002 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:41:50 2019][1679284.268616] Call Trace: [Wed Feb 6 04:41:50 2019][1679284.271259] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.278383] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.285669] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.292517] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:41:50 2019][1679284.299276] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:41:50 2019][1679284.305672] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:41:50 2019][1679284.311996] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:41:50 2019][1679284.319436] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:41:50 2019][1679284.326381] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:41:50 2019][1679284.334603] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:41:50 2019][1679284.341012] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:41:50 2019][1679284.347240] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:41:50 2019][1679284.353986] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:41:50 2019][1679284.359866] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.366997] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.374887] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.381400] [] kthread+0xd1/0xe0 [Wed Feb 6 04:41:50 2019][1679284.386491] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:41:50 2019][1679284.393167] [] 0xffffffffffffffff [Wed Feb 6 04:41:50 2019][1679284.398365] LustreError: dumping log to /tmp/lustre-log.1549457086.126625 [Wed Feb 6 04:41:50 2019][1679284.406154] Pid: 127970, comm: mdt00_010 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:41:50 2019][1679284.416185] Call Trace: [Wed Feb 6 04:41:50 2019][1679284.418827] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.425925] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.433212] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.440060] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:41:50 2019][1679284.446809] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:41:50 2019][1679284.453198] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:41:50 2019][1679284.459525] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:41:50 2019][1679284.466953] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:41:50 2019][1679284.473875] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:41:50 2019][1679284.482093] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:41:50 2019][1679284.488504] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:41:50 2019][1679284.494725] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:41:50 2019][1679284.501493] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:41:50 2019][1679284.507374] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.514504] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.522393] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.528923] [] kthread+0xd1/0xe0 [Wed Feb 6 04:41:50 2019][1679284.534016] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:41:50 2019][1679284.540670] [] 0xffffffffffffffff [Wed Feb 6 04:41:50 2019][1679284.545850] Pid: 17940, comm: mdt00_047 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:41:50 2019][1679284.555777] Call Trace: [Wed Feb 6 04:41:50 2019][1679284.558411] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.565536] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.572817] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.579684] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:41:50 2019][1679284.586426] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:41:50 2019][1679284.592842] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:41:50 2019][1679284.599148] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:41:50 2019][1679284.606588] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:41:50 2019][1679284.613509] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:41:50 2019][1679284.621730] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:41:50 2019][1679284.628129] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:41:50 2019][1679284.634365] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:41:50 2019][1679284.641111] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:41:50 2019][1679284.646993] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.654102] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.662018] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.668515] [] kthread+0xd1/0xe0 [Wed Feb 6 04:41:50 2019][1679284.673611] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:41:50 2019][1679284.680252] [] 0xffffffffffffffff [Wed Feb 6 04:41:50 2019][1679284.685442] Pid: 17959, comm: mdt03_035 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:41:50 2019][1679284.695351] Call Trace: [Wed Feb 6 04:41:50 2019][1679284.697991] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.705090] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.712378] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.719225] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:41:50 2019][1679284.725989] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:41:50 2019][1679284.732382] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:41:50 2019][1679284.738695] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:41:50 2019][1679284.746129] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:41:50 2019][1679284.753051] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:41:50 2019][1679284.761269] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:41:50 2019][1679284.767678] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:41:50 2019][1679284.773905] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:41:50 2019][1679284.780650] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:41:50 2019][1679284.786523] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.793657] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.801541] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.808055] [] kthread+0xd1/0xe0 [Wed Feb 6 04:41:50 2019][1679284.813140] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:41:50 2019][1679284.819798] [] 0xffffffffffffffff [Wed Feb 6 04:41:50 2019][1679284.824981] Pid: 18566, comm: mdt02_056 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:41:50 2019][1679284.834906] Call Trace: [Wed Feb 6 04:41:50 2019][1679284.837542] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.844666] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.851941] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.858813] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:41:50 2019][1679284.865546] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:41:50 2019][1679284.871949] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:41:50 2019][1679284.878254] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:41:50 2019][1679284.885704] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:41:50 2019][1679284.892622] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:41:50 2019][1679284.900850] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:41:50 2019][1679284.907251] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:41:50 2019][1679284.913489] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:41:50 2019][1679284.920225] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:41:50 2019][1679284.926120] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.933224] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.941128] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:41:50 2019][1679284.947631] [] kthread+0xd1/0xe0 [Wed Feb 6 04:41:51 2019][1679284.952734] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:41:51 2019][1679284.959382] [] 0xffffffffffffffff [Wed Feb 6 04:42:25 2019][1679319.052468] LNet: Service thread pid 17907 was inactive for 200.76s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 04:42:25 2019][1679319.065504] LNet: Skipped 3 previous similar messages [Wed Feb 6 04:42:25 2019][1679319.070739] LustreError: dumping log to /tmp/lustre-log.1549457121.17907 [Wed Feb 6 04:43:30 2019][1679384.051991] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 04:43:30 2019][1679384.068163] LustreError: 126625:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549456886, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa04db3a37080/0x8c1de53942325aa7 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 109 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518f212d50 expref: -99 pid: 126625 timeout: 0 lvb_type: 0 [Wed Feb 6 04:43:30 2019][1679384.108679] LustreError: 126625:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 3 previous similar messages [Wed Feb 6 04:44:11 2019][1679425.538702] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 04:44:11 2019][1679425.549313] Lustre: Skipped 97 previous similar messages [Wed Feb 6 04:48:24 2019][1679678.698373] LustreError: 17953:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549457180, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa055076ea640/0x8c1de539423440f6 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 109 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93519015acb0 expref: -99 pid: 17953 timeout: 0 lvb_type: 0 [Wed Feb 6 04:48:24 2019][1679678.738706] LustreError: 17953:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 5 previous similar messages [Wed Feb 6 04:48:25 2019][1679679.513383] Lustre: 17911:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 04:48:25 2019][1679679.513383] req@ffffa0579aae5d00 x1624700593089248/t0(0) o36->fabd17e2-268b-b1ba-b568-7f6550034520@10.9.106.16@o2ib4:431/0 lens 560/2888 e 24 to 0 dl 1549457486 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 04:48:25 2019][1679679.542631] Lustre: 17911:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages [Wed Feb 6 04:48:42 2019][1679696.377121] Lustre: fir-MDT0003: Client dc527ec4-db62-14c0-459e-fcb26c78037c (at 10.9.107.8@o2ib4) reconnecting [Wed Feb 6 04:48:42 2019][1679696.387385] Lustre: Skipped 106 previous similar messages [Wed Feb 6 04:53:19 2019][1679973.621753] Lustre: 17916:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 04:53:19 2019][1679973.621753] req@ffffa05ab396ce00 x1624699247211056/t0(0) o36->7d9770de-7aa6-9591-8f67-d8757265aaf1@10.9.105.37@o2ib4:725/0 lens 528/2888 e 1 to 0 dl 1549457780 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 04:53:19 2019][1679973.650915] Lustre: 17916:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 5 previous similar messages [Wed Feb 6 04:53:37 2019][1679991.825897] LNet: Service thread pid 17953 was inactive for 613.12s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 04:53:37 2019][1679991.843011] LNet: Skipped 4 previous similar messages [Wed Feb 6 04:53:37 2019][1679991.848242] Pid: 17953, comm: mdt00_049 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:53:37 2019][1679991.858177] Call Trace: [Wed Feb 6 04:53:37 2019][1679991.860813] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:53:37 2019][1679991.867928] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:53:37 2019][1679991.875226] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:53:37 2019][1679991.882077] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:53:37 2019][1679991.888839] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:53:37 2019][1679991.895252] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:53:37 2019][1679991.901579] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:53:37 2019][1679991.909017] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:53:37 2019][1679991.915950] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:53:37 2019][1679991.924184] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:53:37 2019][1679991.930593] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:53:37 2019][1679991.936821] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:53:37 2019][1679991.943577] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:53:37 2019][1679991.949448] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:53:38 2019][1679991.956593] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:53:38 2019][1679991.964486] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:53:38 2019][1679991.971001] [] kthread+0xd1/0xe0 [Wed Feb 6 04:53:38 2019][1679991.976090] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:53:38 2019][1679991.982752] [] 0xffffffffffffffff [Wed Feb 6 04:53:38 2019][1679991.987948] LustreError: dumping log to /tmp/lustre-log.1549457793.17953 [Wed Feb 6 04:54:12 2019][1680026.624580] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 04:54:12 2019][1680026.635196] Lustre: Skipped 107 previous similar messages [Wed Feb 6 04:58:43 2019][1680297.459174] Lustre: fir-MDT0003: Client dc527ec4-db62-14c0-459e-fcb26c78037c (at 10.9.107.8@o2ib4) reconnecting [Wed Feb 6 04:58:43 2019][1680297.469438] Lustre: Skipped 103 previous similar messages [Wed Feb 6 05:04:13 2019][1680627.711287] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 05:04:13 2019][1680627.721892] Lustre: Skipped 106 previous similar messages [Wed Feb 6 05:08:44 2019][1680898.521293] Lustre: fir-MDT0003: Client c50e9e63-bc69-ffb4-d9c5-0a1d77a8b849 (at 10.9.106.60@o2ib4) reconnecting [Wed Feb 6 05:08:44 2019][1680898.531650] Lustre: Skipped 104 previous similar messages [Wed Feb 6 05:14:14 2019][1681228.798448] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 05:14:14 2019][1681228.809061] Lustre: Skipped 110 previous similar messages [Wed Feb 6 05:18:45 2019][1681499.605804] Lustre: fir-MDT0003: Client dc527ec4-db62-14c0-459e-fcb26c78037c (at 10.9.107.8@o2ib4) reconnecting [Wed Feb 6 05:18:45 2019][1681499.616072] Lustre: Skipped 104 previous similar messages [Wed Feb 6 05:24:15 2019][1681829.885723] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 05:24:15 2019][1681829.896327] Lustre: Skipped 105 previous similar messages [Wed Feb 6 05:28:46 2019][1682100.675479] Lustre: fir-MDT0003: Client c50e9e63-bc69-ffb4-d9c5-0a1d77a8b849 (at 10.9.106.60@o2ib4) reconnecting [Wed Feb 6 05:28:46 2019][1682100.685828] Lustre: Skipped 103 previous similar messages [Wed Feb 6 05:33:55 2019][1682409.509386] LNet: Service thread pid 127297 was inactive for 200.19s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 05:33:55 2019][1682409.526580] Pid: 127297, comm: mdt02_010 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 05:33:55 2019][1682409.536580] Call Trace: [Wed Feb 6 05:33:55 2019][1682409.539224] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 05:33:55 2019][1682409.546353] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 05:33:55 2019][1682409.553659] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 05:33:55 2019][1682409.560509] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 05:33:55 2019][1682409.567273] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 05:33:55 2019][1682409.573671] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 05:33:55 2019][1682409.579996] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 05:33:55 2019][1682409.587433] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 05:33:55 2019][1682409.594365] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 05:33:55 2019][1682409.602583] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 05:33:55 2019][1682409.608972] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 05:33:55 2019][1682409.615215] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 05:33:55 2019][1682409.621953] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 05:33:55 2019][1682409.627835] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 05:33:55 2019][1682409.634967] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 05:33:55 2019][1682409.642875] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 05:33:55 2019][1682409.649377] [] kthread+0xd1/0xe0 [Wed Feb 6 05:33:55 2019][1682409.654477] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 05:33:55 2019][1682409.661127] [] 0xffffffffffffffff [Wed Feb 6 05:33:55 2019][1682409.666334] LustreError: dumping log to /tmp/lustre-log.1549460211.127297 [Wed Feb 6 05:34:17 2019][1682430.972804] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 05:34:17 2019][1682430.983416] Lustre: Skipped 103 previous similar messages [Wed Feb 6 05:35:35 2019][1682509.312186] Lustre: fir-MDT0000-osp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 05:35:35 2019][1682509.328351] LustreError: 127297:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549460011, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0001 lock: ffffa07ad386e9c0/0x8c1de53942464230 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93519b6e7a9f expref: -99 pid: 127297 timeout: 0 lvb_type: 0 [Wed Feb 6 05:35:58 2019][1682532.390378] LNet: Service thread pid 17894 was inactive for 200.33s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 05:35:58 2019][1682532.407486] Pid: 17894, comm: mdt00_034 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 05:35:58 2019][1682532.417402] Call Trace: [Wed Feb 6 05:35:58 2019][1682532.420044] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 05:35:58 2019][1682532.427165] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 05:35:58 2019][1682532.434461] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 05:35:58 2019][1682532.441310] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 05:35:58 2019][1682532.448060] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 05:35:58 2019][1682532.454458] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 05:35:58 2019][1682532.460797] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 05:35:58 2019][1682532.468238] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 05:35:58 2019][1682532.475168] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 05:35:58 2019][1682532.483387] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 05:35:58 2019][1682532.489800] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 05:35:58 2019][1682532.496023] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 05:35:58 2019][1682532.502780] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 05:35:58 2019][1682532.508658] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 05:35:58 2019][1682532.515798] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 05:35:58 2019][1682532.523690] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 05:35:58 2019][1682532.530226] [] kthread+0xd1/0xe0 [Wed Feb 6 05:35:58 2019][1682532.535318] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 05:35:58 2019][1682532.541981] [] 0xffffffffffffffff [Wed Feb 6 05:35:58 2019][1682532.547177] LustreError: dumping log to /tmp/lustre-log.1549460334.17894 [Wed Feb 6 05:37:38 2019][1682632.056185] LustreError: 17894:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549460133, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0001 lock: ffffa059e09b0d80/0x8c1de53942470ba1 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93519bbf8468 expref: -99 pid: 17894 timeout: 0 lvb_type: 0 [Wed Feb 6 05:38:47 2019][1682701.745495] Lustre: fir-MDT0003: Client dc527ec4-db62-14c0-459e-fcb26c78037c (at 10.9.107.8@o2ib4) reconnecting [Wed Feb 6 05:38:47 2019][1682701.755760] Lustre: Skipped 106 previous similar messages [Wed Feb 6 05:40:30 2019][1682804.158572] Lustre: 18267:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 05:40:30 2019][1682804.158572] req@ffffa0777e5e8f00 x1624676066262640/t0(0) o36->a26bda41-9e3a-f8bb-aa63-ba992cd69aad@10.9.0.62@o2ib4:536/0 lens 520/2888 e 24 to 0 dl 1549460611 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 05:42:32 2019][1682926.273552] Lustre: 17916:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 05:42:32 2019][1682926.273552] req@ffffa089d76d9e00 x1624676066761408/t0(0) o36->801be733-785b-a675-6116-74f5d07a121a@10.9.0.61@o2ib4:658/0 lens 520/2888 e 18 to 0 dl 1549460733 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 05:44:18 2019][1683032.059891] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 05:44:18 2019][1683032.070512] Lustre: Skipped 112 previous similar messages [Wed Feb 6 05:48:48 2019][1683302.819978] Lustre: fir-MDT0003: Client c50e9e63-bc69-ffb4-d9c5-0a1d77a8b849 (at 10.9.106.60@o2ib4) reconnecting [Wed Feb 6 05:48:48 2019][1683302.830329] Lustre: Skipped 105 previous similar messages [Wed Feb 6 05:54:19 2019][1683633.146752] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 05:54:19 2019][1683633.157365] Lustre: Skipped 105 previous similar messages [Wed Feb 6 05:58:49 2019][1683903.880449] Lustre: fir-MDT0003: Client dc527ec4-db62-14c0-459e-fcb26c78037c (at 10.9.107.8@o2ib4) reconnecting [Wed Feb 6 05:58:49 2019][1683903.890711] Lustre: Skipped 107 previous similar messages [Wed Feb 6 06:04:20 2019][1684234.233754] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 06:04:20 2019][1684234.244364] Lustre: Skipped 112 previous similar messages [Wed Feb 6 06:08:50 2019][1684504.964278] Lustre: fir-MDT0003: Client c50e9e63-bc69-ffb4-d9c5-0a1d77a8b849 (at 10.9.106.60@o2ib4) reconnecting [Wed Feb 6 06:08:50 2019][1684504.974630] Lustre: Skipped 106 previous similar messages [Wed Feb 6 06:14:21 2019][1684835.320571] Lustre: fir-MDT0003: Connection restored to 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) [Wed Feb 6 06:14:21 2019][1684835.331183] Lustre: Skipped 104 previous similar messages [Wed Feb 6 06:18:52 2019][1685106.028020] Lustre: fir-MDT0003: Client dc527ec4-db62-14c0-459e-fcb26c78037c (at 10.9.107.8@o2ib4) reconnecting [Wed Feb 6 06:18:52 2019][1685106.038293] Lustre: Skipped 105 previous similar messages [Wed Feb 6 06:20:34 2019][1685208.637889] LNet: Service thread pid 129279 completed after 10321.72s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 06:20:34 2019][1685208.654494] LNet: Skipped 10 previous similar messages [Wed Feb 6 06:23:05 2019][1685359.555030] LNet: Service thread pid 17805 completed after 10299.84s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 06:23:05 2019][1685359.567840] Lustre: 17840:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (10217:1s); client may timeout. req@ffffa07752e05a00 x1624669478053248/t8857365247(0) o36->5c820789-8e24-ad89-a0df-b1759dd671b0@10.9.101.56@o2ib4:263/0 lens 520/424 e 1 to 0 dl 1549463160 ref 1 fl Complete:/0/0 rc 0/0 [Wed Feb 6 06:23:05 2019][1685359.601057] LNet: Skipped 50 previous similar messages [Wed Feb 6 06:23:05 2019][1685359.645892] LNetError: 129249:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.19.4@tcp from [Wed Feb 6 06:23:05 2019][1685359.656591] LNetError: 129249:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 12783630 previous similar messages [Wed Feb 6 06:23:05 2019][1685359.667915] Lustre: 129249:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549463161/real 1549463161] req@ffffa062a6867b00 x1623667908490528/t0(0) o104->fir-MDT0001@10.10.19.4@tcp:15/16 lens 296/224 e 0 to 1 dl 1549463172 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 [Wed Feb 6 06:23:05 2019][1685359.695641] Lustre: 129249:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 12783610 previous similar messages [Wed Feb 6 06:24:20 2019][1685434.646761] LNetError: 129249:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.19.4@tcp from [Wed Feb 6 06:24:20 2019][1685434.657460] LNetError: 129249:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 10110226 previous similar messages [Wed Feb 6 06:24:20 2019][1685434.668774] Lustre: 129249:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549463236/real 1549463236] req@ffffa062a6867b00 x1623667908490528/t0(0) o104->fir-MDT0001@10.10.19.4@tcp:15/16 lens 296/224 e 0 to 1 dl 1549463247 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 06:24:20 2019][1685434.696472] Lustre: 129249:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10110226 previous similar messages [Wed Feb 6 06:25:34 2019][1685508.672383] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 06:25:34 2019][1685508.682383] LustreError: 129796:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549463010, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0001 lock: ffffa0634f134140/0x8c1de5394259a73d lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a3dedc66 expref: -99 pid: 129796 timeout: 0 lvb_type: 0 [Wed Feb 6 06:25:34 2019][1685508.728874] Lustre: Skipped 1 previous similar message [Wed Feb 6 06:25:34 2019][1685508.734384] Lustre: fir-MDT0000-osp-MDT0001: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [Wed Feb 6 06:25:34 2019][1685508.744215] Lustre: Skipped 99 previous similar messages [Wed Feb 6 06:25:36 2019][1685510.431387] LustreError: 129249:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.19.4@tcp) failed to reply to blocking AST (req@ffffa062a6867b00 x1623667908490528 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffffa05ce7f50d80/0x8c1de53942598a79 lrc: 4/0,0 mode: PR/PR res: [0x240000ce5:0x4:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a76f25 expref: 11 pid: 847 timeout: 1685645 lvb_type: 0 [Wed Feb 6 06:25:36 2019][1685510.473632] LustreError: 129249:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages [Wed Feb 6 06:25:36 2019][1685510.483981] LustreError: 138-a: fir-MDT0001: A client on nid 10.10.19.4@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 06:25:36 2019][1685510.496512] LustreError: Skipped 3 previous similar messages [Wed Feb 6 06:25:36 2019][1685510.502370] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.19.4@tcp ns: mdt-fir-MDT0001_UUID lock: ffffa05ce7f50d80/0x8c1de53942598a79 lrc: 3/0,0 mode: PR/PR res: [0x240000ce5:0x4:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a76f25 expref: 12 pid: 847 timeout: 0 lvb_type: 0 [Wed Feb 6 06:25:36 2019][1685510.539175] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages [Wed Feb 6 06:25:36 2019][1685510.549653] LNet: Service thread pid 129249 completed after 9973.98s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 06:25:36 2019][1685510.566157] LNet: Skipped 14 previous similar messages [Wed Feb 6 06:26:39 2019][1685573.166893] LustreError: 17834:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549463075, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa0795878f740/0x8c1de539425da57c lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 95 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a3e08e20 expref: -99 pid: 17834 timeout: 0 lvb_type: 0 [Wed Feb 6 06:26:39 2019][1685573.207143] LustreError: 17834:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 31 previous similar messages [Wed Feb 6 06:27:58 2019][1685652.100358] Lustre: fir-MDT0001: haven't heard from client 375c01b4-7d81-9426-7dfa-7b0a1dfe6d49 (at 10.8.6.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa06398b9e800, cur 1549463454 expire 1549463304 last 1549463227 [Wed Feb 6 06:27:58 2019][1685652.122063] Lustre: Skipped 1 previous similar message [Wed Feb 6 06:28:22 2019][1685676.118366] LNetError: 17840:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.11@tcp from [Wed Feb 6 06:28:22 2019][1685676.129153] LNetError: 17840:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 10197414 previous similar messages [Wed Feb 6 06:28:22 2019][1685676.140421] Lustre: 17840:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549463478/real 1549463478] req@ffffa0778f019200 x1623667908787584/t0(0) o104->fir-MDT0003@10.10.114.11@tcp:15/16 lens 296/224 e 0 to 1 dl 1549463489 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 [Wed Feb 6 06:28:22 2019][1685676.168210] Lustre: 17840:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10197414 previous similar messages [Wed Feb 6 06:31:48 2019][1685882.567388] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 06:31:48 2019][1685882.583558] LustreError: 17860:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549463384, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa061d108f2c0/0x8c1de5394265740a lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 93 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a4ad1508 expref: -99 pid: 17860 timeout: 0 lvb_type: 0 [Wed Feb 6 06:31:48 2019][1685882.623805] LustreError: 17860:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 14 previous similar messages [Wed Feb 6 06:33:22 2019][1685976.121130] LNetError: 17997:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.19.4@tcp from [Wed Feb 6 06:33:22 2019][1685976.131738] LNetError: 17997:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 45088611 previous similar messages [Wed Feb 6 06:33:22 2019][1685976.143142] Lustre: 129247:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549463778/real 1549463778] req@ffffa07ac62a7200 x1623667908935136/t0(0) o104->fir-MDT0003@10.10.114.11@tcp:15/16 lens 296/224 e 0 to 1 dl 1549463789 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 06:33:22 2019][1685976.171018] Lustre: 129247:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 45092240 previous similar messages [Wed Feb 6 06:34:22 2019][1686036.435629] LustreError: 17997:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.19.4@tcp) failed to reply to blocking AST (req@ffffa06ed5658300 x1623667908990080 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffffa075893c9680/0x8c1de5394273c39c lrc: 4/0,0 mode: PR/PR res: [0x240000ceb:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a778b7 expref: 22 pid: 126630 timeout: 1686171 lvb_type: 0 [Wed Feb 6 06:34:22 2019][1686036.478048] LustreError: 138-a: fir-MDT0001: A client on nid 10.10.19.4@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 06:34:22 2019][1686036.490582] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.19.4@tcp ns: mdt-fir-MDT0001_UUID lock: ffffa075893c9680/0x8c1de5394273c39c lrc: 3/0,0 mode: PR/PR res: [0x240000ceb:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a778b7 expref: 23 pid: 126630 timeout: 0 lvb_type: 0 [Wed Feb 6 06:34:22 2019][1686036.527739] LustreError: 17997:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa0665194a000 ns: mdt-fir-MDT0001_UUID lock: ffffa077020c5580/0x8c1de5394273d34e lrc: 1/0,0 mode: --/PW res: [0x240000ceb:0x4:0x0].0x0 bits 0x40/0x0 rrc: 3 type: IBT flags: 0x54a01000000000 nid: 10.10.19.4@tcp remote: 0x844b529590a77920 expref: 9 pid: 17997 timeout: 0 lvb_type: 0 [Wed Feb 6 06:34:22 2019][1686036.562018] LustreError: 17997:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 3 previous similar messages [Wed Feb 6 06:35:54 2019][1686128.436850] LustreError: 17977:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa086d388e400 ns: mdt-fir-MDT0003_UUID lock: ffffa089b93d5e80/0x8c1de53942859bbd lrc: 1/0,0 mode: EX/EX res: [0x280000e0c:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.114.11@tcp remote: 0xdbb126caab60e958 expref: 4 pid: 17977 timeout: 0 lvb_type: 3 [Wed Feb 6 06:35:54 2019][1686128.503571] Lustre: fir-MDT0003: Connection restored to df95ea78-d1f5-1941-92aa-83203459f24a (at 10.10.114.11@tcp) [Wed Feb 6 06:35:54 2019][1686128.514093] Lustre: Skipped 7 previous similar messages [Wed Feb 6 06:36:50 2019][1686184.579814] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 06:36:50 2019][1686184.595971] Lustre: Skipped 1 previous similar message [Wed Feb 6 06:36:50 2019][1686184.601303] LustreError: 2937:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549463686, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa059c7f498c0/0x8c1de539427217ed lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 85 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a59394ae expref: -99 pid: 2937 timeout: 0 lvb_type: 0 [Wed Feb 6 06:36:50 2019][1686184.641401] LustreError: 2937:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 33 previous similar messages [Wed Feb 6 06:38:25 2019][1686279.437593] LustreError: 129254:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.115.2@tcp) failed to reply to blocking AST (req@ffffa0629ef29200 x1623667909253744 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa067d08172c0/0x8c1de5394289bfb8 lrc: 4/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 47 type: IBT flags: 0x60200400000020 nid: 10.10.115.2@tcp remote: 0xb8e1c27d85cfec01 expref: 14 pid: 129253 timeout: 1686414 lvb_type: 0 [Wed Feb 6 06:38:25 2019][1686279.480451] LustreError: 129254:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message [Wed Feb 6 06:38:25 2019][1686279.490722] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.115.2@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 06:38:25 2019][1686279.503342] LustreError: Skipped 1 previous similar message [Wed Feb 6 06:38:25 2019][1686279.509115] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.115.2@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa067d08172c0/0x8c1de5394289bfb8 lrc: 3/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 45 type: IBT flags: 0x60200400000020 nid: 10.10.115.2@tcp remote: 0xb8e1c27d85cfec01 expref: 15 pid: 129253 timeout: 0 lvb_type: 0 [Wed Feb 6 06:38:25 2019][1686279.546502] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [Wed Feb 6 06:38:25 2019][1686279.556765] LustreError: 17977:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa086cf97cc00 ns: mdt-fir-MDT0003_UUID lock: ffffa089b93d0000/0x8c1de5394289c71a lrc: 1/0,0 mode: EX/EX res: [0x280000e11:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.115.2@tcp remote: 0xb8e1c27d85cfec39 expref: 4 pid: 17977 timeout: 0 lvb_type: 3 [Wed Feb 6 06:39:15 2019][1686329.106107] Lustre: fir-MDT0003: haven't heard from client afad253f-2143-f49d-3f33-c08f8de9022c (at 10.8.8.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa06fe4523400, cur 1549464131 expire 1549463981 last 1549463904 [Wed Feb 6 06:39:15 2019][1686329.127897] Lustre: Skipped 13 previous similar messages [Wed Feb 6 06:40:31 2019][1686405.106751] Lustre: fir-MDT0003: haven't heard from client a62a683a-0e26-f38d-6315-c77b5607ae54 (at 10.9.107.2@o2ib4) in 163 seconds. I think it's dead, and I am evicting it. exp ffffa05a024b1800, cur 1549464207 expire 1549464057 last 1549464044 [Wed Feb 6 06:40:31 2019][1686405.128628] Lustre: Skipped 9 previous similar messages [Wed Feb 6 06:41:35 2019][1686469.114730] Lustre: fir-MDT0001: haven't heard from client a62a683a-0e26-f38d-6315-c77b5607ae54 (at 10.9.107.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa08b768b1800, cur 1549464271 expire 1549464121 last 1549464044 [Wed Feb 6 06:41:47 2019][1686481.106950] Lustre: fir-MDT0003: haven't heard from client 582c0bf9-0c65-cb8a-6846-32e5a6d43ca1 (at 10.9.107.43@o2ib4) in 212 seconds. I think it's dead, and I am evicting it. exp ffffa062e617cc00, cur 1549464283 expire 1549464133 last 1549464071 [Wed Feb 6 06:41:47 2019][1686481.128904] Lustre: Skipped 5 previous similar messages [Wed Feb 6 06:41:52 2019][1686486.578255] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 06:41:52 2019][1686486.594415] Lustre: Skipped 1 previous similar message [Wed Feb 6 06:41:52 2019][1686486.598262] LustreError: 17834:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549463988, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0001 lock: ffffa07ae8b7bf00/0x8c1de539428a8b44 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a676a0d3 expref: -99 pid: 17834 timeout: 0 lvb_type: 0 [Wed Feb 6 06:41:52 2019][1686486.598265] LustreError: 17834:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 31 previous similar messages [Wed Feb 6 06:42:51 2019][1686545.103463] Lustre: fir-MDT0001: haven't heard from client 1ee60be4-911f-02ce-275f-36587bcd7a56 (at 10.9.107.19@o2ib4) in 158 seconds. I think it's dead, and I am evicting it. exp ffffa07b400d7400, cur 1549464347 expire 1549464197 last 1549464189 [Wed Feb 6 06:42:51 2019][1686545.125423] Lustre: Skipped 4 previous similar messages [Wed Feb 6 06:43:22 2019][1686576.125967] LNetError: 129372:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.15.5@tcp from [Wed Feb 6 06:43:22 2019][1686576.136661] LNetError: 129372:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 138091968 previous similar messages [Wed Feb 6 06:43:22 2019][1686576.147969] Lustre: 2937:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549464378/real 1549464378] req@ffffa04fb1391200 x1623667909631200/t0(0) o104->fir-MDT0001@10.10.19.4@tcp:15/16 lens 296/224 e 0 to 1 dl 1549464389 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 06:43:22 2019][1686576.147972] Lustre: 2937:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 138475920 previous similar messages [Wed Feb 6 06:43:27 2019][1686581.440043] LustreError: 17995:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.15.5@tcp) failed to reply to blocking AST (req@ffffa061d4e6e900 x1623667909578096 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa073e7e772c0/0x8c1de539429a1964 lrc: 4/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 46 type: IBT flags: 0x60200400000020 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349c6d7 expref: 14 pid: 18678 timeout: 1686716 lvb_type: 0 [Wed Feb 6 06:43:27 2019][1686581.482551] LustreError: 17995:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages [Wed Feb 6 06:43:27 2019][1686581.492818] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.15.5@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 06:43:27 2019][1686581.505330] LustreError: Skipped 3 previous similar messages [Wed Feb 6 06:43:27 2019][1686581.511208] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.15.5@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa073e7e772c0/0x8c1de539429a1964 lrc: 3/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 46 type: IBT flags: 0x60200400000020 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349c6d7 expref: 15 pid: 18678 timeout: 0 lvb_type: 0 [Wed Feb 6 06:43:27 2019][1686581.548333] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages [Wed Feb 6 06:43:27 2019][1686581.558643] LustreError: 129372:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa067f2f7dc00 ns: mdt-fir-MDT0003_UUID lock: ffffa05c78fa6780/0x8c1de539429a1f61 lrc: 1/0,0 mode: EX/EX res: [0x280000e15:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349c70f expref: 4 pid: 129372 timeout: 0 lvb_type: 3 [Wed Feb 6 06:43:27 2019][1686581.592997] LustreError: 129372:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 4 previous similar messages [Wed Feb 6 06:43:51 2019][1686605.746217] LustreError: 126624:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549464107, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa059802fa400/0x8c1de5394297af65 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 67 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a6c4893f expref: -99 pid: 126624 timeout: 0 lvb_type: 0 [Wed Feb 6 06:43:51 2019][1686605.786638] LustreError: 126624:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 45 previous similar messages [Wed Feb 6 06:44:00 2019][1686614.108464] Lustre: fir-MDT0003: haven't heard from client 1ee60be4-911f-02ce-275f-36587bcd7a56 (at 10.9.107.19@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa052532a4000, cur 1549464416 expire 1549464266 last 1549464189 [Wed Feb 6 06:45:58 2019][1686732.470742] Lustre: fir-MDT0003: Connection restored to 7324bf3d-6a04-b1ff-c579-e86bbeb8be32 (at 10.10.16.2@tcp) [Wed Feb 6 06:45:58 2019][1686732.481100] Lustre: Skipped 13 previous similar messages [Wed Feb 6 06:47:32 2019][1686826.589995] Lustre: fir-MDT0000-osp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 06:47:32 2019][1686826.593011] LustreError: 17973:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549464328, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffffa06608f87980/0x8c1de539429ac3da lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 79 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a74d3a1c expref: -99 pid: 17973 timeout: 0 lvb_type: 0 [Wed Feb 6 06:47:32 2019][1686826.593014] LustreError: 17973:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 33 previous similar messages [Wed Feb 6 06:47:32 2019][1686826.657508] Lustre: Skipped 1 previous similar message [Wed Feb 6 06:47:44 2019][1686838.102026] Lustre: fir-MDT0003: haven't heard from client ee310e5c-55e4-0a0f-af99-ae0536891531 (at 10.8.27.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa062cb0b7000, cur 1549464640 expire 1549464490 last 1549464413 [Wed Feb 6 06:52:03 2019][1687097.112334] Lustre: fir-MDT0003: haven't heard from client e1335c4d-79ef-dfc5-87fe-76c36a260b43 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa074bb84b800, cur 1549464899 expire 1549464749 last 1549464672 [Wed Feb 6 06:52:03 2019][1687097.134123] Lustre: Skipped 1 previous similar message [Wed Feb 6 06:53:22 2019][1687176.130802] LNetError: 18335:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.15.6@tcp from [Wed Feb 6 06:53:22 2019][1687176.141407] LNetError: 18335:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 153231139 previous similar messages [Wed Feb 6 06:53:22 2019][1687176.152806] Lustre: 129280:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549464978/real 1549464978] req@ffffa0510a6cbc00 x1623667910357584/t0(0) o104->fir-MDT0001@10.10.19.4@tcp:15/16 lens 296/224 e 0 to 1 dl 1549464989 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 06:53:22 2019][1687176.180502] Lustre: 129280:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 154254205 previous similar messages [Wed Feb 6 06:53:31 2019][1687185.444896] LustreError: 17828:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.115.2@tcp) failed to reply to blocking AST (req@ffffa089c170dd00 x1623667910276896 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa05aab57a400/0x8c1de53942c062ec lrc: 4/0,0 mode: PR/PR res: [0x280000e11:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.115.2@tcp remote: 0xb8e1c27d85cfed74 expref: 10 pid: 129262 timeout: 1687320 lvb_type: 0 [Wed Feb 6 06:53:31 2019][1687185.444898] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.15.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 06:53:31 2019][1687185.444902] LustreError: Skipped 14 previous similar messages [Wed Feb 6 06:53:31 2019][1687185.444929] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.15.5@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa0522e055580/0x8c1de53942aeffcc lrc: 3/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 50 type: IBT flags: 0x60200400000020 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349c835 expref: 11 pid: 129267 timeout: 0 lvb_type: 0 [Wed Feb 6 06:53:31 2019][1687185.444932] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 12 previous similar messages [Wed Feb 6 06:53:31 2019][1687185.444989] LustreError: 17995:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa06b4bc91c00 ns: mdt-fir-MDT0003_UUID lock: ffffa065596acec0/0x8c1de53942c06420 lrc: 3/0,0 mode: PW/PW res: [0x280000e15:0x1:0x0].0x0 bits 0x40/0x0 rrc: 3 type: IBT flags: 0x50200000000000 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349c84a expref: 11 pid: 17995 timeout: 0 lvb_type: 0 [Wed Feb 6 06:53:31 2019][1687185.444991] LustreError: 17995:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 11 previous similar messages [Wed Feb 6 06:53:31 2019][1687185.597883] LustreError: 17828:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 19 previous similar messages [Wed Feb 6 06:56:02 2019][1687336.475771] Lustre: fir-MDT0003: Connection restored to 7324bf3d-6a04-b1ff-c579-e86bbeb8be32 (at 10.10.16.2@tcp) [Wed Feb 6 06:56:02 2019][1687336.486120] Lustre: Skipped 17 previous similar messages [Wed Feb 6 07:03:22 2019][1687776.135639] LNetError: 127198:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.11@tcp from [Wed Feb 6 07:03:22 2019][1687776.146502] LNetError: 127198:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 181707608 previous similar messages [Wed Feb 6 07:03:22 2019][1687776.157639] Lustre: 17823:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549465578/real 1549465578] req@ffffa0672a8a4e00 x1623667911390352/t0(0) o104->fir-MDT0003@10.10.15.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549465589 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 07:03:22 2019][1687776.157642] Lustre: 17823:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 183534013 previous similar messages [Wed Feb 6 07:03:35 2019][1687789.449763] LustreError: 129796:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.15.5@tcp) failed to reply to blocking AST (req@ffffa0716d639200 x1623667911390336 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa061c5980000/0x8c1de539432ef071 lrc: 4/0,0 mode: PR/PR res: [0x280000e15:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349c8a5 expref: 10 pid: 129796 timeout: 1687924 lvb_type: 0 [Wed Feb 6 07:03:35 2019][1687789.449765] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.16.2@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 07:03:35 2019][1687789.449770] LustreError: Skipped 23 previous similar messages [Wed Feb 6 07:03:35 2019][1687789.449793] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.15.6@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa04d1e71bf00/0x8c1de539432ef0e1 lrc: 3/0,0 mode: PR/PR res: [0x280000e16:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7caf8 expref: 11 pid: 129259 timeout: 0 lvb_type: 0 [Wed Feb 6 07:03:35 2019][1687789.449796] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 16 previous similar messages [Wed Feb 6 07:03:35 2019][1687789.449863] LustreError: 17823:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa05f1ee19400 ns: mdt-fir-MDT0003_UUID lock: ffffa071d4dcec00/0x8c1de539432ef112 lrc: 1/0,0 mode: --/PW res: [0x280000e16:0x2:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT flags: 0x54a01000000000 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7cb06 expref: 11 pid: 17823 timeout: 0 lvb_type: 0 [Wed Feb 6 07:03:35 2019][1687789.449867] LustreError: 17823:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 16 previous similar messages [Wed Feb 6 07:03:35 2019][1687789.602492] LustreError: 129796:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 23 previous similar messages [Wed Feb 6 07:06:06 2019][1687940.451184] LustreError: 129595:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa07b08178000 x1623667911849008/t0(0) o104->fir-MDT0003@10.10.115.2@tcp:15/16 lens 296/224 e 0 to 1 dl 1549465753 ref 1 fl Rpc:eX/2/ffffffff rc -5/-1 [Wed Feb 6 07:06:06 2019][1687940.451577] Lustre: fir-MDT0003: Connection restored to df95ea78-d1f5-1941-92aa-83203459f24a (at 10.10.114.11@tcp) [Wed Feb 6 07:06:06 2019][1687940.451579] Lustre: Skipped 21 previous similar messages [Wed Feb 6 07:13:22 2019][1688376.140480] LNetError: 129249:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.19.4@tcp from [Wed Feb 6 07:13:22 2019][1688376.151170] LNetError: 129249:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 172082321 previous similar messages [Wed Feb 6 07:13:22 2019][1688376.162486] Lustre: 18335:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549466178/real 1549466178] req@ffffa07166e5c500 x1623667912733008/t0(0) o104->fir-MDT0003@10.10.15.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549466189 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 07:13:22 2019][1688376.162489] Lustre: 18335:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 173456609 previous similar messages [Wed Feb 6 07:13:39 2019][1688393.454634] LustreError: 17908:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.15.5@tcp) failed to reply to blocking AST (req@ffffa07948808000 x1623667912733344 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa0698e80af40/0x8c1de539436dface lrc: 4/0,0 mode: PR/PR res: [0x280000cc3:0x8d:0x0].0x0 bits 0x13/0x0 rrc: 57 type: IBT flags: 0x60200400000020 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349c907 expref: 10 pid: 17978 timeout: 1688528 lvb_type: 0 [Wed Feb 6 07:13:39 2019][1688393.454638] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.15.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 07:13:39 2019][1688393.454640] LustreError: Skipped 22 previous similar messages [Wed Feb 6 07:13:39 2019][1688393.454658] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.15.6@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa05a3a96af40/0x8c1de539438bfd16 lrc: 3/0,0 mode: PR/PR res: [0x280000e16:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7cb61 expref: 11 pid: 17904 timeout: 0 lvb_type: 0 [Wed Feb 6 07:13:39 2019][1688393.454661] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 17 previous similar messages [Wed Feb 6 07:13:39 2019][1688393.454739] LustreError: 18335:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa060cb50b000 ns: mdt-fir-MDT0003_UUID lock: ffffa0768928ec00/0x8c1de539438bfe20 lrc: 3/0,0 mode: --/PW res: [0x280000e16:0x2:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT flags: 0x54a01000000000 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7cb6f expref: 6 pid: 18335 timeout: 0 lvb_type: 0 [Wed Feb 6 07:13:39 2019][1688393.454741] LustreError: 18335:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 17 previous similar messages [Wed Feb 6 07:13:39 2019][1688393.607206] LustreError: 17908:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 19 previous similar messages [Wed Feb 6 07:16:10 2019][1688544.485021] Lustre: fir-MDT0003: Connection restored to 7324bf3d-6a04-b1ff-c579-e86bbeb8be32 (at 10.10.16.2@tcp) [Wed Feb 6 07:16:10 2019][1688544.495365] Lustre: Skipped 40 previous similar messages [Wed Feb 6 07:18:41 2019][1688695.457113] LustreError: 17973:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa0665ba67500 x1623667913505808/t0(0) o104->fir-MDT0003@10.10.16.2@tcp:15/16 lens 296/224 e 0 to 1 dl 1549466508 ref 1 fl Rpc:eX/2/ffffffff rc -5/-1 [Wed Feb 6 07:23:22 2019][1688976.145321] LNetError: 17930:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.19.4@tcp from [Wed Feb 6 07:23:22 2019][1688976.155934] LNetError: 17930:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 96685591 previous similar messages [Wed Feb 6 07:23:22 2019][1688976.167316] Lustre: 17930:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549466778/real 1549466778] req@ffffa07166e59e00 x1623667914340768/t0(0) o104->fir-MDT0001@10.10.19.4@tcp:15/16 lens 296/224 e 0 to 1 dl 1549466789 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 07:23:22 2019][1688976.194951] Lustre: 17930:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 96862629 previous similar messages [Wed Feb 6 07:25:23 2019][1689097.460308] LustreError: 17930:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.19.4@tcp) failed to reply to blocking AST (req@ffffa07166e59e00 x1623667914340768 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffffa052fcb52400/0x8c1de5394407f2c4 lrc: 4/0,0 mode: PR/PR res: [0x240000cf3:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a7ba57 expref: 16 pid: 17919 timeout: 1689232 lvb_type: 0 [Wed Feb 6 07:25:23 2019][1689097.502649] LustreError: 17930:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 8 previous similar messages [Wed Feb 6 07:25:23 2019][1689097.512921] LustreError: 138-a: fir-MDT0001: A client on nid 10.10.19.4@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 07:25:23 2019][1689097.525454] LustreError: Skipped 9 previous similar messages [Wed Feb 6 07:25:23 2019][1689097.531322] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.19.4@tcp ns: mdt-fir-MDT0001_UUID lock: ffffa052fcb52400/0x8c1de5394407f2c4 lrc: 3/0,0 mode: PR/PR res: [0x240000cf3:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a7ba57 expref: 17 pid: 17919 timeout: 0 lvb_type: 0 [Wed Feb 6 07:25:23 2019][1689097.568313] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 7 previous similar messages [Wed Feb 6 07:25:23 2019][1689097.578471] LustreError: 17930:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa074b07d0800 ns: mdt-fir-MDT0001_UUID lock: ffffa0770e152400/0x8c1de5394407f2e0 lrc: 3/0,0 mode: PW/PW res: [0x240000cf3:0x4:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x50200000000000 nid: 10.10.19.4@tcp remote: 0x844b529590a7ba65 expref: 3 pid: 17930 timeout: 0 lvb_type: 0 [Wed Feb 6 07:25:23 2019][1689097.612763] LustreError: 17930:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 7 previous similar messages [Wed Feb 6 07:27:54 2019][1689248.462024] Lustre: fir-MDT0001: Connection restored to 04a6727c-f0a5-409b-d667-2e88045cb817 (at 10.10.19.4@tcp) [Wed Feb 6 07:27:54 2019][1689248.472366] Lustre: Skipped 11 previous similar messages [Wed Feb 6 07:33:22 2019][1689576.150157] LNetError: 129250:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.19.4@tcp from [Wed Feb 6 07:33:22 2019][1689576.160854] LNetError: 129250:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 112904562 previous similar messages [Wed Feb 6 07:33:22 2019][1689576.172240] Lustre: 128084:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549467378/real 1549467378] req@ffffa05696a52d00 x1623667916010992/t0(0) o104->fir-MDT0001@10.10.113.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549467389 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 07:33:22 2019][1689576.172243] Lustre: 128084:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 112999037 previous similar messages [Wed Feb 6 07:35:27 2019][1689701.465179] LustreError: 129250:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.19.4@tcp) failed to reply to blocking AST (req@ffffa07031257b00 x1623667915995024 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffffa054ea771440/0x8c1de53944861ed1 lrc: 4/0,0 mode: PR/PR res: [0x240000cf3:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a7bbd1 expref: 16 pid: 17877 timeout: 1689836 lvb_type: 0 [Wed Feb 6 07:35:27 2019][1689701.507590] LustreError: 129250:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 6 previous similar messages [Wed Feb 6 07:35:27 2019][1689701.517937] LustreError: 138-a: fir-MDT0001: A client on nid 10.10.19.4@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 07:35:27 2019][1689701.530448] LustreError: Skipped 6 previous similar messages [Wed Feb 6 07:35:27 2019][1689701.536310] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.19.4@tcp ns: mdt-fir-MDT0001_UUID lock: ffffa054ea771440/0x8c1de53944861ed1 lrc: 3/0,0 mode: PR/PR res: [0x240000cf3:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.19.4@tcp remote: 0x844b529590a7bbd1 expref: 17 pid: 17877 timeout: 0 lvb_type: 0 [Wed Feb 6 07:35:27 2019][1689701.573270] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 6 previous similar messages [Wed Feb 6 07:35:27 2019][1689701.583356] LustreError: 129250:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa0652166d800 ns: mdt-fir-MDT0001_UUID lock: ffffa071f04d69c0/0x8c1de53944861f02 lrc: 5/0,0 mode: PW/PW res: [0x240000cf3:0x4:0x0].0x0 bits 0x40/0x0 rrc: 3 type: IBT flags: 0x50200000000000 nid: 10.10.19.4@tcp remote: 0x844b529590a7bbdf expref: 4 pid: 129250 timeout: 0 lvb_type: 0 [Wed Feb 6 07:35:27 2019][1689701.617804] LustreError: 129250:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 6 previous similar messages [Wed Feb 6 07:37:58 2019][1689852.467065] Lustre: fir-MDT0001: Connection restored to 04a6727c-f0a5-409b-d667-2e88045cb817 (at 10.10.19.4@tcp) [Wed Feb 6 07:37:58 2019][1689852.477415] Lustre: Skipped 11 previous similar messages [Wed Feb 6 07:40:55 2019][1690029.566079] Lustre: fir-MDT0003: Received new LWP connection from 10.0.10.51@o2ib7, removing former export from same NID [Wed Feb 6 07:40:55 2019][1690029.577113] Lustre: Skipped 1 previous similar message [Wed Feb 6 07:41:08 2019][1690042.717507] Lustre: fir-MDT0001: Received new LWP connection from 10.0.10.51@o2ib7, removing former export from same NID [Wed Feb 6 07:41:28 2019][1690062.750089] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [Wed Feb 6 07:41:28 2019][1690062.760346] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.51@o2ib7 (5): c: 0, oc: 1, rc: 8 [Wed Feb 6 07:41:28 2019][1690062.773025] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 07:41:30 2019][1690064.750101] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 1 seconds [Wed Feb 6 07:41:31 2019][1690065.750136] Lustre: fir-MDT0000-lwp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 07:41:31 2019][1690065.766294] Lustre: Skipped 1 previous similar message [Wed Feb 6 07:41:35 2019][1690069.750137] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 1 seconds [Wed Feb 6 07:41:35 2019][1690069.760481] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages [Wed Feb 6 07:41:35 2019][1690069.770076] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [Wed Feb 6 07:41:37 2019][1690071.622454] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.0.10.106@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 07:41:37 2019][1690071.639990] LustreError: Skipped 10 previous similar messages [Wed Feb 6 07:41:38 2019][1690072.716299] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.8.12.26@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 07:41:41 2019][1690075.750189] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [Wed Feb 6 07:41:41 2019][1690075.760534] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 1 previous similar message [Wed Feb 6 07:41:43 2019][1690077.305122] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.105@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 07:41:43 2019][1690077.322672] LustreError: Skipped 2 previous similar messages [Wed Feb 6 07:41:50 2019][1690084.210987] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.8.15.3@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 07:41:54 2019][1690088.481717] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.9.104.44@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 07:41:54 2019][1690088.499266] LustreError: Skipped 17 previous similar messages [Wed Feb 6 07:42:01 2019][1690095.219357] Lustre: fir-MDT0000-osp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 07:42:01 2019][1690095.750357] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 2 seconds [Wed Feb 6 07:42:02 2019][1690097.037786] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.9.105.6@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 07:42:02 2019][1690097.055245] LustreError: Skipped 74 previous similar messages [Wed Feb 6 07:42:18 2019][1690113.094267] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.9.101.13@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 07:42:19 2019][1690113.111810] LustreError: Skipped 55 previous similar messages [Wed Feb 6 07:42:33 2019][1690127.750610] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [Wed Feb 6 07:42:33 2019][1690127.760957] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 5 previous similar messages [Wed Feb 6 07:42:51 2019][1690145.436934] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.9.101.22@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 07:42:51 2019][1690145.454477] LustreError: Skipped 118 previous similar messages [Wed Feb 6 07:43:23 2019][1690177.751016] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [Wed Feb 6 07:43:23 2019][1690177.761361] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 6 previous similar messages [Wed Feb 6 07:43:55 2019][1690209.504495] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.8.16.8@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 07:43:55 2019][1690209.521866] LustreError: Skipped 345 previous similar messages [Wed Feb 6 07:44:01 2019][1690215.524318] LNet: Service thread pid 18265 was inactive for 200.67s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 07:44:01 2019][1690215.541430] Pid: 18265, comm: mdt02_049 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 07:44:01 2019][1690215.551342] Call Trace: [Wed Feb 6 07:44:01 2019][1690215.553983] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 07:44:01 2019][1690215.561106] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 07:44:01 2019][1690215.568403] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 07:44:01 2019][1690215.575249] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 07:44:01 2019][1690215.582008] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 07:44:01 2019][1690215.588407] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 07:44:01 2019][1690215.594729] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 07:44:01 2019][1690215.602183] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 07:44:01 2019][1690215.609109] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 07:44:01 2019][1690215.617326] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 07:44:01 2019][1690215.623738] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 07:44:01 2019][1690215.629964] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 07:44:01 2019][1690215.636721] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 07:44:01 2019][1690215.642599] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 07:44:01 2019][1690215.649739] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:44:01 2019][1690215.657628] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:44:01 2019][1690215.664142] [] kthread+0xd1/0xe0 [Wed Feb 6 07:44:01 2019][1690215.669233] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:44:01 2019][1690215.675901] [] 0xffffffffffffffff [Wed Feb 6 07:44:01 2019][1690215.681099] LustreError: dumping log to /tmp/lustre-log.1549468017.18265 [Wed Feb 6 07:44:03 2019][1690217.572332] LNet: Service thread pid 18269 was inactive for 200.44s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 07:44:03 2019][1690217.589441] Pid: 18269, comm: mdt02_052 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 07:44:03 2019][1690217.599354] Call Trace: [Wed Feb 6 07:44:03 2019][1690217.601991] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 07:44:03 2019][1690217.609120] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 07:44:03 2019][1690217.616422] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 07:44:03 2019][1690217.623272] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 07:44:03 2019][1690217.630043] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 07:44:03 2019][1690217.636437] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 07:44:03 2019][1690217.642760] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 07:44:03 2019][1690217.650200] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 07:44:03 2019][1690217.657131] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 07:44:03 2019][1690217.665349] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 07:44:03 2019][1690217.671746] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 07:44:03 2019][1690217.677982] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 07:44:03 2019][1690217.684728] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 07:44:03 2019][1690217.690620] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 07:44:03 2019][1690217.697766] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:44:03 2019][1690217.705667] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:44:03 2019][1690217.712170] [] kthread+0xd1/0xe0 [Wed Feb 6 07:44:03 2019][1690217.717253] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:44:03 2019][1690217.723908] [] 0xffffffffffffffff [Wed Feb 6 07:44:03 2019][1690217.729104] LustreError: dumping log to /tmp/lustre-log.1549468019.18269 [Wed Feb 6 07:44:05 2019][1690219.620353] LNet: Service thread pid 17853 was inactive for 200.18s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 07:44:05 2019][1690219.637467] Pid: 17853, comm: mdt03_026 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 07:44:05 2019][1690219.647378] Call Trace: [Wed Feb 6 07:44:05 2019][1690219.650024] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 07:44:05 2019][1690219.657153] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 07:44:05 2019][1690219.664449] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 07:44:05 2019][1690219.671298] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 07:44:05 2019][1690219.678064] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 07:44:05 2019][1690219.684462] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 07:44:05 2019][1690219.690785] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 07:44:05 2019][1690219.698223] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 07:44:05 2019][1690219.705156] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 07:44:05 2019][1690219.713374] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 07:44:05 2019][1690219.719784] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 07:44:05 2019][1690219.726024] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 07:44:05 2019][1690219.732785] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 07:44:05 2019][1690219.738662] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 07:44:05 2019][1690219.745804] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:44:05 2019][1690219.753696] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:44:05 2019][1690219.760215] [] kthread+0xd1/0xe0 [Wed Feb 6 07:44:05 2019][1690219.765305] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:44:05 2019][1690219.771968] [] 0xffffffffffffffff [Wed Feb 6 07:44:05 2019][1690219.777172] LustreError: dumping log to /tmp/lustre-log.1549468021.17853 [Wed Feb 6 07:44:07 2019][1690221.668373] LNet: Service thread pid 127216 was inactive for 200.50s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 07:44:07 2019][1690221.685568] Pid: 127216, comm: mdt02_007 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 07:44:07 2019][1690221.695562] Call Trace: [Wed Feb 6 07:44:07 2019][1690221.698201] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 07:44:07 2019][1690221.705311] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 07:44:07 2019][1690221.712598] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 07:44:07 2019][1690221.719440] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 07:44:07 2019][1690221.726188] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 07:44:07 2019][1690221.732588] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 07:44:07 2019][1690221.738903] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 07:44:07 2019][1690221.746339] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 07:44:07 2019][1690221.753263] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 07:44:07 2019][1690221.761472] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 07:44:07 2019][1690221.767874] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 07:44:07 2019][1690221.774101] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 07:44:07 2019][1690221.780865] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 07:44:07 2019][1690221.786737] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 07:44:07 2019][1690221.793880] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:44:07 2019][1690221.801765] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:44:07 2019][1690221.808278] [] kthread+0xd1/0xe0 [Wed Feb 6 07:44:07 2019][1690221.813372] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:44:07 2019][1690221.820031] [] 0xffffffffffffffff [Wed Feb 6 07:44:07 2019][1690221.825227] LustreError: dumping log to /tmp/lustre-log.1549468023.127216 [Wed Feb 6 07:44:09 2019][1690223.204380] Pid: 126632, comm: mdt03_000 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 07:44:09 2019][1690223.214375] Call Trace: [Wed Feb 6 07:44:09 2019][1690223.217006] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 07:44:09 2019][1690223.224105] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 07:44:09 2019][1690223.231405] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 07:44:09 2019][1690223.238240] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 07:44:09 2019][1690223.244990] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 07:44:09 2019][1690223.251380] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 07:44:09 2019][1690223.257695] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 07:44:09 2019][1690223.265126] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 07:44:09 2019][1690223.272048] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 07:44:09 2019][1690223.280259] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 07:44:09 2019][1690223.286661] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 07:44:09 2019][1690223.292877] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 07:44:09 2019][1690223.299638] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 07:44:09 2019][1690223.305504] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 07:44:09 2019][1690223.312624] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:44:09 2019][1690223.320513] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:44:09 2019][1690223.327031] [] kthread+0xd1/0xe0 [Wed Feb 6 07:44:09 2019][1690223.332111] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:44:09 2019][1690223.338764] [] 0xffffffffffffffff [Wed Feb 6 07:44:09 2019][1690223.343943] LustreError: dumping log to /tmp/lustre-log.1549468025.126632 [Wed Feb 6 07:44:09 2019][1690223.462424] LNet: Service thread pid 18265 completed after 208.60s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 07:44:09 2019][1690223.478752] LNet: Skipped 26 previous similar messages [Wed Feb 6 07:44:15 2019][1690229.264851] Lustre: fir-MDT0003: Received new LWP connection from 10.0.10.51@o2ib7, removing former export from same NID [Wed Feb 6 07:44:15 2019][1690229.275901] Lustre: Skipped 3 previous similar messages [Wed Feb 6 07:45:38 2019][1690312.509297] LNetError: 17972:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.6@tcp from [Wed Feb 6 07:45:38 2019][1690312.519999] LNetError: 17972:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 88664004 previous similar messages [Wed Feb 6 07:45:38 2019][1690312.531238] Lustre: 17972:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549468114/real 1549468114] req@ffffa0624fd0c800 x1623667917874048/t0(0) o104->fir-MDT0001@10.10.113.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549468125 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 [Wed Feb 6 07:45:38 2019][1690312.558924] Lustre: 17972:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 88799768 previous similar messages [Wed Feb 6 07:48:09 2019][1690463.471323] LustreError: 17972:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.6@tcp) failed to reply to blocking AST (req@ffffa0624fd0c800 x1623667917874048 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffffa05a69553180/0x8c1de53944f715bb lrc: 4/0,0 mode: PR/PR res: [0x240000cf7:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 12 type: IBT flags: 0x60200400000020 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7f8a1 expref: 15 pid: 126625 timeout: 1690598 lvb_type: 0 [Wed Feb 6 07:48:09 2019][1690463.514023] LustreError: 17972:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 5 previous similar messages [Wed Feb 6 07:48:09 2019][1690463.524296] LustreError: 138-a: fir-MDT0001: A client on nid 10.10.113.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 07:48:09 2019][1690463.536915] LustreError: Skipped 5 previous similar messages [Wed Feb 6 07:48:09 2019][1690463.542775] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.6@tcp ns: mdt-fir-MDT0001_UUID lock: ffffa05a69553180/0x8c1de53944f715bb lrc: 3/0,0 mode: PR/PR res: [0x240000cf7:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 12 type: IBT flags: 0x60200400000020 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7f8a1 expref: 16 pid: 126625 timeout: 0 lvb_type: 0 [Wed Feb 6 07:48:09 2019][1690463.580097] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages [Wed Feb 6 07:48:09 2019][1690463.590182] LustreError: 17972:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa08a15bb2400 ns: mdt-fir-MDT0001_UUID lock: ffffa0639dffb840/0x8c1de53944f715d7 lrc: 1/0,0 mode: --/PW res: [0x240000cf7:0x1:0x0].0x0 bits 0x40/0x0 rrc: 9 type: IBT flags: 0x54a01000000000 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7f8af expref: 4 pid: 17972 timeout: 0 lvb_type: 0 [Wed Feb 6 07:48:09 2019][1690463.624554] LustreError: 17972:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 5 previous similar messages [Wed Feb 6 07:48:09 2019][1690463.635113] Lustre: fir-MDT0001: Connection restored to db7578a7-294b-6d6d-61e8-bb06998c6482 (at 10.10.113.6@tcp) [Wed Feb 6 07:48:09 2019][1690463.645556] Lustre: Skipped 23 previous similar messages [Wed Feb 6 07:55:38 2019][1690912.514927] LNetError: 129595:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.6@tcp from [Wed Feb 6 07:55:38 2019][1690912.525703] LNetError: 129595:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 80462681 previous similar messages [Wed Feb 6 07:55:38 2019][1690912.537009] Lustre: 129595:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549468714/real 1549468714] req@ffffa076f9c6e300 x1623667919018624/t0(0) o104->fir-MDT0001@10.10.113.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549468725 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 07:55:38 2019][1690912.564815] Lustre: 129595:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 80462681 previous similar messages [Wed Feb 6 07:58:20 2019][1691074.476243] LustreError: 129595:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.6@tcp) failed to reply to blocking AST (req@ffffa076f9c6e300 x1623667919397648 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffffa059c123e540/0x8c1de539456c1652 lrc: 4/0,0 mode: PR/PR res: [0x240000cf7:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 12 type: IBT flags: 0x60200400000020 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7fdb7 expref: 15 pid: 127970 timeout: 1691209 lvb_type: 0 [Wed Feb 6 07:58:20 2019][1691074.519007] LustreError: 129595:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages [Wed Feb 6 07:58:20 2019][1691074.529361] LustreError: 138-a: fir-MDT0001: A client on nid 10.10.113.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 07:58:20 2019][1691074.541994] LustreError: Skipped 3 previous similar messages [Wed Feb 6 07:58:20 2019][1691074.547856] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.6@tcp ns: mdt-fir-MDT0001_UUID lock: ffffa059c123e540/0x8c1de539456c1652 lrc: 3/0,0 mode: PR/PR res: [0x240000cf7:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 12 type: IBT flags: 0x60200400000020 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7fdb7 expref: 16 pid: 127970 timeout: 0 lvb_type: 0 [Wed Feb 6 07:58:20 2019][1691074.585208] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages [Wed Feb 6 07:58:20 2019][1691074.595325] LustreError: 129595:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa06ee1762c00 ns: mdt-fir-MDT0001_UUID lock: ffffa07600aa72c0/0x8c1de539456c1675 lrc: 3/0,0 mode: PW/PW res: [0x240000cf7:0x1:0x0].0x0 bits 0x40/0x0 rrc: 9 type: IBT flags: 0x50200000000000 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7fdc5 expref: 3 pid: 129595 timeout: 0 lvb_type: 0 [Wed Feb 6 07:58:20 2019][1691074.629871] LustreError: 129595:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 3 previous similar messages [Wed Feb 6 07:58:20 2019][1691074.640631] Lustre: fir-MDT0001: Connection restored to db7578a7-294b-6d6d-61e8-bb06998c6482 (at 10.10.113.6@tcp) [Wed Feb 6 07:58:20 2019][1691074.651062] Lustre: Skipped 3 previous similar messages [Wed Feb 6 08:05:38 2019][1691512.519770] LNetError: 127273:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.15.6@tcp from [Wed Feb 6 08:05:38 2019][1691512.530469] LNetError: 127273:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 91059082 previous similar messages [Wed Feb 6 08:05:38 2019][1691512.541773] Lustre: 126628:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549469314/real 1549469314] req@ffffa06811745400 x1623667921472624/t0(0) o104->fir-MDT0003@10.10.15.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549469325 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 08:05:38 2019][1691512.541778] Lustre: 126628:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 91327857 previous similar messages [Wed Feb 6 08:08:20 2019][1691674.645249] Lustre: fir-MDT0001: Connection restored to 6027016b-6cf9-c910-aa2a-2d5d5eab4abd (at 10.8.4.23@o2ib6) [Wed Feb 6 08:08:20 2019][1691674.655683] Lustre: Skipped 236 previous similar messages [Wed Feb 6 08:09:03 2019][1691717.481427] LustreError: 17903:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.112.8@tcp) failed to reply to blocking AST (req@ffffa0885fa37b00 x1623667922303648 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa0772f7c0fc0/0x8c1de53945feb715 lrc: 4/0,0 mode: PR/PR res: [0x280000e38:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.112.8@tcp remote: 0x8ea3e4d271873be2 expref: 19 pid: 18269 timeout: 1691852 lvb_type: 0 [Wed Feb 6 08:09:03 2019][1691717.523926] LustreError: 17903:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 5 previous similar messages [Wed Feb 6 08:09:03 2019][1691717.534191] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.112.8@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 08:09:03 2019][1691717.546789] LustreError: Skipped 5 previous similar messages [Wed Feb 6 08:09:03 2019][1691717.552650] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.8@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa0772f7c0fc0/0x8c1de53945feb715 lrc: 3/0,0 mode: PR/PR res: [0x280000e38:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.112.8@tcp remote: 0x8ea3e4d271873be2 expref: 20 pid: 18269 timeout: 0 lvb_type: 0 [Wed Feb 6 08:09:03 2019][1691717.589775] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages [Wed Feb 6 08:09:03 2019][1691717.599891] LustreError: 17903:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa05ae8c64c00 ns: mdt-fir-MDT0003_UUID lock: ffffa089d4f6d340/0x8c1de53945feb7fc lrc: 1/0,0 mode: --/PW res: [0x280000e38:0x2:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x54a01000000000 nid: 10.10.112.8@tcp remote: 0x8ea3e4d271873bf0 expref: 4 pid: 17903 timeout: 0 lvb_type: 0 [Wed Feb 6 08:09:03 2019][1691717.634245] LustreError: 17903:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 5 previous similar messages [Wed Feb 6 08:10:32 2019][1691806.641733] Lustre: fir-MDT0003: Received new LWP connection from 10.0.10.51@o2ib7, removing former export from same NID [Wed Feb 6 08:10:43 2019][1691817.833239] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [Wed Feb 6 08:11:21 2019][1691855.696357] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.9.104.49@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 08:11:21 2019][1691855.713894] LustreError: Skipped 12 previous similar messages [Wed Feb 6 08:11:39 2019][1691873.478315] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.9.101.56@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 08:11:39 2019][1691873.495858] LustreError: Skipped 1 previous similar message [Wed Feb 6 08:12:11 2019][1691905.555286] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.9.105.8@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 08:12:11 2019][1691905.572738] LustreError: Skipped 24 previous similar messages [Wed Feb 6 08:12:46 2019][1691940.765225] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [Wed Feb 6 08:12:46 2019][1691940.775570] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages [Wed Feb 6 08:12:46 2019][1691940.785175] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 08:12:46 2019][1691940.801336] Lustre: Skipped 2 previous similar messages [Wed Feb 6 08:12:51 2019][1691945.765287] Lustre: fir-MDT0000-lwp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 08:12:52 2019][1691946.765307] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [Wed Feb 6 08:12:57 2019][1691951.765312] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [Wed Feb 6 08:12:57 2019][1691951.775655] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 4 previous similar messages [Wed Feb 6 08:13:15 2019][1691970.001615] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.8.7.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 08:13:15 2019][1691970.018982] LustreError: Skipped 75 previous similar messages [Wed Feb 6 08:13:23 2019][1691977.450536] Lustre: fir-MDT0002-osp-MDT0003: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 08:13:23 2019][1691977.466684] Lustre: Skipped 1 previous similar message [Wed Feb 6 08:13:36 2019][1691991.084191] Lustre: fir-MDT0003: Received new LWP connection from 10.0.10.51@o2ib7, removing former export from same NID [Wed Feb 6 08:13:36 2019][1691991.095245] Lustre: Skipped 3 previous similar messages [Wed Feb 6 08:13:37 2019][1691991.418648] Lustre: fir-MDT0002-osp-MDT0001: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 08:13:37 2019][1691991.434802] Lustre: Skipped 1 previous similar message [Wed Feb 6 08:14:41 2019][1692055.484570] LustreError: 17850:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa089f837e600 x1623667924277936/t0(0) o104->fir-MDT0003@10.10.115.2@tcp:15/16 lens 296/224 e 0 to 1 dl 1549469868 ref 1 fl Rpc:eX/2/ffffffff rc -5/-1 [Wed Feb 6 08:15:38 2019][1692112.524602] LNetError: 129242:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.15.6@tcp from [Wed Feb 6 08:15:38 2019][1692112.535304] LNetError: 129242:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 157611877 previous similar messages [Wed Feb 6 08:15:38 2019][1692112.546690] Lustre: 17860:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549469914/real 1549469914] req@ffffa055a625c500 x1623667925024992/t0(0) o104->fir-MDT0003@10.10.112.8@tcp:15/16 lens 296/224 e 0 to 1 dl 1549469925 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 08:15:38 2019][1692112.546694] Lustre: 17860:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 158582628 previous similar messages [Wed Feb 6 08:18:44 2019][1692298.598632] Lustre: fir-MDT0001: Connection restored to aa3b9181-076e-9d38-c41a-e93a0915658f (at 10.8.22.16@o2ib6) [Wed Feb 6 08:18:44 2019][1692298.609157] Lustre: Skipped 171 previous similar messages [Wed Feb 6 08:19:07 2019][1692321.486305] Lus0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.112.8@tcp) failed to reply to blocking AST (req@ffffa055a625c500 x1623667925669984 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa086cfce4140/0x8c1de5394667629e lrc: 4/0,0 mode: PR/PR res: [0x280000e34:0x3:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.112.8@tcp remote: 0x8ea3e4d271873cfa expref: 12 pid: 127195 timeout: 1692456 lvb_type: 0 [Wed Feb 6 08:19:07 2019][1692321.528901] LustreError: 17860:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 14 previous similar messages [Wed Feb 6 08:19:07 2019][1692321.539253] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.112.8@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 08:19:07 2019][1692321.551859] LustreError: Skipped 14 previous similar messages [Wed Feb 6 08:19:07 2019][1692321.557815] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.8@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa086cfce4140/0x8c1de5394667629e lrc: 3/0,0 mode: PR/PR res: [0x280000e34:0x3:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.112.8@tcp remote: 0x8ea3e4d271873cfa expref: 13 pid: 127195 timeout: 0 lvb_type: 0 [Wed Feb 6 08:19:07 2019][1692321.595038] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 12 previous similar messages [Wed Feb 6 08:19:07 2019][1692321.605207] LustreError: 17860:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa072b8a43000 ns: mdt-fir-MDT0003_UUID lock: ffffa0766f2d06c0/0x8c1de539466762ba lrc: 3/0,0 mode: PW/PW res: [0x280000e34:0x3:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x50200000000000 nid: 10.10.112.8@tcp remote: 0x8ea3e4d271873d08 expref: 4 pid: 17860 timeout: 0 lvb_type: 0 [Wed Feb 6 08:19:07 2019][1692321.639575] LustreError: 17860:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 10 previous similar messages [Wed Feb 6 08:19:43 2019][1692357.486664] LustreError: 127196:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa078fe988300 x1623667925840720/t0(0) o104->fir-MDT0003@10.10.115.2@tcp:15/16 lens 296/224 e 0 to 1 dl 1549470170 ref 1 fl Rpc:eX/2/ffffffff rc -5/-1 [Wed Feb 6 08:19:43 2019][1692357.508905] LustreError: 127196:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1 previous similar message [Wed Feb 6 08:25:38 2019][1692712.529441] LNetError: 19141:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.112.8@tcp from [Wed Feb 6 08:25:38 2019][1692712.529444] LNetError: 17999:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.115.2@tcp from [Wed Feb 6 08:25:38 2019][1692712.529448] LNetError: 17999:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 179978921 previous similar messages [Wed Feb 6 08:25:38 2019][1692712.551439] Lustre: 17999:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549470514/real 1549470514] req@ffffa0791bcd7b00 x1623667928522016/t0(0) o104->fir-MDT0003@10.10.115.2@tcp:15/16 lens 296/224 e 0 to 1 dl 1549470525 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 08:25:38 2019][1692712.551441] Lustre: 17999:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 181652599 previous similar messages [Wed Feb 6 08:25:38 2019][1692712.600386] LNetError: 19141:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 23156 previous similar messages [Wed Feb 6 08:29:11 2019][1692925.491180] LustreError: 17919:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.112.8@tcp) failed to reply to blocking AST (req@ffffa05aea03b900 x1623667928828608 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa089e66b0000/0x8c1de53946e794b9 lrc: 4/0,0 mode: PR/PR res: [0x280000e34:0x3:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.112.8@tcp remote: 0x8ea3e4d271873e04 expref: 12 pid: 14718 timeout: 1693060 lvb_type: 0 [Wed Feb 6 08:29:11 2019][1692925.533680] LustreError: 17919:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 22 previous similar messages [Wed Feb 6 08:29:11 2019][1692925.544032] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.112.8@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 08:29:11 2019][1692925.556631] LustreError: Skipped 21 previous similar messages [Wed Feb 6 08:29:11 2019][1692925.562581] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.8@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa089e66b0000/0x8c1de53946e794b9 lrc: 3/0,0 mode: PR/PR res: [0x280000e34:0x3:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.112.8@tcp remote: 0x8ea3e4d271873e04 expref: 13 pid: 14718 timeout: 0 lvb_type: 0 [Wed Feb 6 08:29:11 2019][1692925.599720] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 21 previous similar messages [Wed Feb 6 08:29:11 2019][1692925.609896] LustreError: 17919:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa0728a7f3400 ns: mdt-fir-MDT0003_UUID lock: ffffa0539b5d0240/0x8c1de53946e794d5 lrc: 5/0,0 mode: PW/PW res: [0x280000e34:0x3:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT flags: 0x50200000000000 nid: 10.10.112.8@tcp remote: 0x8ea3e4d271873e12 expref: 7 pid: 17919 timeout: 0 lvb_type: 0 [Wed Feb 6 08:29:11 2019][1692925.644275] LustreError: 17919:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 21 previous similar messages [Wed Feb 6 08:29:11 2019][1692925.655096] Lustre: fir-MDT0003: Connection restored to 6e2bb65e-930e-1303-68e8-03e2c472f77f (at 10.10.112.8@tcp) [Wed Feb 6 08:29:11 2019][1692925.665532] Lustre: Skipped 138 previous similar messages [Wed Feb 6 08:35:38 2019][1693312.534278] LNetError: 129278:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.115.2@tcp from [Wed Feb 6 08:35:38 2019][1693312.545059] LNetError: 129278:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 224050695 previous similar messages [Wed Feb 6 08:35:38 2019][1693312.556287] Lustre: 129245:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549471114/real 1549471114] req@ffffa051c2c13900 x1623667930640272/t0(0) o104->fir-MDT0003@10.10.19.4@tcp:15/16 lens 296/224 e 0 to 1 dl 1549471125 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 08:35:38 2019][1693312.556290] Lustre: 129245:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 227342130 previous similar messages [Wed Feb 6 08:39:51 2019][1693565.496329] LustreError: 19060:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.115.2@tcp) failed to reply to blocking AST (req@ffffa06332d7aa00 x1623667931195488 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa07035504800/0x8c1de539477231fe lrc: 4/0,0 mode: PR/PR res: [0x280000e39:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.115.2@tcp remote: 0xb8e1c27d85cff187 expref: 10 pid: 129278 timeout: 1693700 lvb_type: 0 [Wed Feb 6 08:39:51 2019][1693565.496334] LustreError: 138-a: fir-MDT0003: A client on nid 10.10.16.2@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 08:39:51 2019][1693565.496336] LustreError: Skipped 24 previous similar messages [Wed Feb 6 08:39:51 2019][1693565.496360] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.15.5@tcp ns: mdt-fir-MDT0003_UUID lock: ffffa06735217980/0x8c1de539477232c2 lrc: 3/0,0 mode: PR/PR res: [0x280000e3d:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349cc72 expref: 11 pid: 129271 timeout: 0 lvb_type: 0 [Wed Feb 6 08:39:51 2019][1693565.496363] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 25 previous similar messages [Wed Feb 6 08:39:51 2019][1693565.496441] LustreError: 17995:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa051da4c5c00 ns: mdt-fir-MDT0003_UUID lock: ffffa0695d487740/0x8c1de539477232fa lrc: 3/0,0 mode: PW/PW res: [0x280000e3d:0x2:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x50200000000000 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349cc80 expref: 4 pid: 17995 timeout: 0 lvb_type: 0 [Wed Feb 6 08:39:51 2019][1693565.496445] LustreError: 17995:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 25 previous similar messages [Wed Feb 6 08:39:51 2019][1693565.497131] Lustre: fir-MDT0003: Connection restored to 7324bf3d-6a04-b1ff-c579-e86bbeb8be32 (at 10.10.16.2@tcp) [Wed Feb 6 08:39:51 2019][1693565.497132] Lustre: Skipped 27 previous similar messages [Wed Feb 6 08:39:51 2019][1693565.664954] LustreError: 19060:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 29 previous similar messages [Wed Feb 6 08:50:34 2019][1694208.179857] Lustre: fir-MDT0001: haven't heard from client 40ad8941-e56d-16a8-06a4-9f03d85f8e6b (at 10.9.107.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa05b1f844000, cur 1549472010 expire 1549471860 last 1549471783 [Wed Feb 6 08:50:34 2019][1694208.201824] Lustre: Skipped 1 previous similar message [Wed Feb 6 08:50:45 2019][1694219.181111] Lustre: fir-MDT0003: haven't heard from client 40ad8941-e56d-16a8-06a4-9f03d85f8e6b (at 10.9.107.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa063f9041c00, cur 1549472021 expire 1549471871 last 1549471794 [Wed Feb 6 08:54:47 2019][1694461.730631] Lustre: fir-MDT0001: Connection restored to 8ae4cc68-204d-d2bb-4908-94f49692744b (at 10.8.20.15@o2ib6) [Wed Feb 6 08:54:47 2019][1694461.741155] Lustre: Skipped 7 previous similar messages [Wed Feb 6 09:04:43 2019][1695057.166391] Lustre: fir-MDT0003: haven't heard from client e7c11665-b408-cbb8-ba80-fcc49917f67e (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa075c86fd000, cur 1549472859 expire 1549472709 last 1549472632 [Wed Feb 6 09:06:53 2019][1695187.974900] Lustre: fir-MDT0001: Connection restored to cef0c972-4cf2-c1ae-1fb9-c404ed41cb0f (at 10.8.8.22@o2ib6) [Wed Feb 6 09:06:53 2019][1695187.985335] Lustre: Skipped 1 previous similar message [Wed Feb 6 09:10:52 2019][1695426.702330] LNet: Service thread pid 17994 was inactive for 200.36s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 09:10:52 2019][1695426.719459] LNet: Skipped 1 previous similar message [Wed Feb 6 09:10:52 2019][1695426.724609] Pid: 17994, comm: mdt01_050 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 09:10:52 2019][1695426.734519] Call Trace: [Wed Feb 6 09:10:52 2019][1695426.737156] [] 0xffffffffffffffff [Wed Feb 6 09:10:52 2019][1695426.742364] LustreError: dumping log to /tmp/lustre-log.1549473228.17994 [Wed Feb 6 09:12:34 2019][1695528.440861] Lustre: 18178:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549473323/real 1549473329] req@ffffa05f243d1200 x1623667950589872/t0(0) o106->fir-MDT0003@10.9.105.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1549473330 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Wed Feb 6 09:12:34 2019][1695528.468375] Lustre: 18178:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 85888834 previous similar messages [Wed Feb 6 09:12:37 2019][1695531.763738] Lustre: fir-OST000d-osc-MDT0003: Connection to fir-OST000d (at 10.0.10.104@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 09:12:39 2019][1695533.528168] Lustre: fir-OST0021-osc-MDT0001: Connection to fir-OST0021 (at 10.0.10.106@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 09:12:39 2019][1695533.821504] Lustre: fir-MDT0003: Client 219823d8-b457-54ef-a05b-a82af09f0709 (at 10.9.105.58@o2ib4) reconnecting [Wed Feb 6 09:12:39 2019][1695533.831850] Lustre: Skipped 75 previous similar messages [Wed Feb 6 09:12:39 2019][1695533.990735] LNetError: 126427:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Wed Feb 6 09:12:45 2019][1695539.773225] Lustre: fir-OST001f-osc-MDT0003: Connection to fir-OST001f (at 10.0.10.106@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 09:12:45 2019][1695539.789472] Lustre: Skipped 1 previous similar message [Wed Feb 6 09:12:49 2019][1695543.867015] LNet: 126419:0:(o2iblnd_cb.c:1484:kiblnd_reconnect_peer()) Abort reconnection of 10.0.10.203@o2ib7: connected [Wed Feb 6 09:12:51 2019][1695545.404341] Lustre: fir-OST0021-osc-MDT0001: Connection to fir-OST0021 (at 10.0.10.106@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 09:12:51 2019][1695545.420586] Lustre: Skipped 7 previous similar messages [Wed Feb 6 09:13:03 2019][1695558.069417] Lustre: fir-OST001b-osc-MDT0003: Connection to fir-OST001b (at 10.0.10.106@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 09:13:03 2019][1695558.085661] Lustre: Skipped 4 previous similar messages [Wed Feb 6 09:13:09 2019][1695563.695042] Lustre: fir-MDT0003: Received new LWP connection from 0@lo, removing former export from same NID [Wed Feb 6 09:13:16 2019][1695570.504958] Lustre: mdt_io: This server is not able to keep up with request traffic (cpu-bound). [Wed Feb 6 09:13:16 2019][1695570.513922] Lustre: 129301:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=1, svcEst=32, delay=7597 [Wed Feb 6 09:13:16 2019][1695570.524531] Lustre: 129301:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 1 previous similar message [Wed Feb 6 09:13:16 2019][1695570.534198] Lustre: 129301:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-3s), not sending early reply. Consider increasing at_early_margin (5)? req@ffffa05d988d1c50 x1624700577218912/t0(0) o3->63b89084-a198-377b-0bdb-5ed4c4e0cd41@10.9.106.63@o2ib4:459/0 lens 488/440 e 0 to 0 dl 1549473369 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:13:16 2019][1695570.565266] Lustre: 129301:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [Wed Feb 6 09:13:16 2019][1695570.578165] LustreError: 127235:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffffa05d988d1c50 x1624700577218912/t0(0) o3->63b89084-a198-377b-0bdb-5ed4c4e0cd41@10.9.106.63@o2ib4:459/0 lens 488/440 e 0 to 0 dl 1549473369 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:13:16 2019][1695570.602530] Lustre: fir-MDT0003: Bulk IO read error with 63b89084-a198-377b-0bdb-5ed4c4e0cd41 (at 10.9.106.63@o2ib4), client will retry: rc -110 [Wed Feb 6 09:13:16 2019][1695570.615658] Lustre: 127235:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43:3s); client may timeout. req@ffffa05d988d1c50 x1624700577218912/t0(0) o3->63b89084-a198-377b-0bdb-5ed4c4e0cd41@10.9.106.63@o2ib4:459/0 lens 488/440 e 0 to 0 dl 1549473369 ref 1 fl Complete:/0/ffffffff rc -110/-1 [Wed Feb 6 09:13:22 2019][1695577.028595] LustreError: 127225:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(131072) req@ffffa05b447e7850 x1624673089075248/t0(0) o4->c95418d8-abb0-d4fc-c763-439a35e76a6c@10.9.102.50@o2ib4:485/0 lens 488/448 e 1 to 0 dl 1549473395 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:13:22 2019][1695577.053788] Lustre: fir-MDT0001: Bulk IO write error with c95418d8-abb0-d4fc-c763-439a35e76a6c (at 10.9.102.50@o2ib4), client will retry: rc = -110 [Wed Feb 6 09:13:22 2019][1695577.067168] Lustre: Skipped 2 previous similar messages [Wed Feb 6 09:13:44 2019][1695598.430141] LustreError: 18044:0:(tgt_handler.c:644:process_req_last_xid()) @@@ Unexpected xid 5c5a1a91020c0 vs. last_xid 5c5a1a91020ef [Wed Feb 6 09:13:44 2019][1695598.430141] req@ffffa05d90dc9200 x1624673000366272/t0(0) o101->f97f4802-8c0b-e9cd-e8d7-93692decf22a@10.9.102.8@o2ib4:580/0 lens 480/0 e 0 to 0 dl 1549473490 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [Wed Feb 6 09:13:51 2019][1695605.459887] Lustre: 126461:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549473399/real 1549473399] req@ffffa0661deb9200 x1623667951206720/t0(0) o13->fir-OST0000-osc-MDT0001@10.0.10.101@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1549473406 ref 1 fl Rpc:RX/0/ffffffff rc 0/-1 [Wed Feb 6 09:13:51 2019][1695605.488344] Lustre: 126461:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2921 previous similar messages [Wed Feb 6 09:13:51 2019][1695605.498615] Lustre: fir-OST0000-osc-MDT0001: Connection to fir-OST0000 (at 10.0.10.101@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 09:13:51 2019][1695605.514855] Lustre: Skipped 23 previous similar messages [Wed Feb 6 09:13:54 2019][1695609.110865] Lustre: fir-MDT0001: Client 386cceaf-b887-f800-5980-0888a79dd601 (at 10.9.105.63@o2ib4) reconnecting [Wed Feb 6 09:13:54 2019][1695609.121213] Lustre: Skipped 456 previous similar messages [Wed Feb 6 09:13:59 2019][1695614.042197] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Wed Feb 6 09:13:59 2019][1695614.052633] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.202@o2ib7 (4): c: 1, oc: 1, rc: 8 [Wed Feb 6 09:14:04 2019][1695618.416912] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 3 seconds [Wed Feb 6 09:14:04 2019][1695618.427166] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 3 previous similar messages [Wed Feb 6 09:14:04 2019][1695618.437595] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.201@o2ib7 (8): c: 0, oc: 0, rc: 8 [Wed Feb 6 09:14:04 2019][1695618.449846] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 3 previous similar messages [Wed Feb 6 09:14:04 2019][1695618.459860] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.202@o2ib7: 0 seconds [Wed Feb 6 09:14:04 2019][1695618.470283] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 2 previous similar messages [Wed Feb 6 09:14:05 2019][1695619.672931] LustreError: 127265:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(131072) req@ffffa07b382d9c50 x1624673106110320/t0(0) o4->f650715e-6794-f2e6-d7e7-52e67122a477@10.9.102.70@o2ib4:520/0 lens 488/448 e 2 to 0 dl 1549473430 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:14:05 2019][1695619.698157] Lustre: fir-MDT0001: Bulk IO write error with f650715e-6794-f2e6-d7e7-52e67122a477 (at 10.9.102.70@o2ib4), client will retry: rc = -110 [Wed Feb 6 09:14:06 2019][1695620.535995] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.9.107.9@o2ib4 ns: mdt-fir-MDT0003_UUID lock: ffffa07b24680240/0x8c1de5394927b006 lrc: 3/0,0 mode: PR/PR res: [0x280000de1:0xfc:0x0].0x0 bits 0x40/0x0 rrc: 728 type: IBT flags: 0x60000400010020 nid: 10.9.107.9@o2ib4 remote: 0x853f1c456dd12604 expref: 5724 pid: 129793 timeout: 1695606 lvb_type: 0 [Wed Feb 6 09:14:06 2019][1695620.574282] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 4 previous similar messages [Wed Feb 6 09:14:07 2019][1695621.572043] LustreError: 126428:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffffa063b9a4c800 [Wed Feb 6 09:14:07 2019][1695621.583127] LustreError: 126428:0:(events.c:305:request_in_callback()) event type 2, status -103, service mdt_io [Wed Feb 6 09:14:07 2019][1695621.593565] LustreError: 129240:0:(pack_generic.c:590:__lustre_unpack_msg()) message length 0 too small for magic/version check [Wed Feb 6 09:14:07 2019][1695621.605220] LustreError: 129240:0:(sec.c:2068:sptlrpc_svc_unwrap_request()) error unpacking request from 12345-10.8.17.8@o2ib6 x1624705096089616 [Wed Feb 6 09:14:07 2019][1695621.794904] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.204@o2ib7: 0 seconds [Wed Feb 6 09:14:07 2019][1695621.805329] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 83 previous similar messages [Wed Feb 6 09:14:07 2019][1695621.815223] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Wed Feb 6 09:14:07 2019][1695621.825653] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 4 previous similar messages [Wed Feb 6 09:14:07 2019][1695621.836080] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.108@o2ib7 (4): c: 0, oc: 0, rc: 8 [Wed Feb 6 09:14:07 2019][1695621.848325] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 4 previous similar messages [Wed Feb 6 09:14:07 2019][1695621.849500] LustreError: 130112:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk READ req@ffffa05d988d2c50 x1624736164804768/t0(0) o3->38ab0677-0c51-35a5-8e38-bb5f254042a2@10.8.17.4@o2ib6:535/0 lens 488/440 e 0 to 0 dl 1549473445 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:14:07 2019][1695621.849524] Lustre: fir-MDT0003: Bulk IO read error with 38ab0677-0c51-35a5-8e38-bb5f254042a2 (at 10.8.17.4@o2ib6), client will retry: rc -110 [Wed Feb 6 09:14:07 2019][1695621.933990] LNetError: 126432:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Wed Feb 6 09:14:07 2019][1695621.946765] LNetError: 126432:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message [Wed Feb 6 09:14:09 2019][1695624.107081] LNetError: 126433:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Wed Feb 6 09:14:12 2019][1695626.794942] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.103@o2ib7: 0 seconds [Wed Feb 6 09:14:12 2019][1695626.805375] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 45 previous similar messages [Wed Feb 6 09:14:18 2019][1695632.514272] LustreError: 129259:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa0578cb1d400 x1623667953387824/t0(0) o104->fir-MDT0003@10.9.103.1@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 09:14:21 2019][1695635.302386] LustreError: 129259:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa05a10b17b00 x1623667954791248/t0(0) o104->fir-MDT0003@10.9.103.1@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 09:14:21 2019][1695635.323654] LustreError: 129259:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [Wed Feb 6 09:14:32 2019][1695646.810196] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.9.106.6@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 09:14:32 2019][1695646.827649] LustreError: Skipped 56 previous similar messages [Wed Feb 6 09:14:44 2019][1695658.481324] LustreError: 127535:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(131072) req@ffffa07b382da050 x1624673027453136/t0(0) o4->f5fe3def-7261-239e-1d83-4ac81d4cfaa5@10.9.102.61@o2ib4:591/0 lens 488/448 e 1 to 0 dl 1549473501 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:14:44 2019][1695658.506519] Lustre: fir-MDT0001: Bulk IO write error with f5fe3def-7261-239e-1d83-4ac81d4cfaa5 (at 10.9.102.61@o2ib4), client will retry: rc = -110 [Wed Feb 6 09:14:48 2019][1695662.769225] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [ldlm_bl_05:10646] [Wed Feb 6 09:14:48 2019][1695662.777141] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 09:14:48 2019][1695662.882384] CPU: 7 PID: 10646 Comm: ldlm_bl_05 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 09:14:48 2019][1695662.894800] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 09:14:48 2019][1695662.902540] task: ffffa07b7647b0c0 ti: ffffa05452fa0000 task.ti: ffffa05452fa0000 [Wed Feb 6 09:14:48 2019][1695662.910191] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 09:14:48 2019][1695662.920689] RSP: 0018:ffffa05452fa3bb0 EFLAGS: 00000282 [Wed Feb 6 09:14:48 2019][1695662.926175] RAX: 0000000000000001 RBX: ffffa0633c8094a0 RCX: ffffa0633c8094a0 [Wed Feb 6 09:14:48 2019][1695662.933478] RDX: ffffa05452fa3ca8 RSI: ffffa05bfdf05100 RDI: ffffa0540fd1a1c0 [Wed Feb 6 09:14:48 2019][1695662.940786] RBP: ffffa05452fa3c08 R08: ffffa05452fa3ca8 R09: 0000000080000f12 [Wed Feb 6 09:14:48 2019][1695662.948092] R10: 0000000000000012 R11: ffffa05bfdf05100 R12: ffffa05452fa3ca8 [Wed Feb 6 09:14:48 2019][1695662.955398] R13: 0000000080000f12 R14: 0000000000000012 R15: ffffa05bfdf05100 [Wed Feb 6 09:14:48 2019][1695662.962705] FS: 00007f5bb9793740(0000) GS:ffffa08bbf440000(0000) knlGS:0000000000000000 [Wed Feb 6 09:14:48 2019][1695662.970963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 09:14:48 2019][1695662.976883] CR2: 00007f5bb92a61e0 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Wed Feb 6 09:14:48 2019][1695662.984190] Call Trace: [Wed Feb 6 09:14:48 2019][1695662.986845] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 09:14:48 2019][1695662.994524] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 09:14:48 2019][1695663.002458] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 09:14:48 2019][1695663.009616] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 09:14:48 2019][1695663.016776] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:14:48 2019][1695663.025235] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 09:14:48 2019][1695663.033436] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 09:14:48 2019][1695663.040485] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:14:48 2019][1695663.046261] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:14:48 2019][1695663.053831] [] kthread+0xd1/0xe0 [Wed Feb 6 09:14:48 2019][1695663.058881] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:14:48 2019][1695663.065151] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:14:48 2019][1695663.071768] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:14:48 2019][1695663.078033] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 09:15:06 2019][1695680.345580] Lustre: fir-OST0011-osc-MDT0003: Connection to fir-OST0011 (at 10.0.10.104@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 09:15:06 2019][1695680.361819] Lustre: Skipped 59 previous similar messages [Wed Feb 6 09:15:07 2019][1695681.732248] Lustre: fir-MDT0001: Received new LWP connection from 10.0.10.51@o2ib7, removing former export from same NID [Wed Feb 6 09:15:16 2019][1695690.731030] Lustre: fir-MDT0001: Received new LWP connection from 10.0.10.51@o2ib7, removing former export from same NID [Wed Feb 6 09:15:16 2019][1695690.769449] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [ldlm_bl_05:10646] [Wed Feb 6 09:15:16 2019][1695690.777370] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 09:15:16 2019][1695690.882613] CPU: 7 PID: 10646 Comm: ldlm_bl_05 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 09:15:16 2019][1695690.895028] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 09:15:16 2019][1695690.902767] task: ffffa07b7647b0c0 ti: ffffa05452fa0000 task.ti: ffffa05452fa0000 [Wed Feb 6 09:15:16 2019][1695690.910420] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 09:15:16 2019][1695690.920917] RSP: 0018:ffffa05452fa3bb0 EFLAGS: 00000282 [Wed Feb 6 09:15:16 2019][1695690.926401] RAX: 0000000000000001 RBX: ffffa0633c8094a0 RCX: ffffa0633c8094a0 [Wed Feb 6 09:15:16 2019][1695690.933707] RDX: ffffa05452fa3ca8 RSI: ffffa05bfdf05100 RDI: ffffa0732db5f980 [Wed Feb 6 09:15:16 2019][1695690.941013] RBP: ffffa05452fa3c08 R08: ffffa05452fa3ca8 R09: 0000000080000f12 [Wed Feb 6 09:15:16 2019][1695690.948319] R10: 0000000000000012 R11: ffffa05bfdf05100 R12: ffffa05452fa3ca8 [Wed Feb 6 09:15:16 2019][1695690.955626] R13: 0000000080000f12 R14: 0000000000000012 R15: ffffa05bfdf05100 [Wed Feb 6 09:15:16 2019][1695690.962932] FS: 00007f5bb9793740(0000) GS:ffffa08bbf440000(0000) knlGS:0000000000000000 [Wed Feb 6 09:15:16 2019][1695690.971193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 09:15:16 2019][1695690.977112] CR2: 00007f5bb92a61e0 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Wed Feb 6 09:15:16 2019][1695690.984419] Call Trace: [Wed Feb 6 09:15:16 2019][1695690.987082] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 09:15:16 2019][1695690.994769] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 09:15:16 2019][1695691.002711] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 09:15:16 2019][1695691.009877] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 09:15:16 2019][1695691.017045] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:15:16 2019][1695691.025513] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 09:15:16 2019][1695691.033722] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 09:15:16 2019][1695691.040775] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:15:16 2019][1695691.046559] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:15:16 2019][1695691.054127] [] kthread+0xd1/0xe0 [Wed Feb 6 09:15:16 2019][1695691.059178] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:15:16 2019][1695691.065446] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:15:16 2019][1695691.072057] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:15:16 2019][1695691.078322] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 09:15:21 2019][1695695.342480] INFO: rcu_sched self-detected stall on CPU[1695695.343481] INFO: rcu_sched detected stalls on CPUs/tasks: { 7} (detected by 45, t=60002 jiffies, g=72716369, c=72716368, q=45951) [Wed Feb 6 09:15:21 2019][1695695.343498] Task dump for CPU 7: [Wed Feb 6 09:15:21 2019][1695695.343500] ldlm_bl_05 R running task 0 10646 2 0x00000088 [Wed Feb 6 09:15:21 2019][1695695.343503] Call Trace: [Wed Feb 6 09:15:21 2019][1695695.343561] [] ? ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.343597] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.343630] [] ? ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.343662] [] ? __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.343694] [] ? ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.343727] [] ? ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.343762] [] ? ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.343767] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:15:21 2019][1695695.343802] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.343805] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:15:21 2019][1695695.343807] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:15:21 2019][1695695.343811] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:15:21 2019][1695695.343812] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:15:21 2019] [Wed Feb 6 09:15:21 2019][1695695.465307] { 7} (t=60126 jiffies g=72716369 c=72716368 q=45951) [Wed Feb 6 09:15:21 2019][1695695.471944] Task dump for CPU 7: [Wed Feb 6 09:15:21 2019][1695695.475352] ldlm_bl_05 R running task 0 10646 2 0x00000088 [Wed Feb 6 09:15:21 2019][1695695.482623] Call Trace: [Wed Feb 6 09:15:21 2019][1695695.485252] [] sched_show_task+0xa8/0x110 [Wed Feb 6 09:15:21 2019][1695695.491723] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 09:15:21 2019][1695695.497300] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 09:15:21 2019][1695695.503390] [] rcu_check_callbacks+0x442/0x730 [Wed Feb 6 09:15:21 2019][1695695.509660] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 09:15:21 2019][1695695.515933] [] update_process_times+0x46/0x80 [Wed Feb 6 09:15:21 2019][1695695.522110] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 09:15:21 2019][1695695.528027] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 09:15:21 2019][1695695.533864] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 09:15:21 2019][1695695.540126] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 09:15:21 2019][1695695.546137] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 09:15:21 2019][1695695.552837] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 09:15:21 2019][1695695.559362] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 09:15:21 2019][1695695.565711] [] ? ldlm_inodebits_compat_queue+0x19f/0x440 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.574353] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.582039] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.589983] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.597150] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.604308] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.612767] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.620971] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.628018] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:15:21 2019][1695695.633796] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:15:21 2019][1695695.641364] [] kthread+0xd1/0xe0 [Wed Feb 6 09:15:21 2019][1695695.646416] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:15:21 2019][1695695.652682] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:15:21 2019][1695695.659295] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:15:21 2019][1695696.062837] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 10.9.103.1@o2ib4 ns: mdt-fir-MDT0003_UUID lock: ffffa077fb611440/0x8c1de53949182241 lrc: 3/0,0 mode: PR/PR res: [0x280000e29:0xe9:0x0].0x0 bits 0x40/0x0 rrc: 30955 type: IBT flags: 0x60000400010020 nid: 10.9.103.1@o2ib4 remote: 0x6afa261f1c79a81c expref: 40290 pid: 126631 timeout: 1695680 lvb_type: 0 [Wed Feb 6 09:15:21 2019][1695696.101436] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 39 previous similar messages [Wed Feb 6 09:15:26 2019][1695700.804225] LustreError: 126604:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.9.106.6@o2ib4 arrived at 1549473502 with bad export cookie 10096477973470406113 [Wed Feb 6 09:15:27 2019][1695702.102151] LustreError: 126604:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.9.106.6@o2ib4 arrived at 1549473502 with bad export cookie 10096477973470406113 [Wed Feb 6 09:15:27 2019][1695702.117956] LustreError: 126604:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 1 previous similar message [Wed Feb 6 09:15:29 2019][1695703.371643] LustreError: 126606:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.9.106.6@o2ib4 arrived at 1549473502 with bad export cookie 10096477973470406113 [Wed Feb 6 09:15:29 2019][1695703.387450] LustreError: 126606:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 1 previous similar message [Wed Feb 6 09:15:31 2019][1695705.771665] LustreError: 12936:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.9.106.6@o2ib4 arrived at 1549473504 with bad export cookie 10096477973470406113 [Wed Feb 6 09:15:31 2019][1695705.787391] LustreError: 12936:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 3 previous similar messages [Wed Feb 6 09:15:33 2019][1695708.083624] LNetError: 7286:0:(o2iblnd_cb.c:3281:kiblnd_cm_callback()) 10.0.10.202@o2ib7 DISCONNECTED [Wed Feb 6 09:15:35 2019][1695709.830343] LustreError: 126604:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.9.106.6@o2ib4 arrived at 1549473504 with bad export cookie 10096477973470406113 [Wed Feb 6 09:15:35 2019][1695709.846150] LustreError: 126604:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 6 previous similar messages [Wed Feb 6 09:15:36 2019][1695710.924609] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 21s! [migration/25:137] [Wed Feb 6 09:15:36 2019][1695710.932610] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 09:15:36 2019][1695711.037853] CPU: 25 PID: 137 Comm: migration/25 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 09:15:36 2019][1695711.050356] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 09:15:36 2019][1695711.058096] task: ffffa05c2996d140 ti: ffffa05c29994000 task.ti: ffffa05c29994000 [Wed Feb 6 09:15:36 2019][1695711.065750] RIP: 0010:[] [] multi_cpu_stop+0x4a/0x110 [Wed Feb 6 09:15:36 2019][1695711.074209] RSP: 0000:ffffa05c29997d98 EFLAGS: 00000246 [Wed Feb 6 09:15:36 2019][1695711.079695] RAX: 0000000000000001 RBX: ffffa07b775cc100 RCX: dead000000000200 [Wed Feb 6 09:15:36 2019][1695711.087001] RDX: ffffa06b7f795ff0 RSI: 0000000000000282 RDI: ffffa0687064bab0 [Wed Feb 6 09:15:36 2019][1695711.094306] RBP: ffffa05c29997dc0 R08: ffffa0687064ba80 R09: 0000000000000001 [Wed Feb 6 09:15:36 2019][1695711.101613] R10: 000000000000bec5 R11: 0000000000000001 R12: ffffa06b7f79ab80 [Wed Feb 6 09:15:36 2019][1695711.108919] R13: ffffa05c29f25c00 R14: ffffffffa72d2eb2 R15: ffffa05c29997d00 [Wed Feb 6 09:15:36 2019][1695711.116225] FS: 00007f81fde6d740(0000) GS:ffffa06b7f780000(0000) knlGS:0000000000000000 [Wed Feb 6 09:15:36 2019][1695711.124485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 09:15:36 2019][1695711.130402] CR2: 00007ffeb07c72e0 CR3: 0000003e74b18000 CR4: 00000000003407e0 [Wed Feb 6 09:15:36 2019][1695711.137711] Call Trace: [Wed Feb 6 09:15:36 2019][1695711.140338] [] ? cpu_stop_should_run+0x50/0x50 [Wed Feb 6 09:15:36 2019][1695711.146603] [] cpu_stopper_thread+0x99/0x150 [Wed Feb 6 09:15:36 2019][1695711.152699] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:15:36 2019][1695711.158358] [] smpboot_thread_fn+0x144/0x1a0 [Wed Feb 6 09:15:36 2019][1695711.164447] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:15:36 2019][1695711.170454] [] kthread+0xd1/0xe0 [Wed Feb 6 09:15:37 2019][1695711.175507] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:15:37 2019][1695711.181773] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:15:37 2019][1695711.188386] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:15:37 2019][1695711.194650] Code: 66 90 66 90 49 89 c5 48 8b 47 18 48 85 c0 0f 84 b3 00 00 00 0f a3 18 19 db 85 db 41 0f 95 c6 45 31 ff 31 c0 0f 1f 44 00 00 f3 90 <41> 8b 5c 24 20 39 c3 74 5d 83 fb 02 74 68 83 fb 03 75 05 45 84 [Wed Feb 6 09:15:38 2019][1695712.851285] LNetError: 126421:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) [Wed Feb 6 09:15:38 2019][1695712.864061] LNetError: 126421:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages [Wed Feb 6 09:15:39 2019][1695713.438198] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 9 seconds [Wed Feb 6 09:15:39 2019][1695713.448458] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 2 previous similar messages [Wed Feb 6 09:15:39 2019][1695713.458886] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.201@o2ib7 (14): c: 0, oc: 2, rc: 8 [Wed Feb 6 09:15:39 2019][1695713.471223] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 2 previous similar messages [Wed Feb 6 09:15:39 2019][1695713.481232] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.101@o2ib7: 26 seconds [Wed Feb 6 09:15:39 2019][1695713.491746] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 18 previous similar messages [Wed Feb 6 09:15:39 2019][1695713.501530] LustreError: 126419:0:(events.c:450:server_bulk_callback()) event type 5, status -110, desc ffffa0616f6a2c00 [Wed Feb 6 09:15:39 2019][1695713.556352] LustreError: 126978:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0024-osc-MDT0001: cannot cleanup orphans: rc = -11 [Wed Feb 6 09:15:40 2019][1695714.211043] LNetError: 18224:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.18.24@o2ib6 from [Wed Feb 6 09:15:40 2019][1695714.221834] LNetError: 18224:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 84967285 previous similar messages [Wed Feb 6 09:15:48 2019][1695722.769705] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [ldlm_bl_05:10646] [Wed Feb 6 09:15:48 2019][1695722.777616] Modules linked in:[1695722.778704] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [ldlm_bl_10:10651] [Wed Feb 6 09:15:48 2019][1695722.778706] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 09:15:48 2019][1695722.778795] CPU: 8 PID: 10651 Comm: ldlm_bl_10 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 09:15:48 2019][1695722.778796] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 09:15:48 2019][1695722.778798] task: ffffa0516a6130c0 ti: ffffa0557bed8000 task.ti: ffffa0557bed8000 [Wed Feb 6 09:15:48 2019][1695722.778799] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x1d0/0x200 [Wed Feb 6 09:15:48 2019][1695722.778808] RSP: 0018:ffffa0557bedbc60 EFLAGS: 00000202 [Wed Feb 6 09:15:48 2019][1695722.778810] RAX: 0000000000000101 RBX: ffffa0557bedbc98 RCX: 0000000000000001 [Wed Feb 6 09:15:48 2019][1695722.778811] RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffffa063b6f02cdc [Wed Feb 6 09:15:48 2019][1695722.778812] RBP: ffffa0557bedbc60 R08: 0000000000000101 R09: 00000000800011b0 [Wed Feb 6 09:15:48 2019][1695722.778813] R10: 0000000000000030 R11: 0000000280000d6c R12: 0000000000000000 [Wed Feb 6 09:15:48 2019][1695722.778814] R13: 0000000000000000 R14: 0000000000000002 R15: ffffa0522759e540 [Wed Feb 6 09:15:48 2019][1695722.778816] FS: 00007fcf1360e700(0000) GS:ffffa05b7ee80000(0000) knlGS:0000000000000000 [Wed Feb 6 09:15:48 2019][1695722.778818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 09:15:48 2019][1695722.778819] CR2: 00007fcf173231d8 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Wed Feb 6 09:15:48 2019][1695722.778820] Call Trace: [Wed Feb 6 09:15:48 2019][1695722.778828] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 09:15:48 2019][1695722.778833] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 09:15:48 2019][1695722.778888] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 09:15:48 2019][1695722.778923] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 09:15:48 2019][1695722.778958] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 09:15:48 2019][1695722.778993] [] ldlm_cancel_lock_for_export.isra.26+0x7e/0x390 [ptlrpc] [Wed Feb 6 09:15:48 2019][1695722.779026] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 09:15:48 2019][1695722.779066] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 09:15:48 2019][1695722.779071] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:15:48 2019][1695722.779107] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:15:48 2019][1695722.779111] [] kthread+0xd1/0xe0 [Wed Feb 6 09:15:48 2019][1695722.779113] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:15:48 2019][1695722.779117] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:15:48 2019][1695722.779119] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:15:48 2019][1695722.779120] Code: fe 00 01 00 00 74 f4 e9 93 fe ff ff 0f 1f 80 00 00 00 00 83 fa 01 75 11 0f 1f 00 e9 68 fe ff ff 0f 1f 00 85 c0 74 0c f3 90 8b 07 <0f> b6 c0 83 f8 03 75 f0 b8 01 00 00 00 66 89 07 5d c3 66 0f 1f [Wed Feb 6 09:15:48 2019] [Wed Feb 6 09:15:48 2019][1695723.112191] osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 09:15:49 2019][1695723.216045] CPU: 7 PID: 10646 Comm: ldlm_bl_05 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 09:15:49 2019][1695723.228462] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 09:15:49 2019][1695723.236201] task: ffffa07b7647b0c0 ti: ffffa05452fa0000 task.ti: ffffa05452fa0000 [Wed Feb 6 09:15:49 2019][1695723.243855] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 09:15:49 2019][1695723.254342] RSP: 0018:ffffa05452fa3bb0 EFLAGS: 00000282 [Wed Feb 6 09:15:49 2019][1695723.259828] RAX: 0000000000000001 RBX: ffffa0633c8094a0 RCX: ffffa0633c8094a0 [Wed Feb 6 09:15:49 2019][1695723.267133] RDX: ffffa05452fa3ca8 RSI: ffffa05bfdf05100 RDI: ffffa0897b7a0000 [Wed Feb 6 09:15:49 2019][1695723.274439] RBP: ffffa05452fa3c08 R08: ffffa05452fa3ca8 R09: 0000000080000f12 [Wed Feb 6 09:15:49 2019][1695723.281746] R10: 0000000000000012 R11: ffffa05bfdf05100 R12: ffffa05452fa3ca8 [Wed Feb 6 09:15:49 2019][1695723.289052] R13: 0000000080000f12 R14: 0000000000000012 R15: ffffa05bfdf05100 [Wed Feb 6 09:15:49 2019][1695723.296359] FS: 00007f5bb9793740(0000) GS:ffffa08bbf440000(0000) knlGS:0000000000000000 [Wed Feb 6 09:15:49 2019][1695723.304616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 09:15:49 2019][1695723.310537] CR2: 00007f5bb92a61e0 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Wed Feb 6 09:15:49 2019][1695723.317843] Call Trace: [Wed Feb 6 09:15:49 2019][1695723.320500] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 09:15:49 2019][1695723.328175] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 09:15:49 2019][1695723.336111] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 09:15:49 2019][1695723.343270] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 09:15:49 2019][1695723.350430] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:15:49 2019][1695723.358889] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 09:15:49 2019][1695723.367090] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 09:15:49 2019][1695723.374138] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:15:49 2019][1695723.379907] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:15:49 2019][1695723.387472] [] kthread+0xd1/0xe0 [Wed Feb 6 09:15:49 2019][1695723.392526] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:15:49 2019][1695723.398793] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:15:49 2019][1695723.405404] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:15:49 2019][1695723.411669] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 09:15:49 2019][1695724.078743] LustreError: 127669:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffffa07b416a4450 x1624672672071632/t0(0) o3->f836dc38-744c-c596-c0a2-f4034f572070@10.9.102.15@o2ib4:663/0 lens 488/440 e 1 to 0 dl 1549473573 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:15:49 2019][1695724.103089] Lustre: fir-MDT0003: Bulk IO read error with f836dc38-744c-c596-c0a2-f4034f572070 (at 10.9.102.15@o2ib4), client will retry: rc -110 [Wed Feb 6 09:16:00 2019][1695734.985802] NMI watchdog: BUG: soft lockup - CPU#32 stuck for 22s! [ldlm_bl_08:10649] [Wed Feb 6 09:16:00 2019][1695734.993804] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 09:16:00 2019][1695735.099048] CPU: 32 PID: 10649 Comm: ldlm_bl_08 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 09:16:00 2019][1695735.111558] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 09:16:00 2019][1695735.119297] task: ffffa0516a612080 ti: ffffa053db990000 task.ti: ffffa053db990000 [Wed Feb 6 09:16:00 2019][1695735.126951] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x1ce/0x200 [Wed Feb 6 09:16:00 2019][1695735.137057] RSP: 0018:ffffa053db993c60 EFLAGS: 00000202 [Wed Feb 6 09:16:00 2019][1695735.142543] RAX: 0000000000000001 RBX: ffffa053db993c98 RCX: 0000000000000001 [Wed Feb 6 09:16:00 2019][1695735.149847] RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffffa063b6f02cdc [Wed Feb 6 09:16:00 2019][1695735.157153] RBP: ffffa053db993c60 R08: 0000000000000101 R09: 00000000800011b0 [Wed Feb 6 09:16:01 2019][1695735.164459] R10: 0000000000000030 R11: 0000000280000d6c R12: 0000000000000000 [Wed Feb 6 09:16:01 2019][1695735.171765] R13: 0000000000000000 R14: 0000000000000002 R15: ffffa07675cdc800 [Wed Feb 6 09:16:01 2019][1695735.179074] FS: 00007fcf1360e700(0000) GS:ffffa05b7f000000(0000) knlGS:0000000000000000 [Wed Feb 6 09:16:01 2019][1695735.187332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 09:16:01 2019][1695735.193252] CR2: 00007efe9c7c8000 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Wed Feb 6 09:16:01 2019][1695735.200558] Call Trace: [Wed Feb 6 09:16:01 2019][1695735.203188] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 09:16:01 2019][1695735.209626] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 09:16:01 2019][1695735.215324] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 09:16:01 2019][1695735.222056] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 09:16:01 2019][1695735.229129] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 09:16:01 2019][1695735.235860] [] ldlm_cancel_lock_for_export.isra.26+0x7e/0x390 [ptlrpc] [Wed Feb 6 09:16:01 2019][1695735.244234] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 09:16:01 2019][1695735.252436] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 09:16:01 2019][1695735.259486] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:16:01 2019][1695735.265261] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:16:01 2019][1695735.272832] [] kthread+0xd1/0xe0 [Wed Feb 6 09:16:01 2019][1695735.277883] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:16:01 2019][1695735.284150] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:16:01 2019][1695735.290761] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:16:01 2019][1695735.297028] Code: 37 81 fe 00 01 00 00 74 f4 e9 93 fe ff ff 0f 1f 80 00 00 00 00 83 fa 01 75 11 0f 1f 00 e9 68 fe ff ff 0f 1f 00 85 c0 74 0c f3 90 <8b> 07 0f b6 c0 83 f8 03 75 f0 b8 01 00 00 00 66 89 07 5d c3 66 [Wed Feb 6 09:16:02 2019][1695736.674871] LustreError: 126637:0:(ldlm_lib.c:3273:target_bulk_io()) @@@ truncated bulk READ 0(4096) req@ffffa06383e10f00 x1624699253673792/t0(0) o37->74adb3c7-1310-d82a-c0b0-2c64f425de3b@10.9.104.61@o2ib4:645/0 lens 448/440 e 1 to 0 dl 1549473555 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:16:02 2019][1695736.674912] Lustre: fir-MDT0003: Bulk IO read error with d405c92c-c32a-78c2-b321-55f4db7bb864 (at 10.9.102.14@o2ib4), client will retry: rc -110 [Wed Feb 6 09:16:02 2019][1695736.712803] LustreError: 126637:0:(ldlm_lib.c:3273:target_bulk_io()) Skipped 2 previous similar messages [Wed Feb 6 09:16:09 2019][1695743.641355] LustreError: 127297:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa0780babd700 x1623667954902656/t0(0) o104->fir-MDT0003@10.8.17.1@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 09:16:09 2019][1695743.662556] LustreError: 127297:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 8 previous similar messages [Wed Feb 6 09:16:21 2019][1695755.612983] LustreError: 13294:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffffa08a19ade300 x1624699237667472/t0(0) o37->c3afa774-0e39-3459-0af2-ddf264214c5b@10.9.104.54@o2ib4:675/0 lens 448/440 e 1 to 0 dl 1549473585 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:16:24 2019][1695759.123351] Lustre: fir-MDT0003: Client 9cd6b017-d662-dea3-a183-2212fb4556a3 (at 10.8.12.26@o2ib6) reconnecting [Wed Feb 6 09:16:24 2019][1695759.133611] Lustre: Skipped 567 previous similar messages [Wed Feb 6 09:16:39 2019][1695773.618151] LustreError: 127066:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa06a10c83000 x1623667955081520/t0(0) o104->fir-MDT0003@10.8.17.1@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 09:16:54 2019][1695788.280727] Lustre: fir-MDT0001: Connection restored to 94c2dba8-b225-c91a-753d-91a7f0495a0f (at 10.9.101.8@o2ib4) [Wed Feb 6 09:16:54 2019][1695788.291246] Lustre: Skipped 1618 previous similar messages [Wed Feb 6 09:16:56 2019][1695791.140401] Lustre: 126481:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549473585/real 1549473585] req@ffffa076ab07a700 x1623667955283152/t0(0) o13->fir-OST0000-osc-MDT0003@10.0.10.101@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1549473592 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Wed Feb 6 09:16:57 2019][1695791.168775] Lustre: 126481:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 41191 previous similar messages [Wed Feb 6 09:16:57 2019][1695791.179133] Lustre: fir-OST0000-osc-MDT0003: Connection to fir-OST0000 (at 10.0.10.101@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 09:16:57 2019][1695791.195368] Lustre: Skipped 32 previous similar messages [Wed Feb 6 09:18:26 2019][1695880.515114] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.9.103.4@o2ib4 ns: mdt-fir-MDT0003_UUID lock: ffffa057252f4c80/0x8c1de539491c20e2 lrc: 3/0,0 mode: PR/PR res: [0x280000e29:0xea:0x0].0x0 bits 0x40/0x0 rrc: 24642 type: IBT flags: 0x60000400010020 nid: 10.9.103.4@o2ib4 remote: 0x615f36ed6cdd0018 expref: 27166 pid: 17885 timeout: 1695866 lvb_type: 0 [Wed Feb 6 09:18:26 2019][1695880.553564] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 197716 previous similar messages [Wed Feb 6 09:18:26 2019][1695880.566914] LustreError: 23741:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.9.103.4@o2ib4 arrived at 1549473682 with bad export cookie 10096477973396257689 [Wed Feb 6 09:18:26 2019][1695880.582649] LustreError: 23741:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 37907 previous similar messages [Wed Feb 6 09:18:26 2019][1695880.620230] LustreError: 17897:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa05312a54b00 x1623667955915024/t0(0) o104->fir-MDT0003@10.9.103.4@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 09:18:34 2019][1695888.621185] LustreError: 17897:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa07424e70900 x1623667956147504/t0(0) o104->fir-MDT0003@10.9.103.4@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 09:18:34 2019][1695888.642366] LustreError: 17897:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 10434 previous similar messages [Wed Feb 6 09:18:56 2019][1695910.501118] LustreError: 17850:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa08a1b3c5a00 x1623667956477120/t0(0) o104->fir-MDT0003@10.9.103.4@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 09:18:56 2019][1695910.522303] LustreError: 17850:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 10836 previous similar messages [Wed Feb 6 09:19:47 2019][1695961.771649] LustreError: 17994:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549473463, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0003_UUID lock: ffffa067dfed0fc0/0x8c1de53948ff7ada lrc: 3/0,1 mode: --/PW res: [0x280000d6c:0x444:0x0].0x0 bits 0x40/0x0 rrc: 169209 type: IBT flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 17994 timeout: 0 lvb_type: 0 [Wed Feb 6 09:19:47 2019][1695961.811693] LustreError: dumping log to /tmp/lustre-log.1549473763.17994 [Wed Feb 6 09:21:26 2019][1696060.564437] LNet: Service thread pid 17885 was inactive for 286.60s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 09:21:26 2019][1696060.581539] Pid: 17885, comm: mdt00_031 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 09:21:26 2019][1696060.591448] Call Trace: [Wed Feb 6 09:21:26 2019][1696060.594086] [] 0xffffffffffffffff [Wed Feb 6 09:21:26 2019][1696060.599289] LustreError: dumping log to /tmp/lustre-log.1549473862.17885 [Wed Feb 6 09:21:28 2019][1696062.500841] Lustre: fir-MDT0003: Client c22e763b-2712-8624-a4bb-1c3145d32fd9 (at 10.8.1.14@o2ib6) reconnecting [Wed Feb 6 09:21:28 2019][1696062.511017] Lustre: Skipped 1033 previous similar messages [Wed Feb 6 09:23:25 2019][1696179.861405] Lustre: 129371:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-353), not sending early reply [Wed Feb 6 09:23:25 2019][1696179.861405] req@ffffa06730ede300 x1624700576351616/t8859504831(0) o36->9f0fc7f1-3017-a377-d5ca-3f45fac7d96c@10.9.106.2@o2ib4:321/0 lens 512/2888 e 16 to 0 dl 1549473986 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:24:13 2019][1696227.308792] Lustre: 126482:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549474021/real 1549474021] req@ffffa089a2a06300 x1623667959184896/t0(0) o6->fir-OST0024-osc-MDT0001@10.0.10.107@o2ib7:28/4 lens 544/432 e 0 to 1 dl 1549474029 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Wed Feb 6 09:24:13 2019][1696227.337180] Lustre: 126482:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 16 previous similar messages [Wed Feb 6 09:24:13 2019][1696227.347278] Lustre: fir-OST0024-osc-MDT0001: Connection to fir-OST0024 (at 10.0.10.107@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 09:24:13 2019][1696227.363514] Lustre: Skipped 16 previous similar messages [Wed Feb 6 09:24:20 2019][1696235.132328] Lustre: 128040:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-3s), not sending early reply. Consider increasing at_early_margin (5)? req@ffffa055b2b3b600 x1624672618612848/t0(0) o103->03d523d0-d33d-b8d0-52c8-1ff235ea28e5@10.9.102.16@o2ib4:368/0 lens 328/0 e 0 to 0 dl 1549474033 ref 2 fl New:H/0/ffffffff rc 0/-1 [Wed Feb 6 09:24:23 2019][1696237.799876] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Wed Feb 6 09:24:23 2019][1696237.810305] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 3 previous similar messages [Wed Feb 6 09:24:23 2019][1696237.820733] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.203@o2ib7 (2): c: 0, oc: 0, rc: 8 [Wed Feb 6 09:24:23 2019][1696237.832975] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 3 previous similar messages [Wed Feb 6 09:24:24 2019][1696238.532915] LustreError: 126978:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0024-osc-MDT0001: cannot cleanup orphans: rc = -11 [Wed Feb 6 09:24:24 2019][1696238.799883] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.203@o2ib7: 1 seconds [Wed Feb 6 09:24:24 2019][1696238.810314] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 85 previous similar messages [Wed Feb 6 09:24:31 2019][1696245.212103] LustreError: 130330:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.102.16@o2ib4: deadline 13:14s ago [Wed Feb 6 09:24:31 2019][1696245.212103] req@ffffa054c3ed7200 x1624672618612592/t0(0) o103->03d523d0-d33d-b8d0-52c8-1ff235ea28e5@10.9.102.16@o2ib4:368/0 lens 328/0 e 0 to 0 dl 1549474033 ref 1 fl Interpret:H/0/ffffffff rc 0/-1 [Wed Feb 6 09:24:31 2019][1696245.244646] Lustre: 130330:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (13:14s); client may timeout. req@ffffa054c3ed7200 x1624672618612592/t0(0) o103->03d523d0-d33d-b8d0-52c8-1ff235ea28e5@10.9.102.16@o2ib4:368/0 lens 328/0 e 0 to 0 dl 1549474033 ref 1 fl Interpret:H/0/ffffffff rc 0/-1 [Wed Feb 6 09:24:31 2019][1696245.274073] Lustre: 130330:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2 previous similar messages [Wed Feb 6 09:24:35 2019][1696249.707644] LNetError: 126428:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Wed Feb 6 09:24:35 2019][1696249.720422] LNetError: 126428:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages [Wed Feb 6 09:25:04 2019][1696278.991520] LNetError: 126423:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Wed Feb 6 09:25:24 2019][1696298.870634] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds [Wed Feb 6 09:25:24 2019][1696298.881063] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 1 previous similar message [Wed Feb 6 09:25:24 2019][1696298.891405] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.202@o2ib7 (0): c: 0, oc: 0, rc: 8 [Wed Feb 6 09:25:24 2019][1696298.903646] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 1 previous similar message [Wed Feb 6 09:25:24 2019][1696299.013937] LNetError: 126436:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Wed Feb 6 09:25:26 2019][1696300.720017] LustreError: 17815:0:(upcall_cache.c:233:upcall_cache_get_entry()) acquire for key 254724: error -110 [Wed Feb 6 09:25:28 2019][1696303.056977] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.202@o2ib7: 5 seconds [Wed Feb 6 09:25:28 2019][1696303.067411] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 10 previous similar messages [Wed Feb 6 09:25:30 2019][1696304.955591] LustreError: 126422:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffffa065bf8d1600 [Wed Feb 6 09:25:31 2019][1696305.471521] LustreError: 130112:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(110766) req@ffffa05d70716c50 x1624732603105680/t0(0) o4->6fa95ecf-4554-3236-796b-9301a5a09ace@10.8.6.24@o2ib6:459/0 lens 488/448 e 3 to 0 dl 1549474124 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:25:31 2019][1696305.496551] Lustre: fir-MDT0003: Bulk IO write error with 6fa95ecf-4554-3236-796b-9301a5a09ace (at 10.8.6.24@o2ib6), client will retry: rc = -110 [Wed Feb 6 09:25:35 2019][1696310.009533] LustreError: 127262:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE req@ffffa05d713e8c50 x1624699768984880/t0(0) o4->2cc850be-dc6d-45a9-7198-5a65d8b1618d@10.9.104.39@o2ib4:520/0 lens 488/448 e 1 to 0 dl 1549474185 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:25:35 2019][1696310.034292] Lustre: fir-MDT0003: Bulk IO write error with 2cc850be-dc6d-45a9-7198-5a65d8b1618d (at 10.9.104.39@o2ib4), client will retry: rc = -110 [Wed Feb 6 09:25:43 2019][1696317.533507] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.9.102.7@o2ib4 ns: mdt-fir-MDT0003_UUID lock: ffffa07b2ba3ad00/0x8c1de53949459b3a lrc: 3/0,0 mode: PR/PR res: [0x280000d4a:0x92:0x0].0x0 bits 0x40/0x0 rrc: 474 type: IBT flags: 0x60000400010020 nid: 10.9.102.7@o2ib4 remote: 0xb09b25d170f5fdab expref: 7608 pid: 127135 timeout: 1696303 lvb_type: 0 [Wed Feb 6 09:25:43 2019][1696317.571848] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 309 previous similar messages [Wed Feb 6 09:25:48 2019][1696322.376815] LustreError: 128085:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa05b04a25000 ns: mdt-fir-MDT0003_UUID lock: ffffa04ef7ed6780/0x8c1de53949a5ea67 lrc: 3/0,0 mode: PR/PR res: [0x280000d42:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT flags: 0x50200000000000 nid: 10.9.102.7@o2ib4 remote: 0xb09b25d170f7f67d expref: 242 pid: 128085 timeout: 0 lvb_type: 0 [Wed Feb 6 09:25:48 2019][1696322.411600] LustreError: 128085:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 4 previous similar messages [Wed Feb 6 09:26:06 2019][1696340.488165] LustreError: 1270:0:(ldlm_lib.c:3273:target_bulk_io()) @@@ truncated bulk READ 0(4096) req@ffffa05b3b461450 x1624699254517296/t0(0) o37->56f157dc-9f05-a349-697a-ac16ba31313e@10.9.104.59@o2ib4:491/0 lens 448/440 e 1 to 0 dl 1549474156 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:26:07 2019][1696341.354925] LustreError: 127264:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(131072) req@ffffa07b382ddc50 x1624673077803248/t0(0) o4->a70355fa-62e2-5007-8b78-0a9448aecdda@10.9.102.59@o2ib4:491/0 lens 488/448 e 2 to 0 dl 1549474156 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:26:07 2019][1696341.380126] Lustre: fir-MDT0001: Bulk IO write error with a70355fa-62e2-5007-8b78-0a9448aecdda (at 10.9.102.59@o2ib4), client will retry: rc = -110 [Wed Feb 6 09:26:08 2019][1696342.533092] LustreError: 127693:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(7452) req@ffffa07b382de050 x1624673476134448/t0(0) o4->e97cf1e5-446b-1854-d0f2-74936bdf8d40@10.9.102.9@o2ib4:485/0 lens 504/448 e 2 to 0 dl 1549474150 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:26:08 2019][1696342.557998] LustreError: 127693:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message [Wed Feb 6 09:26:12 2019][1696346.601930] LustreError: 23728:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.102.7@o2ib4: deadline 38:1s ago [Wed Feb 6 09:26:12 2019][1696346.601930] req@ffffa074d5da8c00 x1624673201861456/t0(0) o103->4882d594-a454-41bf-1d8d-94b894128777@10.9.102.7@o2ib4:482/0 lens 328/0 e 0 to 0 dl 1549474147 ref 1 fl Interpret:H/2/ffffffff rc 0/-1 [Wed Feb 6 09:26:12 2019][1696346.634126] LustreError: 23728:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 4 previous similar messages [Wed Feb 6 09:26:12 2019][1696346.644826] Lustre: 23728:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (38:1s); client may timeout. req@ffffa074d5da8c00 x1624673201861456/t0(0) o103->4882d594-a454-41bf-1d8d-94b894128777@10.9.102.7@o2ib4:482/0 lens 328/0 e 0 to 0 dl 1549474147 ref 1 fl Interpret:H/2/ffffffff rc 0/-1 [Wed Feb 6 09:26:12 2019][1696346.673980] Lustre: 23728:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2 previous similar messages [Wed Feb 6 09:26:14 2019][1696348.533203] LustreError: 126601:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.102.7@o2ib4: deadline 38:3s ago [Wed Feb 6 09:26:14 2019][1696348.533203] req@ffffa075d73d8f00 x1624673201867616/t0(0) o103->4882d594-a454-41bf-1d8d-94b894128777@10.9.102.7@o2ib4:482/0 lens 328/0 e 0 to 0 dl 1549474147 ref 1 fl Interpret:H/2/ffffffff rc 0/-1 [Wed Feb 6 09:26:14 2019][1696348.565484] LustreError: 126601:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 2 previous similar messages [Wed Feb 6 09:26:14 2019][1696348.576268] Lustre: 126601:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (38:3s); client may timeout. req@ffffa075d73d8f00 x1624673201867616/t0(0) o103->4882d594-a454-41bf-1d8d-94b894128777@10.9.102.7@o2ib4:482/0 lens 328/0 e 0 to 0 dl 1549474147 ref 1 fl Interpret:H/2/ffffffff rc 0/-1 [Wed Feb 6 09:26:14 2019][1696348.605509] Lustre: 126601:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2 previous similar messages [Wed Feb 6 09:26:16 2019][1696350.802810] LustreError: 23732:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.102.7@o2ib4: deadline 38:5s ago [Wed Feb 6 09:26:16 2019][1696350.802810] req@ffffa055aca8a100 x1624673201861920/t0(0) o103->4882d594-a454-41bf-1d8d-94b894128777@10.9.102.7@o2ib4:482/0 lens 328/0 e 0 to 0 dl 1549474147 ref 1 fl Interpret:H/2/ffffffff rc 0/-1 [Wed Feb 6 09:26:16 2019][1696350.834994] LustreError: 23732:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [Wed Feb 6 09:26:16 2019][1696350.845698] Lustre: 23732:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (38:5s); client may timeout. req@ffffa055aca8a100 x1624673201861920/t0(0) o103->4882d594-a454-41bf-1d8d-94b894128777@10.9.102.7@o2ib4:482/0 lens 328/0 e 0 to 0 dl 1549474147 ref 1 fl Interpret:H/2/ffffffff rc 0/-1 [Wed Feb 6 09:26:16 2019][1696350.874854] Lustre: 23732:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [Wed Feb 6 09:26:21 2019][1696355.214821] LustreError: 130336:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.102.7@o2ib4: deadline 38:10s ago [Wed Feb 6 09:26:21 2019][1696355.214821] req@ffffa058cee3c500 x1624673201886096/t0(0) o103->4882d594-a454-41bf-1d8d-94b894128777@10.9.102.7@o2ib4:482/0 lens 328/0 e 0 to 0 dl 1549474147 ref 1 fl Interpret:H/2/ffffffff rc 0/-1 [Wed Feb 6 09:26:21 2019][1696355.241677] Lustre: 130322:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (38:10s); client may timeout. req@ffffa058cee38300 x1624673201870912/t0(0) o103->4882d594-a454-41bf-1d8d-94b894128777@10.9.102.7@o2ib4:482/0 lens 328/0 e 0 to 0 dl 1549474147 ref 1 fl Interpret:H/2/ffffffff rc 0/-1 [Wed Feb 6 09:26:21 2019][1696355.241680] Lustre: 130322:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 58 previous similar messages [Wed Feb 6 09:26:21 2019][1696355.286924] LustreError: 130336:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 59 previous similar messages [Wed Feb 6 09:26:23 2019][1696358.044440] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Wed Feb 6 09:26:23 2019][1696358.054876] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.212@o2ib7 (4): c: 0, oc: 1, rc: 8 [Wed Feb 6 09:26:25 2019][1696359.965267] LustreError: 17885:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa05ae94ae900 x1623667959569440/t0(0) o104->fir-MDT0003@10.8.24.5@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 09:26:25 2019][1696359.986364] LustreError: 17885:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [Wed Feb 6 09:26:30 2019][1696364.264442] LNetError: 126430:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Wed Feb 6 09:26:30 2019][1696364.277216] LNetError: 126430:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages [Wed Feb 6 09:26:36 2019][1696370.347311] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.105@o2ib7: 10 seconds [Wed Feb 6 09:26:36 2019][1696370.357831] LNet: 126419:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 4 previous similar messages [Wed Feb 6 09:26:37 2019][1696371.733731] LustreError: 126435:0:(events.c:450:server_bulk_callback()) event type 3, status -103, desc ffffa089ff71f200 [Wed Feb 6 09:26:38 2019][1696372.785769] LustreError: 126672:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE req@ffffa07b416a3450 x1624699537755552/t0(0) o4->23697519-fb1b-8610-edc3-39dba5ac180a@10.9.104.40@o2ib4:540/0 lens 488/448 e 0 to 0 dl 1549474205 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:26:38 2019][1696372.810533] Lustre: fir-MDT0003: Bulk IO write error with 23697519-fb1b-8610-edc3-39dba5ac180a (at 10.9.104.40@o2ib4), client will retry: rc = -110 [Wed Feb 6 09:26:38 2019][1696372.823913] Lustre: Skipped 2 previous similar messages [Wed Feb 6 09:26:43 2019][1696377.340766] Lustre: mdt: This server is not able to keep up with request traffic (cpu-bound). [Wed Feb 6 09:26:43 2019][1696377.349468] Lustre: 129258:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=6, svcEst=600, delay=8246 [Wed Feb 6 09:26:43 2019][1696377.360177] Lustre: 129258:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-4s), not sending early reply. Consider increasing at_early_margin (5)? req@ffffa059933cc200 x1624672932320784/t8859628161(0) o36->d405c92c-c32a-78c2-b321-55f4db7bb864@10.9.102.14@o2ib4:510/0 lens 496/2888 e 20 to 0 dl 1549474175 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:26:43 2019][1696377.392275] Lustre: 129258:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages [Wed Feb 6 09:26:48 2019][1696382.801396] LustreError: 17885:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.102.16@o2ib4) failed to reply to blocking AST (req@ffffa0522421e000 x1623667959473584 status 0 rc -110), evict it ns: mdt-fir-MDT0003_UUID lock: ffffa0792bc3e0c0/0x8c1de53949397002 lrc: 4/0,0 mode: PR/PR res: [0x280000d49:0x66:0x0].0x0 bits 0x40/0x0 rrc: 96744 type: IBT flags: 0x60000400010020 nid: 10.9.102.16@o2ib4 remote: 0xea32b80a2d27ab1f expref: 40045 pid: 17999 timeout: 1696487 lvb_type: 0 [Wed Feb 6 09:26:48 2019][1696382.844943] LustreError: 138-a: fir-MDT0003: A client on nid 10.9.102.16@o2ib4 was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 09:26:48 2019][1696382.857716] LustreError: Skipped 3 previous similar messages [Wed Feb 6 09:26:48 2019][1696382.950657] Lustre: 23732:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (70:5s); client may timeout. req@ffffa0578de37200 x1624705122798048/t0(0) o103->d9d906b5-22c6-448d-c0e5-c1df58bf3870@10.8.31.4@o2ib6:514/0 lens 328/192 e 0 to 0 dl 1549474179 ref 1 fl Complete:H/2/0 rc 0/0 [Wed Feb 6 09:26:48 2019][1696382.979120] Lustre: 23732:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [Wed Feb 6 09:26:49 2019][1696383.238818] LustreError: 12966:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.9.102.16@o2ib4 arrived at 1549474185 with bad export cookie 10096477970670627215 [Wed Feb 6 09:26:50 2019][1696384.215286] LNet: 126419:0:(o2iblnd_cb.c:1484:kiblnd_reconnect_peer()) Abort reconnection of 10.0.10.212@o2ib7: connected [Wed Feb 6 09:26:54 2019][1696389.057416] Lustre: fir-MDT0001: Connection restored to 3de5156d-8e66-984f-011b-183244b66c19 (at 10.8.18.21@o2ib6) [Wed Feb 6 09:26:54 2019][1696389.067936] Lustre: Skipped 1496 previous similar messages [Wed Feb 6 09:27:14 2019][1696409.175638] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Wed Feb 6 09:27:15 2019][1696409.186065] LNetError: 126419:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 4 previous similar messages [Wed Feb 6 09:27:15 2019][1696409.196495] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.202@o2ib7 (4): c: 2, oc: 0, rc: 8 [Wed Feb 6 09:27:15 2019][1696409.208745] LNetError: 126419:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 4 previous similar messages [Wed Feb 6 09:27:23 2019][1696417.827002] LustreError: 18261:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa089ad6e1000 ns: mdt-fir-MDT0003_UUID lock: ffffa08b758e3a80/0x8c1de53949a60eef lrc: 3/0,0 mode: PR/PR res: [0x280000d71:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x50200000000000 nid: 10.9.107.12@o2ib4 remote: 0xf9ff5d7c06d9ff9e expref: 2 pid: 18261 timeout: 0 lvb_type: 0 [Wed Feb 6 09:27:23 2019][1696417.861529] LustreError: 18261:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message [Wed Feb 6 09:27:23 2019][1696417.953251] LustreError: 24582:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.9.101.70@o2ib4 arrived at 1549474219 with bad export cookie 10096477970625601332 [Wed Feb 6 09:27:23 2019][1696417.969055] LustreError: 24582:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 27 previous similar messages [Wed Feb 6 09:27:24 2019][1696418.464246] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [Wed Feb 6 09:27:25 2019][1696419.801336] LNet: 126419:0:(o2iblnd_cb.c:1484:kiblnd_reconnect_peer()) Abort reconnection of 10.0.10.101@o2ib7: connected [Wed Feb 6 09:27:29 2019][1696423.968837] LustreError: 17897:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa0509d155d00 x1623667959681376/t0(0) o104->fir-MDT0003@10.9.101.70@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 09:27:29 2019][1696423.990106] LustreError: 17897:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 2096 previous similar messages [Wed Feb 6 09:27:30 2019][1696424.254404] LustreError: 130203:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(131072) req@ffffa07b416a5c50 x1624673077803248/t0(0) o4->a70355fa-62e2-5007-8b78-0a9448aecdda@10.9.102.59@o2ib4:572/0 lens 488/448 e 0 to 0 dl 1549474237 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Feb 6 09:27:30 2019][1696424.254406] LustreError: 127264:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(131072) req@ffffa07b416a3050 x1624711899394064/t0(0) o4->0edeac5b-ee1e-024f-de97-9e0fc3efb1af@10.8.6.2@o2ib6:581/0 lens 488/448 e 1 to 0 dl 1549474246 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:27:30 2019][1696424.254439] Lustre: fir-MDT0003: Bulk IO write error with 0edeac5b-ee1e-024f-de97-9e0fc3efb1af (at 10.8.6.2@o2ib6), client will retry: rc = -110 [Wed Feb 6 09:27:44 2019][1696438.793477] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [ldlm_bl_11:10652] [Wed Feb 6 09:27:44 2019][1696438.801393] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 09:27:44 2019][1696438.906638] CPU: 9 PID: 10652 Comm: ldlm_bl_11 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 09:27:44 2019][1696438.919053] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 09:27:44 2019][1696438.926795] task: ffffa0516a615140 ti: ffffa05249934000 task.ti: ffffa05249934000 [Wed Feb 6 09:27:44 2019][1696438.934447] RIP: 0010:[] [] ldlm_add_ast_work_item+0x4e/0x3d0 [ptlrpc] [Wed Feb 6 09:27:44 2019][1696438.944413] RSP: 0018:ffffa05249937b80 EFLAGS: 00000286 [Wed Feb 6 09:27:44 2019][1696438.949898] RAX: 0000000001310101 RBX: 000000000001ab80 RCX: ffffa05217bd1920 [Wed Feb 6 09:27:44 2019][1696438.957205] RDX: ffffa05249937ca8 RSI: ffffa0583a3f60c0 RDI: ffffa05a0e8d6c00 [Wed Feb 6 09:27:44 2019][1696438.964510] RBP: ffffa05249937ba0 R08: ffffa05249937ca8 R09: 0000000000000000 [Wed Feb 6 09:27:44 2019][1696438.971816] R10: 000000000000004f R11: ffffa0583a3f60c0 R12: 0000000000000020 [Wed Feb 6 09:27:44 2019][1696438.979121] R13: ffffa04ca948d800 R14: 0000000000000000 R15: ffffffffa72e3ce4 [Wed Feb 6 09:27:44 2019][1696438.986428] FS: 00007fcf13a0f700(0000) GS:ffffa06b7f680000(0000) knlGS:0000000000000000 [Wed Feb 6 09:27:44 2019][1696438.994688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 09:27:44 2019][1696439.000609] CR2: 0000000000412480 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Wed Feb 6 09:27:44 2019][1696439.007915] Call Trace: [Wed Feb 6 09:27:44 2019][1696439.010575] [] ldlm_inodebits_compat_queue+0x198/0x440 [ptlrpc] [Wed Feb 6 09:27:44 2019][1696439.018346] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 09:27:44 2019][1696439.026030] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 09:27:44 2019][1696439.033968] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 09:27:44 2019][1696439.041126] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 09:27:44 2019][1696439.048283] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:27:44 2019][1696439.056747] [] ? lu_context_init+0xd3/0x1f0 [obdclass] [Wed Feb 6 09:27:44 2019][1696439.063731] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 09:27:44 2019][1696439.071928] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 09:27:44 2019][1696439.078978] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:27:44 2019][1696439.084747] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:27:44 2019][1696439.092313] [] kthread+0xd1/0xe0 [Wed Feb 6 09:27:44 2019][1696439.097364] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:27:44 2019][1696439.103633] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:27:44 2019][1696439.110244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:27:44 2019][1696439.116508] Code: 05 45 1d ce ff 01 0f 85 c1 00 00 00 48 8b 43 48 8b 40 1c 85 c0 0f 84 47 02 00 00 4d 85 e4 0f 84 91 01 00 00 4c 8b ab 00 01 00 00 <41> f6 c5 20 75 7f 48 85 db 0f 84 2d 02 00 00 f6 05 06 1d ce ff [Wed Feb 6 09:27:48 2019][1696442.242522] LustreError: 129309:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffffa07b416a1050 x1624673109399440/t0(0) o4->2b3e309f-8d18-a534-0df1-3ea431690a2a@10.9.102.68@o2ib4:605/0 lens 488/448 e 0 to 0 dl 1549474270 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:27:48 2019][1696442.266950] Lustre: fir-MDT0001: Bulk IO write error with 2b3e309f-8d18-a534-0df1-3ea431690a2a (at 10.9.102.68@o2ib4), client will retry: rc = -110 [Wed Feb 6 09:27:48 2019][1696442.280330] Lustre: Skipped 1 previous similar message [Wed Feb 6 09:27:57 2019][1696452.124761] Lustre: fir-MDT0003: Received new LWP connection from 10.0.10.51@o2ib7, removing former export from same NID [Wed Feb 6 09:28:01 2019][1696455.687411] Lustre: fir-MDT0003: Received new LWP connection from 10.0.10.51@o2ib7, removing former export from same NID [Wed Feb 6 09:28:05 2019][1696460.023493] Lustre: fir-MDT0001: Received new LWP connection from 10.0.10.51@o2ib7, removing former export from same NID [Wed Feb 6 09:28:10 2019][1696464.935441] Lustre: fir-MDT0001: Received new LWP connection from 10.0.10.51@o2ib7, removing former export from same NID [Wed Feb 6 09:28:12 2019][1696466.793703] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [ldlm_bl_11:10652] [Wed Feb 6 09:28:12 2019][1696466.801617] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 09:28:12 2019][1696466.906862] CPU: 9 PID: 10652 Comm: ldlm_bl_11 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 09:28:12 2019][1696466.919277] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 09:28:12 2019][1696466.927018] task: ffffa0516a615140 ti: ffffa05249934000 task.ti: ffffa05249934000 [Wed Feb 6 09:28:12 2019][1696466.934670] RIP: 0010:[] [] ldlm_add_ast_work_item+0xd/0x3d0 [ptlrpc] [Wed Feb 6 09:28:12 2019][1696466.944549] RSP: 0018:ffffa05249937ba0 EFLAGS: 00000246 [Wed Feb 6 09:28:12 2019][1696466.950036] RAX: 0000000000b10101 RBX: ffffa05249937ca8 RCX: ffffa05217bd1920 [Wed Feb 6 09:28:12 2019][1696466.957342] RDX: ffffa05249937ca8 RSI: ffffa0583a3f60c0 RDI: ffffa0551bfb1d40 [Wed Feb 6 09:28:12 2019][1696466.964646] RBP: ffffa05249937c08 R08: ffffa05249937ca8 R09: 0000000000000000 [Wed Feb 6 09:28:12 2019][1696466.971954] R10: 000000000000004f R11: ffffa0583a3f60c0 R12: 000000000000004f [Wed Feb 6 09:28:12 2019][1696466.979261] R13: ffffa0583a3f60c0 R14: 000000000001ab80 R15: ffffa05249937ba0 [Wed Feb 6 09:28:12 2019][1696466.986565] FS: 00007fcf13a0f700(0000) GS:ffffa06b7f680000(0000) knlGS:0000000000000000 [Wed Feb 6 09:28:12 2019][1696466.994827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 09:28:12 2019][1696467.000745] CR2: 0000000000412480 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Wed Feb 6 09:28:12 2019][1696467.008051] Call Trace: [Wed Feb 6 09:28:12 2019][1696467.010714] [] ? ldlm_inodebits_compat_queue+0x198/0x440 [ptlrpc] [Wed Feb 6 09:28:12 2019][1696467.018652] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 09:28:12 2019][1696467.026332] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 09:28:12 2019][1696467.034270] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 09:28:12 2019][1696467.041425] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 09:28:12 2019][1696467.048586] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:28:12 2019][1696467.057043] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 09:28:12 2019][1696467.065246] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 09:28:12 2019][1696467.072295] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:28:12 2019][1696467.078073] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:28:12 2019][1696467.085639] [] kthread+0xd1/0xe0 [Wed Feb 6 09:28:12 2019][1696467.090689] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:28:12 2019][1696467.096956] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:28:12 2019][1696467.103569] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:28:12 2019][1696467.109834] Code: 12 00 00 00 04 00 e8 53 dc cb ff 48 c7 c7 00 84 eb c0 e8 c7 75 cb ff 0f 1f 80 00 00 00 00 66 66 66 66 90 55 f6 05 5b 1d ce ff 01 <48> 89 e5 41 56 49 89 d6 41 55 41 54 49 89 f4 53 48 89 fb 74 0d [Wed Feb 6 09:28:20 2019][1696474.257827] LustreError: 40442:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(16856) req@ffffa08b5f453850 x1624699537755552/t0(0) o4->23697519-fb1b-8610-edc3-39dba5ac180a@10.9.104.40@o2ib4:637/0 lens 488/448 e 2 to 0 dl 1549474302 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Feb 6 09:28:20 2019][1696474.282836] Lustre: fir-MDT0003: Bulk IO write error with 23697519-fb1b-8610-edc3-39dba5ac180a (at 10.9.104.40@o2ib4), client will retry: rc = -110 [Wed Feb 6 09:28:23 2019][1696478.110792] INFO: rcu_sched self-detected stall on CPU[1696478.112791] INFO: rcu_sched detected stalls on CPUs/tasks: { 9} (detected by 33, t=60002 jiffies, g=72716853, c=72716852, q=59921) [Wed Feb 6 09:28:23 2019][1696478.112807] Task dump for CPU 9: [Wed Feb 6 09:28:23 2019][1696478.112808] ldlm_bl_11 R running task 0 10652 2 0x00000088 [Wed Feb 6 09:28:23 2019][1696478.112812] Call Trace: [Wed Feb 6 09:28:23 2019][1696478.112864] [] ? ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 09:28:23 2019][1696478.112902] [] ? ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 09:28:23 2019][1696478.112907] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:28:23 2019][1696478.112942] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:28:23 2019][1696478.112945] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:28:23 2019][1696478.112946] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:28:24 2019][1696478.112950] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:28:24 2019][1696478.112951] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:28:24 2019] [Wed Feb 6 09:28:24 2019][1696478.194665] { 9} (t=60086 jiffies g=72716853 c=72716852 q=59922) [Wed Feb 6 09:28:24 2019][1696478.201301] Task dump for CPU 9: [Wed Feb 6 09:28:24 2019][1696478.204707] ldlm_bl_11 R running task 0 10652 2 0x00000088 [Wed Feb 6 09:28:24 2019][1696478.211978] Call Trace: [Wed Feb 6 09:28:24 2019][1696478.214608] [] sched_show_task+0xa8/0x110 [Wed Feb 6 09:28:24 2019][1696478.221073] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 09:28:24 2019][1696478.226648] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 09:28:24 2019][1696478.232739] [] rcu_check_callbacks+0x442/0x730 [Wed Feb 6 09:28:24 2019][1696478.239005] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 09:28:24 2019][1696478.245270] [] update_process_times+0x46/0x80 [Wed Feb 6 09:28:24 2019][1696478.251449] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 09:28:24 2019][1696478.257369] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 09:28:24 2019][1696478.263203] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 09:28:24 2019][1696478.269468] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 09:28:24 2019][1696478.275475] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 09:28:24 2019][1696478.282172] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 09:28:24 2019][1696478.288699] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 09:28:24 2019][1696478.295050] [] ? ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 09:28:24 2019][1696478.303673] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 09:28:24 2019][1696478.311361] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 09:28:24 2019][1696478.319307] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 09:28:24 2019][1696478.326474] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 09:28:24 2019][1696478.333641] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:28:24 2019][1696478.342109] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 09:28:24 2019][1696478.350318] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 09:28:24 2019][1696478.357366] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:28:24 2019][1696478.363145] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:28:24 2019][1696478.370712] [] kthread+0xd1/0xe0 [Wed Feb 6 09:28:24 2019][1696478.375766] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:28:24 2019][1696478.382031] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:28:24 2019][1696478.388643] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:28:52 2019][1696506.794026] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 21s! [ldlm_bl_11:10652] [Wed Feb 6 09:28:52 2019][1696506.801942] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 09:28:52 2019][1696506.907186] CPU: 9 PID: 10652 Comm: ldlm_bl_11 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 09:28:52 2019][1696506.919602] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 09:28:52 2019][1696506.927342] task: ffffa0516a615140 ti: ffffa05249934000 task.ti: ffffa05249934000 [Wed Feb 6 09:28:52 2019][1696506.934996] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 09:28:52 2019][1696506.945490] RSP: 0018:ffffa05249937bb0 EFLAGS: 00000282 [Wed Feb 6 09:28:52 2019][1696506.950977] RAX: 0000000000e90101 RBX: ffffa05217bd1920 RCX: ffffa05217bd1920 [Wed Feb 6 09:28:52 2019][1696506.958283] RDX: ffffa05249937ca8 RSI: ffffa0583a3f60c0 RDI: ffffa068428f3600 [Wed Feb 6 09:28:52 2019][1696506.965588] RBP: ffffa05249937c08 R08: ffffa05249937ca8 R09: 0000000000000000 [Wed Feb 6 09:28:52 2019][1696506.972894] R10: 000000000000004f R11: ffffa0583a3f60c0 R12: ffffa05249937ca8 [Wed Feb 6 09:28:52 2019][1696506.980201] R13: 0000000000000000 R14: 000000000000004f R15: ffffa0583a3f60c0 [Wed Feb 6 09:28:52 2019][1696506.987507] FS: 00007fcf13a0f700(0000) GS:ffffa06b7f680000(0000) knlGS:0000000000000000 [Wed Feb 6 09:28:52 2019][1696506.995767] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 09:28:52 2019][1696507.001686] CR2: 0000000000412480 CR3: 00000027bbe10000 CR4: 00000000003407e0 [Wed Feb 6 09:28:52 2019][1696507.008992] Call Trace: [Wed Feb 6 09:28:52 2019][1696507.011650] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 09:28:52 2019][1696507.019333] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 09:28:52 2019][1696507.027268] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 09:28:52 2019][1696507.034428] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 09:28:52 2019][1696507.041588] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:28:52 2019][1696507.050050] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 09:28:52 2019][1696507.058248] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 09:28:52 2019][1696507.065295] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:28:52 2019][1696507.071064] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:28:52 2019][1696507.078631] [] kthread+0xd1/0xe0 [Wed Feb 6 09:28:52 2019][1696507.083681] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:28:52 2019][1696507.089949] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:28:52 2019][1696507.096562] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:28:52 2019][1696507.102825] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 09:28:56 2019][1696510.836057] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [migration/14:82] [Wed Feb 6 09:28:56 2019][1696510.843971] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses ipmi_si cryptd enclosure ipmi_devintf pcspkr ccp sg i2c_piix4 k10temp ipmi_msghandler dm_multipath acpi_power_meter dm_mod knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) ttm ahci devlink libahci mlx_compat(OE) drm crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas ptp drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 09:28:56 2019][1696510.949213] CPU: 14 PID: 82 Comm: migration/14 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 09:28:56 2019][1696510.961628] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 09:28:56 2019][1696510.969369] task: ffffa06c29918000 ti: ffffa06c29920000 task.ti: ffffa06c29920000 [Wed Feb 6 09:28:56 2019][1696510.977022] RIP: 0010:[] [] multi_cpu_stop+0x4a/0x110 [Wed Feb 6 09:28:56 2019][1696510.985481] RSP: 0000:ffffa06c29923d98 EFLAGS: 00000246 [Wed Feb 6 09:28:56 2019][1696510.990968] RAX: 0000000000000001 RBX: ffffa06b7f8dab80 RCX: dead000000000200 [Wed Feb 6 09:28:56 2019][1696510.998273] RDX: ffffa07b7f6d5ff0 RSI: 0000000000000282 RDI: ffffa05b78fbfab0 [Wed Feb 6 09:28:56 2019][1696511.005579] RBP: ffffa06c29923dc0 R08: ffffa05b78fbfa80 R09: 0000000000000001 [Wed Feb 6 09:28:56 2019][1696511.012885] R10: 000000000000bdb7 R11: 0000000000000001 R12: 0000000000000000 [Wed Feb 6 09:28:56 2019][1696511.020190] R13: 000000000001ab80 R14: ffffa06c29adb000 R15: 000000020000000e [Wed Feb 6 09:28:56 2019][1696511.027497] FS: 00007f06ea4f9880(0000) GS:ffffa07b7f6c0000(0000) knlGS:0000000000000000 [Wed Feb 6 09:28:56 2019][1696511.035757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 09:28:56 2019][1696511.041677] CR2: 00007ffd5943afd8 CR3: 000000103ba82000 CR4: 00000000003407e0 [Wed Feb 6 09:28:56 2019][1696511.048983] Call Trace: [Wed Feb 6 09:28:56 2019][1696511.051612] [] ? cpu_stop_should_run+0x50/0x50 [Wed Feb 6 09:28:56 2019][1696511.057874] [] cpu_stopper_thread+0x99/0x150 [Wed Feb 6 09:28:56 2019][1696511.063968] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:28:56 2019][1696511.069628] [] smpboot_thread_fn+0x144/0x1a0 [Wed Feb 6 09:28:56 2019][1696511.075720] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:28:56 2019][1696511.081725] [] kthread+0xd1/0xe0 [Wed Feb 6 09:28:56 2019][1696511.086778] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:28:56 2019][1696511.093044] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:28:56 2019][1696511.099657] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:28:56 2019][1696511.105923] Code: 66 90 66 90 49 89 c5 48 8b 47 18 48 85 c0 0f 84 b3 00 00 00 0f a3 18 19 db 85 db 41 0f 95 c6 45 31 ff 31 c0 0f 1f 44 00 00 f3 90 <41> 8b 5c 24 20 39 c3 74 5d 83 fb 02 74 68 83 fb 03 75 05 45 84 [Wed Feb 6 09:29:37 2019][1696551.979615] LustreError: 17897:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa057e69e8300 x1623667960520240/t0(0) o104->fir-MDT0003@10.8.31.4@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 09:29:37 2019][1696552.000709] LustreError: 17897:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 5784 previous similar messages [Wed Feb 6 09:33:07 2019][1696762.104624] Lustre: fir-MDT0003: Client 9f0fc7f1-3017-a377-d5ca-3f45fac7d96c (at 10.9.106.2@o2ib4) reconnecting [Wed Feb 6 09:33:07 2019][1696762.114880] Lustre: Skipped 1729 previous similar messages [Wed Feb 6 09:33:32 2019][1696786.585301] Lustre: 127209:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 09:33:32 2019][1696786.585301] req@ffffa04f9f4d3f00 x1624711796972896/t8859817132(0) o36->a1a32c6e-a603-c7ad-bbfa-4137583a5bae@10.8.17.20@o2ib6:173/0 lens 512/2888 e 2 to 0 dl 1549474593 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:33:53 2019][1696807.988148] LustreError: 17897:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa054c4b57200 x1623667963620448/t0(0) o104->fir-MDT0003@10.8.31.4@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 09:33:53 2019][1696808.009243] LustreError: 17897:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 127557 previous similar messages [Wed Feb 6 09:34:05 2019][1696819.312561] LustreError: 126631:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549474321, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0003_UUID lock: ffffa072ba634380/0x8c1de53949a9759d lrc: 3/0,1 mode: --/EX res: [0x280000e26:0x10b:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 126631 timeout: 0 lvb_type: 0 [Wed Feb 6 09:34:07 2019][1696821.383576] LustreError: 127501:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549474323, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0003_UUID lock: ffffa06873b85e80/0x8c1de53949a98747 lrc: 3/1,0 mode: --/PR res: [0x280000d75:0x6:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 127501 timeout: 0 lvb_type: 0 [Wed Feb 6 09:34:31 2019][1696845.977767] LNet: Service thread pid 126631 was inactive for 326.66s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 09:34:31 2019][1696845.994961] Pid: 126631, comm: mdt02_002 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 09:34:31 2019][1696846.004953] Call Trace: [Wed Feb 6 09:34:31 2019][1696846.007584] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:34:31 2019][1696846.014701] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:34:31 2019][1696846.022068] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:34:31 2019][1696846.029069] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:34:31 2019][1696846.036237] [] mdt_reint_object_lock+0x2c/0x60 [mdt] [Wed Feb 6 09:34:31 2019][1696846.043057] [] mdt_reint_striped_lock+0x8c/0x510 [mdt] [Wed Feb 6 09:34:31 2019][1696846.050061] [] mdt_reint_unlink+0x6b2/0x14a0 [mdt] [Wed Feb 6 09:34:31 2019][1696846.056716] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 09:34:31 2019][1696846.062932] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 09:34:31 2019][1696846.069673] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 09:34:31 2019][1696846.075541] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:34:31 2019][1696846.082666] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:34:31 2019][1696846.090553] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:34:31 2019][1696846.097053] [] kthread+0xd1/0xe0 [Wed Feb 6 09:34:31 2019][1696846.102131] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:34:31 2019][1696846.108778] [] 0xffffffffffffffff [Wed Feb 6 09:34:31 2019][1696846.113973] LustreError: dumping log to /tmp/lustre-log.1549474647.126631 [Wed Feb 6 09:34:33 2019][1696848.025792] LNet: Service thread pid 127501 was inactive for 326.64s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 09:34:33 2019][1696848.042981] Pid: 127501, comm: mdt02_011 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 09:34:33 2019][1696848.052973] Call Trace: [Wed Feb 6 09:34:33 2019][1696848.055604] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:34:33 2019][1696848.062721] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:34:33 2019][1696848.070087] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:34:33 2019][1696848.077091] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:34:33 2019][1696848.084257] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:34:33 2019][1696848.091345] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:34:33 2019][1696848.098081] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:34:33 2019][1696848.104737] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:34:33 2019][1696848.111670] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:34:33 2019][1696848.118953] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:34:33 2019][1696848.125285] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:34:33 2019][1696848.132391] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:34:33 2019][1696848.140278] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:34:33 2019][1696848.146780] [] kthread+0xd1/0xe0 [Wed Feb 6 09:34:33 2019][1696848.151866] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:34:33 2019][1696848.158507] [] 0xffffffffffffffff [Wed Feb 6 09:34:34 2019][1696848.163698] LustreError: dumping log to /tmp/lustre-log.1549474649.127501 [Wed Feb 6 09:34:37 2019][1696851.333824] LustreError: 129242:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549474353, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0003_UUID lock: ffffa0653307e0c0/0x8c1de53949acc0c9 lrc: 3/0,1 mode: --/PW res: [0x280000e12:0x3:0x0].0x0 bits 0x2/0x0 rrc: 13 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 129242 timeout: 0 lvb_type: 0 [Wed Feb 6 09:34:41 2019][1696856.097861] LustreError: 17940:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549474357, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0003_UUID lock: ffffa05754f82400/0x8c1de53949ad3df7 lrc: 3/1,0 mode: --/PR res: [0x280000e12:0x3:0x0].0x0 bits 0x13/0x8 rrc: 13 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 17940 timeout: 0 lvb_type: 0 [Wed Feb 6 09:35:08 2019][1696882.995760] LNet: Service thread pid 127501 completed after 361.61s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 09:35:44 2019][1696918.524500] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 159s: evicting client at 10.8.31.4@o2ib6 ns: mdt-fir-MDT0003_UUID lock: ffffa07a80970b40/0x8c1de539483f023c lrc: 3/0,0 mode: PR/PR res: [0x280000e13:0xbc:0x0].0x0 bits 0x40/0x0 rrc: 136550 type: IBT flags: 0x60000400010020 nid: 10.8.31.4@o2ib6 remote: 0xf4816cc354aa539b expref: 136638 pid: 847 timeout: 1696896 lvb_type: 0 [Wed Feb 6 09:35:44 2019][1696918.562848] LustreError: 126609:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 87126 previous similar messages [Wed Feb 6 09:36:55 2019][1696989.296809] Lustre: fir-MDT0003: Connection restored to ba761368-8a01-ecd0-1153-4bf3475f3fd0 (at 10.8.1.25@o2ib6) [Wed Feb 6 09:36:55 2019][1696989.307246] Lustre: Skipped 1015 previous similar messages [Wed Feb 6 09:37:06 2019][1697000.195554] Lustre: fir-MDT0003: haven't heard from client 961b1d3a-e8f6-ab43-e9b3-f245b8a015f0 (at 10.8.10.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa07983723800, cur 1549474802 expire 1549474652 last 1549474575 [Wed Feb 6 09:37:06 2019][1697000.217431] Lustre: Skipped 1 previous similar message [Wed Feb 6 09:38:28 2019][1697082.523687] LNet: Service thread pid 17897 was inactive for 891.11s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 09:38:28 2019][1697082.540799] Pid: 17897, comm: mdt00_035 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 09:38:28 2019][1697082.550711] Call Trace: [Wed Feb 6 09:38:28 2019][1697082.553354] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 09:38:28 2019][1697082.560483] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:38:28 2019][1697082.567859] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Feb 6 09:38:28 2019][1697082.574785] [] mdt_reint_unlink+0x331/0x14a0 [mdt] [Wed Feb 6 09:38:28 2019][1697082.581439] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 09:38:28 2019][1697082.587663] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 09:38:28 2019][1697082.594406] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 09:38:28 2019][1697082.600280] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:38:28 2019][1697082.607412] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:38:28 2019][1697082.615301] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:28 2019][1697082.621810] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1697082.626897] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1697082.633544] [] 0xffffffffffffffff [Wed Feb 6 09:38:28 2019][1697082.638737] LustreError: dumping log to /tmp/lustre-log.1549474884.17897 [Wed Feb 6 09:38:59 2019][1697113.238393] SysRq : Show State [Wed Feb 6 09:38:59 2019][1697113.241657] task PC stack pid father [Wed Feb 6 09:38:59 2019][1697113.247751] systemd S ffffa05c29eb8000 0 1 0 0x00000000 [Wed Feb 6 09:38:59 2019][1697113.255044] Call Trace: [Wed Feb 6 09:38:59 2019][1697113.257681] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697113.262824] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:59 2019][1697113.270053] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Wed Feb 6 09:38:59 2019][1697113.277017] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:59 2019][1697113.283544] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:38:59 2019][1697113.288772] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:59 2019][1697113.294525] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:38:59 2019][1697113.300296] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:59 2019][1697113.306476] kthreadd S ffffa05c29eb9040 0 2 0 0x00000000 [Wed Feb 6 09:38:59 2019][1697113.313745] Call Trace: [Wed Feb 6 09:38:59 2019][1697113.316374] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697113.321524] [] kthreadd+0x2f5/0x300 [Wed Feb 6 09:38:59 2019][1697113.326857] [] ? kthread_create_on_cpu+0x60/0x60 [Wed Feb 6 09:38:59 2019][1697113.333302] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1697113.339943] [] ? kthread_create_on_cpu+0x60/0x60 [Wed Feb 6 09:38:59 2019][1697113.346386] ksoftirqd/0 S ffffa04ca9b60000 0 3 2 0x00000000 [Wed Feb 6 09:38:59 2019][1697113.353654] Call Trace: [Wed Feb 6 09:38:59 2019][1697113.356283] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697113.361426] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:59 2019][1697113.367439] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.373452] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1697113.378507] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.384781] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1697113.391420] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.397687] kworker/0:0H S ffffa04ca9b62080 0 5 2 0x00000000 [Wed Feb 6 09:38:59 2019][1697113.404971] Call Trace: [Wed Feb 6 09:38:59 2019][1697113.407603] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:59 2019][1697113.413789] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697113.418935] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:59 2019][1697113.424682] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:59 2019][1697113.431383] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1697113.436444] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.442711] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1697113.449332] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.455606] migration/0 S ffffa04ca9b65140 0 8 2 0x00000000 [Wed Feb 6 09:38:59 2019][1697113.462883] Call Trace: [Wed Feb 6 09:38:59 2019][1697113.465511] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697113.470651] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:59 2019][1697113.476656] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.482663] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1697113.487716] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.493984] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1697113.500604] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.506879] rcu_bh S ffffa04ca9b66180 0 9 2 0x00000000 [Wed Feb 6 09:38:59 2019][1697113.514156] Call Trace: [Wed Feb 6 09:38:59 2019][1697113.516785] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:59 2019][1697113.522453] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697113.527603] [] rcu_gp_kthread+0xb2/0x710 [Wed Feb 6 09:38:59 2019][1697113.533355] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:59 2019][1697113.539362] [] ? rcu_gp_fqs+0x90/0x90 [Wed Feb 6 09:38:59 2019][1697113.544856] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1697113.549910] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.556184] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1697113.562803] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.569070] rcu_sched S ffffa04ca9ba0000 0 10 2 0x00000000 [Wed Feb 6 09:38:59 2019][1697113.576357] Call Trace: [Wed Feb 6 09:38:59 2019][1697113.578987] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697113.584134] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:38:59 2019][1697113.590150] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:38:59 2019][1697113.596682] [] ? prepare_to_wait+0x56/0x90 [Wed Feb 6 09:38:59 2019][1697113.602602] [] rcu_gp_kthread+0x349/0x710 [Wed Feb 6 09:38:59 2019][1697113.608436] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:59 2019][1697113.614452] [] ? rcu_gp_fqs+0x90/0x90 [Wed Feb 6 09:38:59 2019][1697113.619944] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1697113.624997] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.631265] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1697113.637884] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.644150] lru-add-drain S ffffa04ca9ba1040 0 11 2 0x00000000 [Wed Feb 6 09:38:59 2019][1697113.651424] Call Trace: [Wed Feb 6 09:38:59 2019][1697113.654058] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697113.659205] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:59 2019][1697113.665039] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:59 2019][1697113.671226] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1697113.676287] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.682553] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1697113.689168] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.695442] watchdog/0 S ffffa04ca9ba2080 0 12 2 0x00000000 [Wed Feb 6 09:38:59 2019][1697113.702709] Call Trace: [Wed Feb 6 09:38:59 2019][1697113.705340] [] ? touch_nmi_watchdog+0x30/0x30 [Wed Feb 6 09:38:59 2019][1697113.711528] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:59 2019][1697113.717022] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697113.722170] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:59 2019][1697113.728175] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.734182] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1697113.739244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.745519] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1697113.752138] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.758405] watchdog/1 S ffffa05c2982b0c0 0 13 2 0x00000000 [Wed Feb 6 09:38:59 2019][1697113.765673] Call Trace: [Wed Feb 6 09:38:59 2019][1697113.768301] [] ? touch_nmi_watchdog+0x30/0x30 [Wed Feb 6 09:38:59 2019][1697113.774483] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:59 2019][1697113.779978] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697113.785125] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:59 2019][1697113.791131] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.797138] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1697113.802199] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.808475] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1697113.815093] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.821361] migration/1 S ffffa05c29858000 0 14 2 0x00000000 [Wed Feb 6 09:38:59 2019][1697113.828629] Call Trace: [Wed Feb 6 09:38:59 2019][1697113.831258] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697113.836397] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:59 2019][1697113.842405] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.848419] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1697113.853471] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.859737] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1697113.866349] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.872614] ksoftirqd/1 S ffffa05c29859040 0 15 2 0x00000000 [Wed Feb 6 09:38:59 2019][1697113.879883] Call Trace: [Wed Feb 6 09:38:59 2019][1697113.882513] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697113.887652] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:59 2019][1697113.893659] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.899663] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1697113.904719] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.910993] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1697113.917605] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.923884] kworker/1:0H S ffffa05c2985b0c0 0 17 2 0x00000000 [Wed Feb 6 09:38:59 2019][1697113.931162] Call Trace: [Wed Feb 6 09:38:59 2019][1697113.933796] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:59 2019][1697113.939985] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697113.945130] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:59 2019][1697113.950876] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:59 2019][1697113.957576] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1697113.962631] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.968908] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1697113.975526] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697113.981798] watchdog/2 S ffffa06c29872080 0 19 2 0x00000000 [Wed Feb 6 09:38:59 2019][1697113.989070] Call Trace: [Wed Feb 6 09:38:59 2019][1697113.991698] [] ? touch_nmi_watchdog+0x30/0x30 [Wed Feb 6 09:38:59 2019][1697113.997885] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:59 2019][1697114.003373] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697114.008520] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:59 2019][1697114.014524] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697114.020533] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1697114.025593] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697114.031859] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1697114.038472] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697114.044737] migration/2 S ffffa06c298730c0 0 20 2 0x00000000 [Wed Feb 6 09:38:59 2019][1697114.052007] Call Trace: [Wed Feb 6 09:38:59 2019][1697114.054637] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697114.059784] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:59 2019][1697114.065798] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697114.071805] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1697114.076867] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697114.083141] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1697114.089764] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697114.096037] ksoftirqd/2 S ffffa06c29874100 0 21 2 0x00000000 [Wed Feb 6 09:38:59 2019][1697114.103306] Call Trace: [Wed Feb 6 09:38:59 2019][1697114.105936] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697114.111082] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:59 2019][1697114.117087] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697114.123096] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1697114.128157] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697114.134430] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1697114.141042] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1697114.147310] kworker/2:0H S ffffa06c29899040 0 23 2 0x00000000 [Wed Feb 6 09:38:59 2019][1697114.154585] Call Trace: [Wed Feb 6 09:38:59 2019][1697114.157215] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:59 2019][1697114.163396] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1697114.168544] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:00 2019][1697114.174289] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:00 2019][1697114.180987] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697114.186042] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.192317] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697114.198939] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.205211] watchdog/3 S ffffa07c2941a080 0 25 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697114.212480] Call Trace: [Wed Feb 6 09:39:00 2019][1697114.215111] [] ? touch_nmi_watchdog+0x30/0x30 [Wed Feb 6 09:39:00 2019][1697114.221299] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:00 2019][1697114.226794] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697114.231940] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:00 2019][1697114.237947] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.243953] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697114.249005] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.255271] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697114.261886] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.268158] migration/3 S ffffa07c2941b0c0 0 26 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697114.275426] Call Trace: [Wed Feb 6 09:39:00 2019][1697114.278059] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697114.283207] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:00 2019][1697114.289220] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.295235] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697114.300295] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.306562] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697114.313175] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.319452] ksoftirqd/3 S ffffa07c2941c100 0 27 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697114.326725] Call Trace: [Wed Feb 6 09:39:00 2019][1697114.329355] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697114.334493] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:00 2019][1697114.340500] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.346506] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697114.351559] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.357824] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697114.364439] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.370703] kworker/3:0H S ffffa07c29441040 0 29 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697114.377978] Call Trace: [Wed Feb 6 09:39:00 2019][1697114.380610] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:00 2019][1697114.386791] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697114.391938] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:00 2019][1697114.397686] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:00 2019][1697114.404393] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697114.409453] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.415721] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697114.422342] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.428615] watchdog/4 S ffffa04ca94330c0 0 31 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697114.435885] Call Trace: [Wed Feb 6 09:39:00 2019][1697114.438512] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:00 2019][1697114.443999] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697114.449138] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:00 2019][1697114.455144] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.461151] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697114.466204] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.472478] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697114.479092] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.485365] migration/4 S ffffa04ca9470000 0 32 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697114.492634] Call Trace: [Wed Feb 6 09:39:00 2019][1697114.495263] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697114.500402] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:00 2019][1697114.506410] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.512423] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697114.517476] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.523742] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697114.530356] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.536627] ksoftirqd/4 S ffffa04ca9471040 0 33 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697114.543903] Call Trace: [Wed Feb 6 09:39:00 2019][1697114.546537] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697114.551705] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:00 2019][1697114.557716] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.563721] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697114.568774] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.575042] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697114.581662] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.587930] kworker/4:0 S ffffa04ca9472080 0 34 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697114.595210] Call Trace: [Wed Feb 6 09:39:00 2019][1697114.597843] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:00 2019][1697114.604023] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697114.609162] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:00 2019][1697114.614907] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:00 2019][1697114.621606] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697114.626660] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.632927] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697114.639539] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.645807] kworker/4:0H S ffffa04ca94730c0 0 35 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697114.653086] Call Trace: [Wed Feb 6 09:39:00 2019][1697114.655720] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:00 2019][1697114.661908] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697114.667047] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:00 2019][1697114.672796] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:00 2019][1697114.679500] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697114.684556] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.690828] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697114.697441] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.703708] watchdog/5 S ffffa05c2985e180 0 36 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697114.710977] Call Trace: [Wed Feb 6 09:39:00 2019][1697114.713605] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:00 2019][1697114.719092] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697114.724240] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:00 2019][1697114.730247] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.736258] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697114.741315] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.747588] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697114.754208] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.760474] migration/5 S ffffa05c2985d140 0 37 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697114.767743] Call Trace: [Wed Feb 6 09:39:00 2019][1697114.770374] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697114.775520] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:00 2019][1697114.781527] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.787533] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697114.792587] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.798861] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697114.805484] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.811757] ksoftirqd/5 S ffffa05c298a0000 0 38 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697114.819024] Call Trace: [Wed Feb 6 09:39:00 2019][1697114.821655] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697114.826802] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:00 2019][1697114.832808] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.838815] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697114.843875] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.850142] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697114.856764] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.863037] kworker/5:0H S ffffa05c298a2080 0 40 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697114.870312] Call Trace: [Wed Feb 6 09:39:00 2019][1697114.872945] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:00 2019][1697114.879133] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697114.884280] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:00 2019][1697114.890026] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:00 2019][1697114.896725] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697114.901778] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.908060] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697114.914675] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.920950] watchdog/6 S ffffa06c2989e180 0 41 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697114.928238] Call Trace: [Wed Feb 6 09:39:00 2019][1697114.930865] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:00 2019][1697114.936361] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697114.941509] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:00 2019][1697114.947523] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.953536] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697114.958589] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.964856] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697114.971478] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697114.977752] migration/6 S ffffa06c2989d140 0 42 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697114.985029] Call Trace: [Wed Feb 6 09:39:00 2019][1697114.987660] [] ? cpu_stop_queue_work+0x70/0x70 [Wed Feb 6 09:39:00 2019][1697114.993934] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697114.999081] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:00 2019][1697115.005097] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697115.011110] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697115.016164] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697115.022437] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697115.029051] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697115.035324] ksoftirqd/6 S ffffa06c2989c100 0 43 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697115.042593] Call Trace: [Wed Feb 6 09:39:00 2019][1697115.045221] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697115.050362] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:00 2019][1697115.056367] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697115.062374] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697115.067430] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697115.073703] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697115.080323] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697115.086597] kworker/6:0H S ffffa06c298d8000 0 45 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697115.093871] Call Trace: [Wed Feb 6 09:39:00 2019][1697115.096505] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:00 2019][1697115.102692] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697115.107840] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:00 2019][1697115.113587] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:00 2019][1697115.120295] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1697115.125354] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697115.131624] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1697115.138245] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1697115.144518] watchdog/7 S ffffa07c29446180 0 46 2 0x00000000 [Wed Feb 6 09:39:00 2019][1697115.151787] Call Trace: [Wed Feb 6 09:39:00 2019][1697115.154414] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:00 2019][1697115.159905] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1697115.165051] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:00 2019][1697115.171066] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.177076] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697115.182133] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.188406] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697115.195020] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.201292] migration/7 S ffffa07c29445140 0 47 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697115.208562] Call Trace: [Wed Feb 6 09:39:01 2019][1697115.211193] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697115.216341] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:01 2019][1697115.222353] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.228361] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697115.233423] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.239697] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697115.246318] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.252593] ksoftirqd/7 S ffffa07c29444100 0 48 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697115.259861] Call Trace: [Wed Feb 6 09:39:01 2019][1697115.262490] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697115.267638] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:01 2019][1697115.273651] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.279658] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697115.284712] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.290978] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697115.297601] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.303873] kworker/7:0H S ffffa07c29488000 0 50 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697115.311148] Call Trace: [Wed Feb 6 09:39:01 2019][1697115.313781] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:01 2019][1697115.319969] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697115.325117] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:01 2019][1697115.330871] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:01 2019][1697115.337594] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697115.342649] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.348924] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697115.355538] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.361811] watchdog/8 S ffffa04ca9476180 0 51 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697115.369081] Call Trace: [Wed Feb 6 09:39:01 2019][1697115.371708] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:01 2019][1697115.377203] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697115.382342] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:01 2019][1697115.388350] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.394364] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697115.399426] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.405701] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697115.412322] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.418595] migration/8 S ffffa04ca9475140 0 52 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697115.425865] Call Trace: [Wed Feb 6 09:39:01 2019][1697115.428494] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697115.433642] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:01 2019][1697115.439656] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.445669] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697115.450724] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.456992] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697115.463610] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.469875] ksoftirqd/8 S ffffa04ca9474100 0 53 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697115.477146] Call Trace: [Wed Feb 6 09:39:01 2019][1697115.479777] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697115.484923] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:01 2019][1697115.490931] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.496944] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697115.501996] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.508263] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697115.514884] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.521149] kworker/8:0H S ffffa04ca94c1040 0 55 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697115.528422] Call Trace: [Wed Feb 6 09:39:01 2019][1697115.531058] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:01 2019][1697115.537243] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697115.542384] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:01 2019][1697115.548137] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:01 2019][1697115.554840] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697115.559898] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.566165] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697115.572786] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.579053] watchdog/9 S ffffa05c298a6180 0 56 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697115.586329] Call Trace: [Wed Feb 6 09:39:01 2019][1697115.588958] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:01 2019][1697115.594445] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697115.599586] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:01 2019][1697115.605615] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.611623] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697115.616683] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.622949] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697115.629563] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.635838] migration/9 S ffffa05c298a5140 0 57 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697115.643113] Call Trace: [Wed Feb 6 09:39:01 2019][1697115.645745] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697115.650892] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:01 2019][1697115.656899] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.662912] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697115.667966] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.674241] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697115.680863] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.687136] ksoftirqd/9 S ffffa05c298a4100 0 58 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697115.694403] Call Trace: [Wed Feb 6 09:39:01 2019][1697115.697033] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697115.702172] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:01 2019][1697115.708177] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.714186] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697115.719246] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.725513] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697115.732127] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.738399] kworker/9:0H S ffffa05c298d0000 0 60 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697115.745673] Call Trace: [Wed Feb 6 09:39:01 2019][1697115.748307] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:01 2019][1697115.754496] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697115.759643] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:01 2019][1697115.765398] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:01 2019][1697115.772104] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697115.777158] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.783424] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697115.790038] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.796312] watchdog/10 S ffffa06c298de180 0 61 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697115.803590] Call Trace: [Wed Feb 6 09:39:01 2019][1697115.806217] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:01 2019][1697115.811703] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697115.816844] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:01 2019][1697115.822859] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.828874] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697115.833938] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.840210] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697115.846830] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.853106] migration/10 S ffffa06c298dd140 0 62 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697115.860382] Call Trace: [Wed Feb 6 09:39:01 2019][1697115.863011] [] ? cpu_stop_queue_work+0x70/0x70 [Wed Feb 6 09:39:01 2019][1697115.869288] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697115.874433] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:01 2019][1697115.880441] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.886454] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697115.891508] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.897772] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697115.904385] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.910652] ksoftirqd/10 S ffffa06c298dc100 0 63 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697115.917921] Call Trace: [Wed Feb 6 09:39:01 2019][1697115.920549] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697115.925691] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:01 2019][1697115.931704] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.937708] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697115.942765] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.949037] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697115.955649] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697115.961917] kworker/10:0H S ffffa06c298da080 0 65 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697115.969199] Call Trace: [Wed Feb 6 09:39:01 2019][1697115.971833] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:01 2019][1697115.978020] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697115.983167] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:01 2019][1697115.988913] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:01 2019][1697115.995613] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697116.000675] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697116.006949] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697116.013562] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697116.019837] watchdog/11 S ffffa07c2948e180 0 66 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697116.027113] Call Trace: [Wed Feb 6 09:39:01 2019][1697116.029742] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:01 2019][1697116.035230] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697116.040376] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:01 2019][1697116.046383] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697116.052388] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697116.057441] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697116.063708] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697116.070330] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697116.076603] migration/11 S ffffa07c2948d140 0 67 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697116.083870] Call Trace: [Wed Feb 6 09:39:01 2019][1697116.086504] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697116.091650] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:01 2019][1697116.097664] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697116.103679] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697116.108741] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697116.115017] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697116.121637] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697116.127911] ksoftirqd/11 S ffffa07c2948c100 0 68 2 0x00000000 [Wed Feb 6 09:39:01 2019][1697116.135182] Call Trace: [Wed Feb 6 09:39:01 2019][1697116.137809] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1697116.142949] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:01 2019][1697116.148963] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697116.154969] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1697116.160030] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1697116.166297] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1697116.172917] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.179185] kworker/11:0H S ffffa07c2948a080 0 70 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697116.186465] Call Trace: [Wed Feb 6 09:39:02 2019][1697116.189100] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:02 2019][1697116.195287] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697116.200436] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:02 2019][1697116.206190] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:02 2019][1697116.212896] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697116.217950] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.224217] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697116.230838] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.237114] watchdog/12 S ffffa04ca94c6180 0 71 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697116.244388] Call Trace: [Wed Feb 6 09:39:02 2019][1697116.247019] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:02 2019][1697116.252513] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697116.257652] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:02 2019][1697116.263658] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.269666] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697116.274727] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.281002] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697116.287624] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.293897] migration/12 S ffffa04ca94c5140 0 72 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697116.301166] Call Trace: [Wed Feb 6 09:39:02 2019][1697116.303795] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697116.308943] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:02 2019][1697116.314957] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.320964] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697116.326025] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.332300] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697116.338922] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.345194] ksoftirqd/12 S ffffa04ca94c4100 0 73 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697116.352464] Call Trace: [Wed Feb 6 09:39:02 2019][1697116.355093] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697116.360235] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:02 2019][1697116.366248] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.372262] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697116.377315] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.383591] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697116.390213] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.396486] kworker/12:0H S ffffa04ca94c2080 0 75 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697116.403760] Call Trace: [Wed Feb 6 09:39:02 2019][1697116.406391] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:02 2019][1697116.412571] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697116.417712] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:02 2019][1697116.423467] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:02 2019][1697116.430173] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697116.435225] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.441495] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697116.448113] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.454380] watchdog/13 S ffffa05c298d6180 0 76 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697116.461649] Call Trace: [Wed Feb 6 09:39:02 2019][1697116.464277] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:02 2019][1697116.469763] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697116.474901] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:02 2019][1697116.480910] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.486924] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697116.491986] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.498262] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697116.504882] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.511156] migration/13 S ffffa05c298d5140 0 77 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697116.518425] Call Trace: [Wed Feb 6 09:39:02 2019][1697116.521052] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697116.526194] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:02 2019][1697116.532208] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.538223] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697116.543285] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.549558] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697116.556170] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.562436] ksoftirqd/13 S ffffa05c298d4100 0 78 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697116.569704] Call Trace: [Wed Feb 6 09:39:02 2019][1697116.572351] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697116.577493] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:02 2019][1697116.583507] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.589519] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697116.594574] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.600848] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697116.607460] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.613727] kworker/13:0H S ffffa05c298d2080 0 80 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697116.621000] Call Trace: [Wed Feb 6 09:39:02 2019][1697116.623633] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:02 2019][1697116.629813] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697116.634951] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:02 2019][1697116.640697] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:02 2019][1697116.647398] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697116.652458] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.658725] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697116.665337] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.671603] watchdog/14 S ffffa06c298d9040 0 81 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697116.678872] Call Trace: [Wed Feb 6 09:39:02 2019][1697116.681503] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:02 2019][1697116.686995] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697116.692136] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:02 2019][1697116.698149] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.704157] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697116.709217] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.715485] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697116.722107] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.728381] migration/14 S ffffa06c29918000 0 82 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697116.735658] Call Trace: [Wed Feb 6 09:39:02 2019][1697116.738286] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697116.743435] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:02 2019][1697116.749450] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.755464] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697116.760527] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.766799] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697116.773411] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.779679] ksoftirqd/14 S ffffa06c29919040 0 83 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697116.786957] Call Trace: [Wed Feb 6 09:39:02 2019][1697116.789587] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697116.794734] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:02 2019][1697116.800749] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.806761] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697116.811815] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.818082] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697116.824704] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.830979] kworker/14:0H S ffffa06c2991b0c0 0 85 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697116.838259] Call Trace: [Wed Feb 6 09:39:02 2019][1697116.840892] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:02 2019][1697116.847073] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697116.852220] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:02 2019][1697116.857973] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:02 2019][1697116.864674] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697116.869736] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.876011] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697116.882632] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.888909] watchdog/15 S ffffa07c29489040 0 86 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697116.896182] Call Trace: [Wed Feb 6 09:39:02 2019][1697116.898813] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:02 2019][1697116.904308] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697116.909455] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:02 2019][1697116.915461] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.921467] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697116.926519] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.932787] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697116.939407] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.945673] migration/15 S ffffa07c294f8000 0 87 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697116.952946] Call Trace: [Wed Feb 6 09:39:02 2019][1697116.955581] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697116.960728] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:02 2019][1697116.966743] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.972759] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697116.977819] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.984095] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697116.990716] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697116.996991] ksoftirqd/15 S ffffa07c294f9040 0 88 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697117.004267] Call Trace: [Wed Feb 6 09:39:02 2019][1697117.006894] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697117.012034] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:02 2019][1697117.018042] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697117.024056] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697117.029118] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697117.035391] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697117.042004] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697117.048271] kworker/15:0H S ffffa07c294fb0c0 0 90 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697117.055553] Call Trace: [Wed Feb 6 09:39:02 2019][1697117.058185] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:02 2019][1697117.064365] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697117.069513] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:02 2019][1697117.075268] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:02 2019][1697117.081967] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697117.087028] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697117.093295] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697117.099909] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697117.106184] watchdog/16 S ffffa04ca9550000 0 91 2 0x00000000 [Wed Feb 6 09:39:02 2019][1697117.113460] Call Trace: [Wed Feb 6 09:39:02 2019][1697117.116091] [] ? watchdog_cleanup+0x10/0x10 [Wed Feb 6 09:39:02 2019][1697117.122101] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:02 2019][1697117.127588] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1697117.132730] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:02 2019][1697117.138744] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697117.144758] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1697117.149811] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697117.156080] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1697117.162699] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1697117.168974] migration/16 S ffffa04ca9551040 0 92 2 0x00000000 [Wed Feb 6 09:39:03 2019][1697117.176260] Call Trace: [Wed Feb 6 09:39:03 2019][1697117.178890] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1697117.184038] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:03 2019][1697117.190051] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.196066] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1697117.201128] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.207402] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1697117.214014] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.220278] ksoftirqd/16 S ffffa04ca9552080 0 93 2 0x00000000 [Wed Feb 6 09:39:03 2019][1697117.227547] Call Trace: [Wed Feb 6 09:39:03 2019][1697117.230177] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1697117.235317] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:03 2019][1697117.241321] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.247330] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1697117.252390] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.258666] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1697117.265288] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.271561] kworker/16:0H S ffffa04ca9554100 0 95 2 0x00000000 [Wed Feb 6 09:39:03 2019][1697117.278836] Call Trace: [Wed Feb 6 09:39:03 2019][1697117.281469] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:03 2019][1697117.287656] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1697117.292806] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:03 2019][1697117.298560] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:03 2019][1697117.305267] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1697117.310327] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.316594] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1697117.323207] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.329473] watchdog/17 S ffffa05c298d1040 0 96 2 0x00000000 [Wed Feb 6 09:39:03 2019][1697117.336742] Call Trace: [Wed Feb 6 09:39:03 2019][1697117.339372] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:03 2019][1697117.344864] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1697117.350005] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:03 2019][1697117.356012] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.362023] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1697117.367078] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.373346] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1697117.379966] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.386232] migration/17 S ffffa05c29938000 0 97 2 0x00000000 [Wed Feb 6 09:39:03 2019][1697117.393508] Call Trace: [Wed Feb 6 09:39:03 2019][1697117.396138] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1697117.401285] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:03 2019][1697117.407292] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.413299] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1697117.418360] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.424634] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1697117.431247] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.437513] ksoftirqd/17 S ffffa05c29939040 0 98 2 0x00000000 [Wed Feb 6 09:39:03 2019][1697117.444782] Call Trace: [Wed Feb 6 09:39:03 2019][1697117.447410] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1697117.452551] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:03 2019][1697117.458566] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.464578] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1697117.469632] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.475901] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1697117.482518] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.488786] kworker/17:0H S ffffa05c2993b0c0 0 100 2 0x00000000 [Wed Feb 6 09:39:03 2019][1697117.496060] Call Trace: [Wed Feb 6 09:39:03 2019][1697117.498691] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:03 2019][1697117.504872] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1697117.510019] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:03 2019][1697117.515773] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:03 2019][1697117.522473] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1697117.527526] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.533792] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1697117.540406] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.546681] watchdog/18 S ffffa06c2991e180 0 101 2 0x00000000 [Wed Feb 6 09:39:03 2019][1697117.553957] Call Trace: [Wed Feb 6 09:39:03 2019][1697117.556587] [] ? touch_nmi_watchdog+0x30/0x30 [Wed Feb 6 09:39:03 2019][1697117.562775] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:03 2019][1697117.568269] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1697117.573432] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:03 2019][1697117.579440] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.585447] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1697117.590506] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.596774] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1697117.603388] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.609662] migration/18 S ffffa06c2991d140 0 102 2 0x00000000 [Wed Feb 6 09:39:03 2019][1697117.616939] Call Trace: [Wed Feb 6 09:39:03 2019][1697117.619570] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1697117.624715] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:03 2019][1697117.630723] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.636737] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1697117.641798] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.648073] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1697117.654696] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.660969] ksoftirqd/18 S ffffa06c2991c100 0 103 2 0x00000000 [Wed Feb 6 09:39:03 2019][1697117.668255] Call Trace: [Wed Feb 6 09:39:03 2019][1697117.670886] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1697117.676033] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:03 2019][1697117.682046] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.688061] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1697117.693114] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.699390] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1697117.706010] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.712276] kworker/18:0H S ffffa06c29951040 0 105 2 0x00000000 [Wed Feb 6 09:39:03 2019][1697117.719548] Call Trace: [Wed Feb 6 09:39:03 2019][1697117.722184] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:03 2019][1697117.728369] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1697117.733509] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:03 2019][1697117.739256] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:03 2019][1697117.745963] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1697117.751016] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.757282] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1697117.763896] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.770170] watchdog/19 S ffffa07c294fe180 0 106 2 0x00000000 [Wed Feb 6 09:39:03 2019][1697117.777437] Call Trace: [Wed Feb 6 09:39:03 2019][1697117.780067] [] ? touch_nmi_watchdog+0x30/0x30 [Wed Feb 6 09:39:03 2019][1697117.786256] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:03 2019][1697117.791752] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1697117.796899] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:03 2019][1697117.802912] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.808918] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1697117.813971] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.820240] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1697117.826860] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.833134] migration/19 S ffffa07c294fd140 0 107 2 0x00000000 [Wed Feb 6 09:39:03 2019][1697117.840403] Call Trace: [Wed Feb 6 09:39:03 2019][1697117.843033] [] ? cpu_stop_unpark+0x50/0x50 [Wed Feb 6 09:39:03 2019][1697117.848960] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1697117.854108] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:03 2019][1697117.860114] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.866120] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1697117.871173] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.877448] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1697117.884068] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1697117.890333] ksoftirqd/19 S ffffa07c294fc100 0 108 2 0x00000000 [Wed Feb 6 09:39:03 2019][1697117.897603] Call Trace: [Wed Feb 6 09:39:03 2019][1697117.900233] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1697117.905380] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697117.911386] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697117.917392] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697117.922445] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697117.928712] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697117.935335] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697117.941608] kworker/19:0H S ffffa07c29531040 0 110 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697117.948890] Call Trace: [Wed Feb 6 09:39:04 2019][1697117.951522] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:04 2019][1697117.957701] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697117.962840] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:04 2019][1697117.968587] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:04 2019][1697117.975287] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697117.980350] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697117.986623] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697117.993244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697117.999518] watchdog/20 S ffffa04ca9556180 0 111 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.006787] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.009414] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:04 2019][1697118.014905] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.020049] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697118.026057] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.032073] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.037134] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.043407] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.050020] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.056295] migration/20 S ffffa04ca9555140 0 112 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.063564] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.066192] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.071332] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697118.077346] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.083353] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.088415] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.094689] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.101302] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.107576] ksoftirqd/20 S ffffa04ca95a0000 0 113 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.114844] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.117476] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.122623] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697118.128637] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.134652] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.139712] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.145987] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.152602] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.158875] kworker/20:0H S ffffa04ca95a2080 0 115 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.166147] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.168781] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:04 2019][1697118.174961] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.180109] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:04 2019][1697118.185866] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:04 2019][1697118.192571] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.197623] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.203889] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.210504] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.216777] watchdog/21 S ffffa05c2993e180 0 116 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.224046] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.226675] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:04 2019][1697118.232162] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.237308] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697118.243313] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.249323] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.254384] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.260657] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.267273] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.273548] migration/21 S ffffa05c2993d140 0 117 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.280821] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.283450] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.288590] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697118.294598] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.300610] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.305666] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.311937] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.318553] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.324826] ksoftirqd/21 S ffffa05c2993c100 0 118 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.332095] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.334724] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.339863] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697118.345869] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.351874] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.356927] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.363194] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.369807] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.376073] kworker/21:0H S ffffa05c29969040 0 120 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.383347] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.385980] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:04 2019][1697118.392169] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.397314] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:04 2019][1697118.403061] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:04 2019][1697118.409760] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.414814] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.421090] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.427711] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.433984] watchdog/22 S ffffa06c29956180 0 121 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.441263] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.443889] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:04 2019][1697118.449377] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.454524] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697118.460530] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.466536] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.471590] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.477857] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.484476] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.490742] migration/22 S ffffa06c29955140 0 122 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.498011] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.500642] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.505787] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697118.511793] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.517801] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.522854] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.529129] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.535751] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.542026] ksoftirqd/22 S ffffa06c29954100 0 123 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.549301] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.551932] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.557079] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697118.563092] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.569099] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.574153] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.580429] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.587048] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.593315] kworker/22:0H S ffffa06c29952080 0 125 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.600587] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.603223] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:04 2019][1697118.609410] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.614557] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:04 2019][1697118.620312] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:04 2019][1697118.627019] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.632072] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.638339] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.644962] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.651234] watchdog/23 S ffffa07c29536180 0 126 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.658504] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.661132] [] ? local_clock+0x25/0x30 [Wed Feb 6 09:39:04 2019][1697118.666705] [] ? running_clock+0x9/0x10 [Wed Feb 6 09:39:04 2019][1697118.672371] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:04 2019][1697118.677858] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.682997] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697118.689006] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.695019] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.700071] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.706340] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.712966] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.719233] migration/23 S ffffa07c29535140 0 127 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.726504] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.729132] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.734281] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697118.740295] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.746310] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.751370] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.757638] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.764259] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.770534] ksoftirqd/23 S ffffa07c29534100 0 128 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.777808] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.780439] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.785585] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697118.791594] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.797607] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.802661] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.808935] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.815548] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.821812] kworker/23:0H S ffffa07c29532080 0 130 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.829086] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.831720] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:04 2019][1697118.837901] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.843048] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:04 2019][1697118.848803] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:04 2019][1697118.855511] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.860570] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.866838] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.873459] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.879725] watchdog/24 R running task 0 131 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.886992] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.889624] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:04 2019][1697118.895119] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.900265] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697118.906271] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.912277] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.917330] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.923611] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.930227] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.936502] migration/24 R running task 0 132 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.943779] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.946408] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697118.951548] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697118.957560] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.963567] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697118.968621] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.974888] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697118.981509] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697118.987783] ksoftirqd/24 S ffffa04ca95a4100 0 133 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697118.995050] Call Trace: [Wed Feb 6 09:39:04 2019][1697118.997681] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697119.002828] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697119.008834] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697119.014841] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697119.019902] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697119.026169] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697119.032790] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697119.039064] kworker/24:0H S ffffa04ca95e8000 0 135 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697119.046338] Call Trace: [Wed Feb 6 09:39:04 2019][1697119.048971] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:04 2019][1697119.055159] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697119.060307] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:04 2019][1697119.066054] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:04 2019][1697119.072760] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697119.077813] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697119.084080] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697119.090702] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697119.096976] watchdog/25 S ffffa05c2996e180 0 136 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697119.104264] Call Trace: [Wed Feb 6 09:39:04 2019][1697119.106892] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:04 2019][1697119.112384] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697119.117525] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697119.123539] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697119.129547] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1697119.134608] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697119.140881] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1697119.147495] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1697119.153769] migration/25 S ffffa05c2996d140 0 137 2 0x00000000 [Wed Feb 6 09:39:04 2019][1697119.161036] Call Trace: [Wed Feb 6 09:39:04 2019][1697119.163666] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1697119.168805] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:04 2019][1697119.174811] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.180816] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697119.185870] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.192137] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697119.198759] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.205031] ksoftirqd/25 S ffffa05c2996c100 0 138 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697119.212302] Call Trace: [Wed Feb 6 09:39:05 2019][1697119.214931] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697119.220079] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:05 2019][1697119.226095] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.232108] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697119.237171] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.243444] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697119.250058] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.256331] kworker/25:0H S ffffa05c2996a080 0 140 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697119.263604] Call Trace: [Wed Feb 6 09:39:05 2019][1697119.266239] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:05 2019][1697119.272427] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697119.277576] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:05 2019][1697119.283330] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:05 2019][1697119.290038] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697119.295097] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.301364] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697119.307985] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.314253] watchdog/26 S ffffa06c299b0000 0 141 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697119.321527] Call Trace: [Wed Feb 6 09:39:05 2019][1697119.324157] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:05 2019][1697119.329645] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697119.334794] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:05 2019][1697119.340806] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.346813] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697119.351866] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.358140] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697119.364764] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.371036] migration/26 S ffffa06c299b1040 0 142 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697119.378304] Call Trace: [Wed Feb 6 09:39:05 2019][1697119.380934] [] ? cpu_stop_queue_work+0x70/0x70 [Wed Feb 6 09:39:05 2019][1697119.387209] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697119.392358] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:05 2019][1697119.398372] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.404386] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697119.409440] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.415714] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697119.422334] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.428601] ksoftirqd/26 S ffffa06c299b2080 0 143 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697119.435869] Call Trace: [Wed Feb 6 09:39:05 2019][1697119.438499] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697119.443647] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:05 2019][1697119.449662] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.455677] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697119.460739] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.467013] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697119.473634] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.479908] kworker/26:0H S ffffa06c299b4100 0 145 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697119.487182] Call Trace: [Wed Feb 6 09:39:05 2019][1697119.489816] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:05 2019][1697119.496003] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697119.501142] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:05 2019][1697119.506897] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:05 2019][1697119.513595] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697119.518648] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.524915] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697119.531527] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.537792] watchdog/27 S ffffa07c29578000 0 146 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697119.545060] Call Trace: [Wed Feb 6 09:39:05 2019][1697119.547693] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:05 2019][1697119.553187] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697119.558333] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:05 2019][1697119.564363] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.570372] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697119.575425] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.581692] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697119.588313] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.594586] migration/27 S ffffa07c29579040 0 147 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697119.601854] Call Trace: [Wed Feb 6 09:39:05 2019][1697119.604485] [] ? cpu_stop_queue_work+0x70/0x70 [Wed Feb 6 09:39:05 2019][1697119.610760] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697119.615909] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:05 2019][1697119.621922] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.627930] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697119.632991] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.639265] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697119.645876] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.652144] ksoftirqd/27 S ffffa07c2957a080 0 148 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697119.659419] Call Trace: [Wed Feb 6 09:39:05 2019][1697119.662050] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697119.667188] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:05 2019][1697119.673196] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.679209] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697119.684261] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.690529] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697119.697151] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.703424] kworker/27:0H S ffffa07c2957c100 0 150 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697119.710698] Call Trace: [Wed Feb 6 09:39:05 2019][1697119.713332] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:05 2019][1697119.719520] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697119.724667] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:05 2019][1697119.730414] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:05 2019][1697119.737121] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697119.742174] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.748450] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697119.755071] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.761345] watchdog/28 S ffffa04ca95ee180 0 151 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697119.768613] Call Trace: [Wed Feb 6 09:39:05 2019][1697119.771242] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:05 2019][1697119.776729] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697119.781876] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:05 2019][1697119.787883] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.793896] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697119.798949] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.805217] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697119.811839] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.818112] migration/28 S ffffa04ca95ed140 0 152 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697119.825381] Call Trace: [Wed Feb 6 09:39:05 2019][1697119.828009] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697119.833149] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:05 2019][1697119.839156] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.845169] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697119.850222] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.856489] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697119.863109] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.869377] ksoftirqd/28 S ffffa04ca95ec100 0 153 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697119.876652] Call Trace: [Wed Feb 6 09:39:05 2019][1697119.879282] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697119.884423] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:05 2019][1697119.890437] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.896451] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697119.901504] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.907779] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697119.914400] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.920667] kworker/28:0H S ffffa04ca95ea080 0 155 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697119.927946] Call Trace: [Wed Feb 6 09:39:05 2019][1697119.930572] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:05 2019][1697119.936751] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697119.941891] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:05 2019][1697119.947638] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:05 2019][1697119.954346] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697119.959406] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.965672] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697119.972285] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697119.978552] watchdog/29 S ffffa05c299b8000 0 156 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697119.985830] Call Trace: [Wed Feb 6 09:39:05 2019][1697119.988457] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:05 2019][1697119.993943] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697119.999082] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:05 2019][1697120.005092] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697120.011104] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697120.016157] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697120.022422] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697120.029036] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697120.035311] migration/29 S ffffa05c299b9040 0 157 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697120.042587] Call Trace: [Wed Feb 6 09:39:05 2019][1697120.045215] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697120.050355] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:05 2019][1697120.056363] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697120.062375] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697120.067430] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697120.073703] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697120.080316] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697120.086584] ksoftirqd/29 S ffffa05c299ba080 0 158 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697120.093859] Call Trace: [Wed Feb 6 09:39:05 2019][1697120.096486] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697120.101628] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:05 2019][1697120.107635] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697120.113650] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697120.118710] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697120.124978] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1697120.131599] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1697120.137871] kworker/29:0H S ffffa05c299bc100 0 160 2 0x00000000 [Wed Feb 6 09:39:05 2019][1697120.145145] Call Trace: [Wed Feb 6 09:39:05 2019][1697120.147779] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:05 2019][1697120.153959] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1697120.159107] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:05 2019][1697120.164862] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:05 2019][1697120.171570] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1697120.176630] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.182898] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697120.189517] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.194102] LNet: Service thread pid 17885 completed after 1346.22s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 09:39:06 2019][1697120.194104] LNet: Skipped 1 previous similar message [Wed Feb 6 09:39:06 2019][1697120.217327] watchdog/30 S ffffa06c299b6180 0 161 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697120.224598] Call Trace: [Wed Feb 6 09:39:06 2019][1697120.227229] [] ? touch_nmi_watchdog+0x30/0x30 [Wed Feb 6 09:39:06 2019][1697120.233415] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:06 2019][1697120.238902] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697120.244051] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:06 2019][1697120.250065] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.256078] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697120.261131] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.267397] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697120.274012] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.280285] migration/30 S ffffa06c299b5140 0 162 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697120.287555] Call Trace: [Wed Feb 6 09:39:06 2019][1697120.290182] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697120.295322] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:06 2019][1697120.301337] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.307353] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697120.312413] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.318679] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697120.325292] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.331567] ksoftirqd/30 S ffffa06c299e8000 0 163 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697120.338836] Call Trace: [Wed Feb 6 09:39:06 2019][1697120.341465] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697120.346611] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:06 2019][1697120.352618] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.358626] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697120.363685] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.369952] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697120.376563] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.382829] kworker/30:0 S ffffa06c299e9040 0 164 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697120.390102] Call Trace: [Wed Feb 6 09:39:06 2019][1697120.392727] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:06 2019][1697120.398908] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697120.404056] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:06 2019][1697120.409811] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:06 2019][1697120.416517] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697120.421571] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.427836] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697120.434449] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.440715] kworker/30:0H S ffffa06c299ea080 0 165 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697120.447988] Call Trace: [Wed Feb 6 09:39:06 2019][1697120.450624] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:06 2019][1697120.456812] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697120.461960] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:06 2019][1697120.467712] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:06 2019][1697120.474411] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697120.479464] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.485731] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697120.492344] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.498609] watchdog/31 S ffffa07c2957e180 0 166 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697120.505877] Call Trace: [Wed Feb 6 09:39:06 2019][1697120.508507] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:06 2019][1697120.513992] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697120.519131] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:06 2019][1697120.525138] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.531143] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697120.536199] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.542473] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697120.549094] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.555368] migration/31 S ffffa07c2957d140 0 167 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697120.562637] Call Trace: [Wed Feb 6 09:39:06 2019][1697120.565265] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697120.570406] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:06 2019][1697120.576420] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.582440] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697120.587497] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.593763] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697120.600382] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.606650] ksoftirqd/31 S ffffa07c295a8000 0 168 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697120.613917] Call Trace: [Wed Feb 6 09:39:06 2019][1697120.616547] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697120.621687] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:06 2019][1697120.627702] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.633708] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697120.638759] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.645026] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697120.651639] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.657905] kworker/31:0H S ffffa07c295aa080 0 170 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697120.665178] Call Trace: [Wed Feb 6 09:39:06 2019][1697120.667811] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:06 2019][1697120.673991] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697120.679128] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:06 2019][1697120.684877] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:06 2019][1697120.691576] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697120.696630] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.702904] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697120.709525] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.715790] watchdog/32 S ffffa04ca95e9040 0 171 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697120.723059] Call Trace: [Wed Feb 6 09:39:06 2019][1697120.725689] [] ? touch_nmi_watchdog+0x30/0x30 [Wed Feb 6 09:39:06 2019][1697120.731875] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:06 2019][1697120.737362] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697120.742502] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:06 2019][1697120.748516] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.754522] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697120.759575] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.765840] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697120.772453] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.778719] migration/32 S ffffa04ca9670000 0 172 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697120.785988] Call Trace: [Wed Feb 6 09:39:06 2019][1697120.788617] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697120.793758] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:06 2019][1697120.799772] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.805787] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697120.810849] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.817123] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697120.823744] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.830010] ksoftirqd/32 S ffffa04ca9671040 0 173 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697120.837278] Call Trace: [Wed Feb 6 09:39:06 2019][1697120.839905] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697120.845047] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:06 2019][1697120.851062] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.857066] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697120.862121] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.868386] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697120.874999] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.881266] kworker/32:0H S ffffa04ca96730c0 0 175 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697120.888537] Call Trace: [Wed Feb 6 09:39:06 2019][1697120.891162] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:06 2019][1697120.897341] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697120.902480] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:06 2019][1697120.908227] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:06 2019][1697120.914928] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697120.919988] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.926256] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697120.932877] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.939150] watchdog/33 S ffffa05c299be180 0 176 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697120.946438] Call Trace: [Wed Feb 6 09:39:06 2019][1697120.949065] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:06 2019][1697120.954551] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697120.959692] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:06 2019][1697120.965705] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.971711] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697120.976765] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.983031] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697120.989653] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697120.995929] migration/33 S ffffa05c299bd140 0 177 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697121.003204] Call Trace: [Wed Feb 6 09:39:06 2019][1697121.005835] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697121.010981] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:06 2019][1697121.016988] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697121.023002] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697121.028055] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697121.034320] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697121.040934] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697121.047206] ksoftirqd/33 S ffffa05c29a10000 0 178 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697121.054478] Call Trace: [Wed Feb 6 09:39:06 2019][1697121.057105] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697121.062246] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:06 2019][1697121.068261] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697121.074276] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697121.079335] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697121.085602] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697121.092215] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697121.098481] kworker/33:0H S ffffa05c29a12080 0 180 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697121.105754] Call Trace: [Wed Feb 6 09:39:06 2019][1697121.108387] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:06 2019][1697121.114567] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697121.119705] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:06 2019][1697121.125453] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:06 2019][1697121.132161] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1697121.137222] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697121.143495] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1697121.150110] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1697121.156383] watchdog/34 S ffffa06c299ee180 0 181 2 0x00000000 [Wed Feb 6 09:39:06 2019][1697121.163651] Call Trace: [Wed Feb 6 09:39:06 2019][1697121.166281] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:06 2019][1697121.171767] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1697121.176906] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:07 2019][1697121.182914] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.188927] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697121.193983] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.200254] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697121.206868] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.213143] migration/34 S ffffa06c299ed140 0 182 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697121.220411] Call Trace: [Wed Feb 6 09:39:07 2019][1697121.223041] [] ? cpu_stop_queue_work+0x70/0x70 [Wed Feb 6 09:39:07 2019][1697121.229316] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697121.234463] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:07 2019][1697121.240478] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.246482] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697121.251536] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.257802] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697121.264413] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.270679] ksoftirqd/34 S ffffa06c299ec100 0 183 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697121.277949] Call Trace: [Wed Feb 6 09:39:07 2019][1697121.280577] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697121.285718] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:07 2019][1697121.291723] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.297730] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697121.302784] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.309058] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697121.315672] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.321945] kworker/34:0H S ffffa06c29a18000 0 185 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697121.329219] Call Trace: [Wed Feb 6 09:39:07 2019][1697121.331852] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:07 2019][1697121.338038] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697121.343177] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:07 2019][1697121.348924] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:07 2019][1697121.355624] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697121.360675] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.366943] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697121.373556] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.379820] watchdog/35 S ffffa07c295ae180 0 186 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697121.387089] Call Trace: [Wed Feb 6 09:39:07 2019][1697121.389719] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:07 2019][1697121.395204] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697121.400343] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:07 2019][1697121.406351] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.412357] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697121.417411] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.423683] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697121.430298] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.436572] migration/35 S ffffa07c295ad140 0 187 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697121.443841] Call Trace: [Wed Feb 6 09:39:07 2019][1697121.446468] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697121.451609] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:07 2019][1697121.457624] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.463639] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697121.468700] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.474967] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697121.481588] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.487861] ksoftirqd/35 S ffffa07c295ac100 0 188 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697121.495130] Call Trace: [Wed Feb 6 09:39:07 2019][1697121.497757] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697121.502899] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:07 2019][1697121.508913] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.514920] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697121.519973] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.526246] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697121.532859] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.539125] kworker/35:0H S ffffa07c295e0000 0 190 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697121.546400] Call Trace: [Wed Feb 6 09:39:07 2019][1697121.549034] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:07 2019][1697121.555220] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697121.560359] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:07 2019][1697121.566106] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:07 2019][1697121.572821] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697121.577875] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.584149] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697121.590762] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.597029] watchdog/36 S ffffa04ca9676180 0 191 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697121.604304] Call Trace: [Wed Feb 6 09:39:07 2019][1697121.606935] [] ? watchdog_cleanup+0x10/0x10 [Wed Feb 6 09:39:07 2019][1697121.612941] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:07 2019][1697121.618428] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697121.623575] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:07 2019][1697121.629580] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.635585] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697121.640639] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.646907] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697121.653527] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.659792] migration/36 S ffffa04ca9675140 0 192 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697121.667062] Call Trace: [Wed Feb 6 09:39:07 2019][1697121.669691] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697121.674840] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:07 2019][1697121.680854] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.686859] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697121.691913] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.698188] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697121.704809] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.711084] ksoftirqd/36 S ffffa04ca9674100 0 193 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697121.718360] Call Trace: [Wed Feb 6 09:39:07 2019][1697121.720990] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697121.726137] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:07 2019][1697121.732142] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.738151] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697121.743211] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.749478] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697121.756099] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.762364] kworker/36:0H S ffffa04ca96c1040 0 195 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697121.769638] Call Trace: [Wed Feb 6 09:39:07 2019][1697121.772271] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:07 2019][1697121.778450] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697121.783590] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:07 2019][1697121.789336] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:07 2019][1697121.796035] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697121.801088] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.807355] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697121.813977] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.820248] watchdog/37 S ffffa05c29a16180 0 196 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697121.827520] Call Trace: [Wed Feb 6 09:39:07 2019][1697121.830148] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:07 2019][1697121.835643] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697121.840782] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:07 2019][1697121.846788] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.852792] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697121.857845] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.864113] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697121.870733] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.876999] migration/37 S ffffa05c29a15140 0 197 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697121.884268] Call Trace: [Wed Feb 6 09:39:07 2019][1697121.886898] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697121.892044] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:07 2019][1697121.898050] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.904059] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697121.909120] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.915393] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697121.922006] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.928272] ksoftirqd/37 S ffffa05c29a14100 0 198 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697121.935540] Call Trace: [Wed Feb 6 09:39:07 2019][1697121.938171] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697121.943319] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:07 2019][1697121.949333] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.955339] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697121.960393] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.966668] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697121.973288] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697121.979553] kworker/37:0H S ffffa05c29a48000 0 200 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697121.986828] Call Trace: [Wed Feb 6 09:39:07 2019][1697121.989459] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:07 2019][1697121.995640] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697122.000787] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:07 2019][1697122.006534] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:07 2019][1697122.013240] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697122.018295] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697122.024568] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697122.031182] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697122.037449] watchdog/38 S ffffa06c29a1e180 0 201 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697122.044727] Call Trace: [Wed Feb 6 09:39:07 2019][1697122.047353] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:07 2019][1697122.052841] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697122.057989] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:07 2019][1697122.064004] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697122.070018] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697122.075079] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697122.081346] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697122.087966] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697122.094234] migration/38 S ffffa06c29a1d140 0 202 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697122.101509] Call Trace: [Wed Feb 6 09:39:07 2019][1697122.104139] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697122.109279] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:07 2019][1697122.115295] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697122.121307] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697122.126361] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697122.132636] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1697122.139257] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697122.145520] ksoftirqd/38 S ffffffffa795bad6 0 203 2 0x00000000 [Wed Feb 6 09:39:07 2019][1697122.152791] Call Trace: [Wed Feb 6 09:39:07 2019][1697122.155420] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1697122.160559] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:07 2019][1697122.166563] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:07 2019][1697122.172572] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1697122.177626] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.183900] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697122.190520] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.196785] kworker/38:0H S ffffa06c29a1a080 0 205 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697122.204059] Call Trace: [Wed Feb 6 09:39:08 2019][1697122.206694] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:08 2019][1697122.212880] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697122.218019] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:08 2019][1697122.223769] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:08 2019][1697122.230490] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697122.235543] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.241812] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697122.248432] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.254696] watchdog/39 S ffffa07c295e6180 0 206 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697122.261967] Call Trace: [Wed Feb 6 09:39:08 2019][1697122.264595] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:08 2019][1697122.270083] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697122.275231] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:08 2019][1697122.281245] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.287258] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697122.292311] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.298578] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697122.305192] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.311463] migration/39 S ffffa07c295e5140 0 207 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697122.318734] Call Trace: [Wed Feb 6 09:39:08 2019][1697122.321363] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697122.326503] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:08 2019][1697122.332509] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.338515] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697122.343575] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.349841] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697122.356458] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.362731] ksoftirqd/39 S ffffa07c295e4100 0 208 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697122.370025] Call Trace: [Wed Feb 6 09:39:08 2019][1697122.372654] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697122.377800] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:08 2019][1697122.383807] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.389812] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697122.394864] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.401133] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697122.407754] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.414018] kworker/39:0 S ffffa07c295e30c0 0 209 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697122.421294] Call Trace: [Wed Feb 6 09:39:08 2019][1697122.423926] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:08 2019][1697122.430103] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697122.435243] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:08 2019][1697122.440994] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:08 2019][1697122.447699] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697122.452751] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.459016] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697122.465632] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.471905] kworker/39:0H S ffffa07c295e2080 0 210 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697122.479176] Call Trace: [Wed Feb 6 09:39:08 2019][1697122.481803] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:08 2019][1697122.487982] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697122.493128] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:08 2019][1697122.498876] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:08 2019][1697122.505575] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697122.510628] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.516895] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697122.523516] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.529790] watchdog/40 S ffffa04ca96c6180 0 211 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697122.537060] Call Trace: [Wed Feb 6 09:39:08 2019][1697122.539688] [] ? touch_nmi_watchdog+0x30/0x30 [Wed Feb 6 09:39:08 2019][1697122.545868] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:08 2019][1697122.551362] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697122.556501] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:08 2019][1697122.562506] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.568513] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697122.573565] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.579831] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697122.586445] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.592711] migration/40 S ffffa04ca96c5140 0 212 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697122.599980] Call Trace: [Wed Feb 6 09:39:08 2019][1697122.602610] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697122.607757] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:08 2019][1697122.613762] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.619770] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697122.624831] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.631097] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697122.637709] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.643974] ksoftirqd/40 S ffffa04ca96c4100 0 213 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697122.651243] Call Trace: [Wed Feb 6 09:39:08 2019][1697122.653872] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697122.659013] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:08 2019][1697122.665027] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.671031] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697122.676086] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.682360] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697122.688974] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.695249] kworker/40:0H S ffffa04ca96c2080 0 215 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697122.702529] Call Trace: [Wed Feb 6 09:39:08 2019][1697122.705162] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:08 2019][1697122.711343] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697122.716490] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:08 2019][1697122.722237] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:08 2019][1697122.728933] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697122.733993] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.740263] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697122.746877] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.753152] watchdog/41 S ffffa05c29a4e180 0 216 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697122.760428] Call Trace: [Wed Feb 6 09:39:08 2019][1697122.763057] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:08 2019][1697122.768542] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697122.773681] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:08 2019][1697122.779689] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.785703] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697122.790755] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.797023] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697122.803642] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.809909] migration/41 S ffffa05c29a4d140 0 217 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697122.817177] Call Trace: [Wed Feb 6 09:39:08 2019][1697122.819806] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697122.824946] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:08 2019][1697122.830954] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.836967] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697122.842028] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.848294] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697122.854907] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.861173] ksoftirqd/41 S ffffa05c29a4c100 0 218 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697122.868442] Call Trace: [Wed Feb 6 09:39:08 2019][1697122.871071] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697122.876208] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:08 2019][1697122.882216] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.888222] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697122.893274] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.899542] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697122.906163] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.912436] kworker/41:0H S ffffa05c29a4a080 0 220 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697122.919710] Call Trace: [Wed Feb 6 09:39:08 2019][1697122.922344] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:08 2019][1697122.928531] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697122.933671] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:08 2019][1697122.939418] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:08 2019][1697122.946125] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697122.951177] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.957444] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697122.964057] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697122.970330] watchdog/42 S ffffa06c29a19040 0 221 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697122.977600] Call Trace: [Wed Feb 6 09:39:08 2019][1697122.980230] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:08 2019][1697122.985723] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697122.990863] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:08 2019][1697122.996878] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697123.002893] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697123.007946] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697123.014221] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697123.020842] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697123.027114] migration/42 S ffffa06c29a80000 0 222 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697123.034386] Call Trace: [Wed Feb 6 09:39:08 2019][1697123.037013] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697123.042153] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:08 2019][1697123.048168] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697123.054174] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697123.059234] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697123.065501] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697123.072113] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697123.078379] ksoftirqd/42 S ffffa06c29a81040 0 223 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697123.085665] Call Trace: [Wed Feb 6 09:39:08 2019][1697123.088295] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697123.093433] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:08 2019][1697123.099440] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697123.105446] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697123.110500] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697123.116774] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1697123.123388] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697123.129660] kworker/42:0 S ffffa06c29a82080 0 224 2 0x00000000 [Wed Feb 6 09:39:08 2019][1697123.136933] Call Trace: [Wed Feb 6 09:39:08 2019][1697123.139559] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:08 2019][1697123.145741] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1697123.150887] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:08 2019][1697123.156642] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:08 2019][1697123.163349] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1697123.168410] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1697123.174677] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697123.181290] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.187556] kworker/42:0H S ffffa06c29a830c0 0 225 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697123.194837] Call Trace: [Wed Feb 6 09:39:09 2019][1697123.197472] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:09 2019][1697123.203659] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697123.208807] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:09 2019][1697123.214560] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:09 2019][1697123.221261] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697123.226321] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.232589] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697123.239210] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.245484] watchdog/43 S ffffa07c295e1040 0 226 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697123.252752] Call Trace: [Wed Feb 6 09:39:09 2019][1697123.255382] [] ? touch_nmi_watchdog+0x30/0x30 [Wed Feb 6 09:39:09 2019][1697123.261571] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:09 2019][1697123.267065] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697123.272205] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:09 2019][1697123.278219] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.284225] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697123.289285] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.295554] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697123.302174] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.308449] migration/43 S ffffa07c29648000 0 227 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697123.315718] Call Trace: [Wed Feb 6 09:39:09 2019][1697123.318345] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697123.323485] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:09 2019][1697123.329492] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.335507] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697123.340569] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.346842] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697123.353456] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.359729] ksoftirqd/43 S ffffa07c29649040 0 228 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697123.366999] Call Trace: [Wed Feb 6 09:39:09 2019][1697123.369626] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697123.374767] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:09 2019][1697123.380782] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.386789] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697123.391849] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.398114] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697123.404729] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.411003] kworker/43:0H S ffffa07c2964b0c0 0 230 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697123.418285] Call Trace: [Wed Feb 6 09:39:09 2019][1697123.420919] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:09 2019][1697123.427107] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697123.432254] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:09 2019][1697123.438009] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:09 2019][1697123.444716] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697123.449770] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.456044] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697123.462658] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.468930] watchdog/44 S ffffa04ca9728000 0 231 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697123.476201] Call Trace: [Wed Feb 6 09:39:09 2019][1697123.478829] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:09 2019][1697123.484315] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697123.489463] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:09 2019][1697123.495469] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.501484] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697123.506544] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.512810] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697123.519423] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.525690] migration/44 S ffffa04ca9729040 0 232 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697123.532959] Call Trace: [Wed Feb 6 09:39:09 2019][1697123.535591] [] ? cpu_stop_queue_work+0x70/0x70 [Wed Feb 6 09:39:09 2019][1697123.541862] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697123.547003] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:09 2019][1697123.553015] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.559022] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697123.564074] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.570342] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697123.576963] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.583238] ksoftirqd/44 S ffffa04ca972a080 0 233 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697123.590514] Call Trace: [Wed Feb 6 09:39:09 2019][1697123.593143] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697123.598282] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:09 2019][1697123.604288] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.610295] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697123.615356] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.621624] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697123.628244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.634510] kworker/44:0H S ffffa04ca972c100 0 235 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697123.641783] Call Trace: [Wed Feb 6 09:39:09 2019][1697123.644415] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:09 2019][1697123.650598] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697123.655744] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:09 2019][1697123.661499] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:09 2019][1697123.668207] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697123.673268] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.679535] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697123.686155] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.692420] watchdog/45 S ffffa05c29a49040 0 236 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697123.699689] Call Trace: [Wed Feb 6 09:39:09 2019][1697123.702319] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:09 2019][1697123.707815] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697123.712961] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:09 2019][1697123.718968] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.724983] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697123.730046] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.736320] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697123.742941] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.749213] migration/45 S ffffa05c29a88000 0 237 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697123.756483] Call Trace: [Wed Feb 6 09:39:09 2019][1697123.759113] [] ? cpu_stop_queue_work+0x70/0x70 [Wed Feb 6 09:39:09 2019][1697123.765388] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697123.770534] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:09 2019][1697123.776540] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.782547] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697123.787607] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.793874] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697123.800488] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.806763] ksoftirqd/45 S ffffa05c29a89040 0 238 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697123.814040] Call Trace: [Wed Feb 6 09:39:09 2019][1697123.816669] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697123.821816] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:09 2019][1697123.827824] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.833836] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697123.838890] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.845156] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697123.851776] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.858042] kworker/45:0H S ffffa05c29a8b0c0 0 240 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697123.865316] Call Trace: [Wed Feb 6 09:39:09 2019][1697123.867949] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:09 2019][1697123.874131] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697123.879277] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:09 2019][1697123.885024] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:09 2019][1697123.891730] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697123.896785] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.903059] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697123.909671] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.915937] watchdog/46 S ffffa06c29a86180 0 241 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697123.923206] Call Trace: [Wed Feb 6 09:39:09 2019][1697123.925836] [] ? touch_nmi_watchdog+0x30/0x30 [Wed Feb 6 09:39:09 2019][1697123.932023] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:09 2019][1697123.937518] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697123.942657] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:09 2019][1697123.948664] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.954671] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697123.959733] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.966006] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697123.972629] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697123.978900] migration/46 S ffffa06c29a85140 0 242 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697123.986168] Call Trace: [Wed Feb 6 09:39:09 2019][1697123.988799] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697123.993940] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:09 2019][1697123.999953] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697124.005959] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697124.011013] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697124.017286] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697124.023901] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697124.030175] ksoftirqd/46 S ffffa06c29a84100 0 243 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697124.037444] Call Trace: [Wed Feb 6 09:39:09 2019][1697124.040071] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697124.045212] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:09 2019][1697124.051225] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697124.057234] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697124.062295] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697124.068568] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697124.075180] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697124.081446] kworker/46:0H S ffffa06c29ab1040 0 245 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697124.088720] Call Trace: [Wed Feb 6 09:39:09 2019][1697124.091354] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:09 2019][1697124.097541] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697124.102681] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:09 2019][1697124.108427] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:09 2019][1697124.115126] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697124.120181] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697124.126453] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1697124.133066] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697124.139330] watchdog/47 S ffffa07c2964e180 0 246 2 0x00000000 [Wed Feb 6 09:39:09 2019][1697124.146600] Call Trace: [Wed Feb 6 09:39:09 2019][1697124.149229] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:09 2019][1697124.154716] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1697124.159854] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:09 2019][1697124.165861] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:09 2019][1697124.171868] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1697124.176930] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.183203] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.189817] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.196090] migration/47 S ffffa07c2964d140 0 247 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697124.203360] Call Trace: [Wed Feb 6 09:39:10 2019][1697124.205987] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697124.211129] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:10 2019][1697124.217144] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.223157] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697124.228212] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.234483] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.241097] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.247363] ksoftirqd/47 S ffffa07c2964c100 0 248 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697124.254634] Call Trace: [Wed Feb 6 09:39:10 2019][1697124.257262] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697124.262410] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:39:10 2019][1697124.268423] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.274429] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697124.279483] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.285758] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.292380] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.298653] kworker/47:0H S ffffa07c29681040 0 250 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697124.305927] Call Trace: [Wed Feb 6 09:39:10 2019][1697124.308561] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:10 2019][1697124.314747] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697124.319888] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:10 2019][1697124.325642] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:10 2019][1697124.332339] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697124.337395] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.343668] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.350282] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.356546] kdevtmpfs S ffffa04ca9022080 0 255 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697124.363815] Call Trace: [Wed Feb 6 09:39:10 2019][1697124.366445] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697124.371587] [] devtmpfsd+0x16d/0x180 [Wed Feb 6 09:39:10 2019][1697124.376993] [] ? handle_create.isra.2+0x220/0x220 [Wed Feb 6 09:39:10 2019][1697124.383529] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697124.388590] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.394864] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.401475] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.407742] netns S ffffa04ca90230c0 0 256 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697124.415016] Call Trace: [Wed Feb 6 09:39:10 2019][1697124.417648] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697124.422789] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:10 2019][1697124.428631] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:10 2019][1697124.434817] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697124.439870] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.446136] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.452748] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.459016] khungtaskd S ffffa04ca9024100 0 257 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697124.466291] Call Trace: [Wed Feb 6 09:39:10 2019][1697124.468922] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697124.474062] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:10 2019][1697124.480075] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:10 2019][1697124.485736] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:10 2019][1697124.492270] [] schedule_timeout_interruptible+0x1e/0x20 [Wed Feb 6 09:39:10 2019][1697124.499331] [] watchdog+0x4c/0x2c0 [Wed Feb 6 09:39:10 2019][1697124.504560] [] ? reset_hung_task_detector+0x20/0x20 [Wed Feb 6 09:39:10 2019][1697124.511268] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697124.516330] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.522603] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.529224] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.535491] writeback S ffffa04ca9025140 0 258 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697124.542764] Call Trace: [Wed Feb 6 09:39:10 2019][1697124.545398] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697124.550544] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:10 2019][1697124.556380] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:10 2019][1697124.562567] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697124.567627] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.573893] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.580506] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.586770] kintegrityd S ffffa04ca9026180 0 259 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697124.594045] Call Trace: [Wed Feb 6 09:39:10 2019][1697124.596677] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697124.601816] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:10 2019][1697124.607652] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:10 2019][1697124.613838] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697124.618892] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.625167] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.631788] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.638076] bioset S ffffa04ca9020000 0 260 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697124.645351] Call Trace: [Wed Feb 6 09:39:10 2019][1697124.647976] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697124.653116] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:10 2019][1697124.658950] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:10 2019][1697124.665137] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697124.670189] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.676455] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.683067] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.689335] bioset S ffffa04ca9021040 0 261 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697124.696606] Call Trace: [Wed Feb 6 09:39:10 2019][1697124.699232] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697124.704373] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:10 2019][1697124.710212] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:10 2019][1697124.716392] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697124.721447] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.727722] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.734341] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.740607] bioset S ffffa05b7f130000 0 262 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697124.747879] Call Trace: [Wed Feb 6 09:39:10 2019][1697124.750506] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697124.755652] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:10 2019][1697124.761485] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:10 2019][1697124.767665] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697124.772718] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.778994] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.785613] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.791878] kblockd S ffffa05b7f131040 0 263 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697124.799153] Call Trace: [Wed Feb 6 09:39:10 2019][1697124.801788] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697124.806934] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:10 2019][1697124.812766] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:10 2019][1697124.818947] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697124.824011] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.830285] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.836904] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.843169] md S ffffa05b7f132080 0 264 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697124.850443] Call Trace: [Wed Feb 6 09:39:10 2019][1697124.853077] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697124.858225] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:10 2019][1697124.864065] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:10 2019][1697124.870243] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697124.875297] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.881563] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.888177] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.894451] edac-poller S ffffa05b7f1330c0 0 265 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697124.901724] Call Trace: [Wed Feb 6 09:39:10 2019][1697124.904357] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697124.909495] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:10 2019][1697124.915329] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:10 2019][1697124.921509] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697124.926561] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.932829] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.939449] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.945715] watchdogd S ffffa05b7f134100 0 266 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697124.952987] Call Trace: [Wed Feb 6 09:39:10 2019][1697124.955621] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697124.960759] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:10 2019][1697124.966594] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:10 2019][1697124.972781] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697124.977835] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.984112] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697124.990730] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697124.996995] irq/26-AMD-Vi S ffffa05b7f136180 0 270 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697125.004267] Call Trace: [Wed Feb 6 09:39:10 2019][1697125.006894] [] ? irq_finalize_oneshot.part.31+0xf0/0xf0 [Wed Feb 6 09:39:10 2019][1697125.013940] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697125.019079] [] irq_thread+0xd1/0x180 [Wed Feb 6 09:39:10 2019][1697125.024480] [] ? irq_forced_thread_fn+0x50/0x50 [Wed Feb 6 09:39:10 2019][1697125.030841] [] ? irq_thread_check_affinity+0xf0/0xf0 [Wed Feb 6 09:39:10 2019][1697125.037628] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697125.042690] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697125.048963] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697125.055575] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697125.061842] irq/27-AMD-Vi S ffffa05b7f135140 0 271 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697125.069120] Call Trace: [Wed Feb 6 09:39:10 2019][1697125.071748] [] ? irq_finalize_oneshot.part.31+0xf0/0xf0 [Wed Feb 6 09:39:10 2019][1697125.078793] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697125.083932] [] irq_thread+0xd1/0x180 [Wed Feb 6 09:39:10 2019][1697125.089333] [] ? irq_forced_thread_fn+0x50/0x50 [Wed Feb 6 09:39:10 2019][1697125.095695] [] ? irq_thread_check_affinity+0xf0/0xf0 [Wed Feb 6 09:39:10 2019][1697125.102489] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697125.107542] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697125.113817] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1697125.120437] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697125.126702] irq/28-AMD-Vi S ffffa05b7c5e8000 0 272 2 0x00000000 [Wed Feb 6 09:39:10 2019][1697125.133972] Call Trace: [Wed Feb 6 09:39:10 2019][1697125.136600] [] ? irq_finalize_oneshot.part.31+0xf0/0xf0 [Wed Feb 6 09:39:10 2019][1697125.143646] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1697125.148785] [] irq_thread+0xd1/0x180 [Wed Feb 6 09:39:10 2019][1697125.154186] [] ? irq_forced_thread_fn+0x50/0x50 [Wed Feb 6 09:39:10 2019][1697125.160539] [] ? irq_thread_check_affinity+0xf0/0xf0 [Wed Feb 6 09:39:10 2019][1697125.167335] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1697125.172395] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1697125.178662] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697125.185283] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.191547] irq/29-AMD-Vi S ffffa05b7c5e9040 0 273 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697125.198816] Call Trace: [Wed Feb 6 09:39:11 2019][1697125.201447] [] ? irq_finalize_oneshot.part.31+0xf0/0xf0 [Wed Feb 6 09:39:11 2019][1697125.208501] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697125.213648] [] irq_thread+0xd1/0x180 [Wed Feb 6 09:39:11 2019][1697125.219050] [] ? irq_forced_thread_fn+0x50/0x50 [Wed Feb 6 09:39:11 2019][1697125.225409] [] ? irq_thread_check_affinity+0xf0/0xf0 [Wed Feb 6 09:39:11 2019][1697125.232196] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697125.237257] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.243523] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697125.250135] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.256401] kswapd0 S ffffa05b7c5ee180 0 278 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697125.263670] Call Trace: [Wed Feb 6 09:39:11 2019][1697125.266301] [] ? calculate_normal_threshold+0x27/0x50 [Wed Feb 6 09:39:11 2019][1697125.273180] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697125.278320] [] kswapd+0x38f/0x440 [Wed Feb 6 09:39:11 2019][1697125.283460] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:11 2019][1697125.289467] [] ? balance_pgdat+0x5e0/0x5e0 [Wed Feb 6 09:39:11 2019][1697125.295393] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697125.300446] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.306711] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697125.313326] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.319598] kswapd1 S ffffa05b7c690000 0 279 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697125.326868] Call Trace: [Wed Feb 6 09:39:11 2019][1697125.329498] [] ? calculate_normal_threshold+0x27/0x50 [Wed Feb 6 09:39:11 2019][1697125.336371] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697125.341517] [] kswapd+0x38f/0x440 [Wed Feb 6 09:39:11 2019][1697125.346658] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:11 2019][1697125.352673] [] ? balance_pgdat+0x5e0/0x5e0 [Wed Feb 6 09:39:11 2019][1697125.358601] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697125.363660] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.369926] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697125.376541] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.382807] kswapd2 S ffffa05b7c691040 0 280 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697125.390083] Call Trace: [Wed Feb 6 09:39:11 2019][1697125.392713] [] ? calculate_normal_threshold+0x27/0x50 [Wed Feb 6 09:39:11 2019][1697125.399586] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697125.404726] [] kswapd+0x38f/0x440 [Wed Feb 6 09:39:11 2019][1697125.409873] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:11 2019][1697125.415879] [] ? balance_pgdat+0x5e0/0x5e0 [Wed Feb 6 09:39:11 2019][1697125.421800] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697125.426861] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.433133] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697125.439746] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.446015] kswapd3 S ffffa05b7c692080 0 281 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697125.453292] Call Trace: [Wed Feb 6 09:39:11 2019][1697125.455920] [] ? calculate_normal_threshold+0x27/0x50 [Wed Feb 6 09:39:11 2019][1697125.462791] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697125.467933] [] kswapd+0x38f/0x440 [Wed Feb 6 09:39:11 2019][1697125.473081] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:11 2019][1697125.479095] [] ? balance_pgdat+0x5e0/0x5e0 [Wed Feb 6 09:39:11 2019][1697125.485019] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697125.490073] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.496341] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697125.502956] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.509231] ksmd S ffffa05b7c6930c0 0 282 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697125.516505] Call Trace: [Wed Feb 6 09:39:11 2019][1697125.519134] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697125.524277] [] ksm_scan_thread+0x16a/0x240 [Wed Feb 6 09:39:11 2019][1697125.530203] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:11 2019][1697125.536211] [] ? ksm_do_scan+0x11b0/0x11b0 [Wed Feb 6 09:39:11 2019][1697125.542135] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697125.547189] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.553457] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697125.560077] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.566342] khugepaged S ffffa05b7c694100 0 283 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697125.573612] Call Trace: [Wed Feb 6 09:39:11 2019][1697125.576241] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697125.581379] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:11 2019][1697125.587389] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:11 2019][1697125.593922] [] khugepaged+0x226/0x480 [Wed Feb 6 09:39:11 2019][1697125.599414] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:11 2019][1697125.605421] [] ? khugepaged_scan_mm_slot+0xf00/0xf00 [Wed Feb 6 09:39:11 2019][1697125.612206] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697125.617260] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.623529] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697125.630147] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.636413] crypto S ffffa05b7c695140 0 284 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697125.643687] Call Trace: [Wed Feb 6 09:39:11 2019][1697125.646319] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697125.651461] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:11 2019][1697125.657301] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:11 2019][1697125.663480] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697125.668533] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.674799] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697125.681414] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.687687] kthrotld S ffffa05b7c6e6180 0 292 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697125.694959] Call Trace: [Wed Feb 6 09:39:11 2019][1697125.697592] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697125.702733] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:11 2019][1697125.708574] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:11 2019][1697125.714754] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697125.719814] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.726081] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697125.732702] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.738967] kmpath_rdacd S ffffa06b7c6c9040 0 297 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697125.746241] Call Trace: [Wed Feb 6 09:39:11 2019][1697125.748873] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697125.754018] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:11 2019][1697125.759856] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:11 2019][1697125.766042] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697125.771095] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.777362] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697125.783975] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.790239] kaluad S ffffa06b7c6ca080 0 298 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697125.797515] Call Trace: [Wed Feb 6 09:39:11 2019][1697125.800147] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697125.805285] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:11 2019][1697125.811120] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:11 2019][1697125.817306] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697125.822360] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.828627] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697125.835249] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.841521] kworker/0:2 S ffffa05b7c70e180 0 299 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697125.848794] Call Trace: [Wed Feb 6 09:39:11 2019][1697125.851427] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:11 2019][1697125.857608] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697125.862755] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:11 2019][1697125.868503] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:11 2019][1697125.875210] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697125.880272] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.886547] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697125.893169] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.899443] kpsmoused S ffffa05b7c709040 0 302 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697125.906713] Call Trace: [Wed Feb 6 09:39:11 2019][1697125.909340] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697125.914479] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:11 2019][1697125.920312] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:11 2019][1697125.926491] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697125.931543] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.937810] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697125.944423] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.950690] ipv6_addrconf S ffffa05b7c70b0c0 0 304 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697125.957970] Call Trace: [Wed Feb 6 09:39:11 2019][1697125.960603] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697125.965743] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:11 2019][1697125.971576] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:11 2019][1697125.977756] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697125.982816] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697125.989081] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697125.995694] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697126.001962] deferwq S ffffa05b7c109040 0 318 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697126.009244] Call Trace: [Wed Feb 6 09:39:11 2019][1697126.011876] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697126.017021] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:11 2019][1697126.022857] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:11 2019][1697126.029038] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697126.034099] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697126.040372] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697126.046986] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697126.053260] kworker/40:1 S ffffa05b7c10b0c0 0 320 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697126.060533] Call Trace: [Wed Feb 6 09:39:11 2019][1697126.063167] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:11 2019][1697126.069354] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697126.074495] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:11 2019][1697126.080248] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:11 2019][1697126.086947] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697126.092000] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697126.098266] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697126.104881] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697126.111152] kworker/34:1 S ffffa06c29ab2080 0 325 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697126.118426] Call Trace: [Wed Feb 6 09:39:11 2019][1697126.121061] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:11 2019][1697126.127249] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1697126.132397] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:11 2019][1697126.138142] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:11 2019][1697126.144842] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1697126.149896] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697126.156169] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1697126.162782] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1697126.169049] kauditd S ffffa05b7c2530c0 0 440 2 0x00000000 [Wed Feb 6 09:39:11 2019][1697126.176327] Call Trace: [Wed Feb 6 09:39:11 2019][1697126.178956] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697126.184105] [] kauditd_thread+0x184/0x1e0 [Wed Feb 6 09:39:12 2019][1697126.189947] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:12 2019][1697126.195697] [] ? audit_printk_skb+0x70/0x70 [Wed Feb 6 09:39:12 2019][1697126.201703] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697126.206758] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.213031] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697126.219646] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.225919] kworker/23:1 S ffffa08b7a888000 0 805 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697126.233193] Call Trace: [Wed Feb 6 09:39:12 2019][1697126.235826] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:12 2019][1697126.242005] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697126.247145] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:12 2019][1697126.252900] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:12 2019][1697126.259606] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697126.264659] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.270927] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697126.277548] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.283822] kworker/10:1 S ffffa072fc0a8000 0 1089 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697126.291095] Call Trace: [Wed Feb 6 09:39:12 2019][1697126.293731] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:12 2019][1697126.299916] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697126.305054] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:12 2019][1697126.310803] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:12 2019][1697126.317512] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697126.322572] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.328837] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697126.335451] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.341717] kworker/30:1 S ffffa07b7647a080 0 2164 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697126.348989] Call Trace: [Wed Feb 6 09:39:12 2019][1697126.351624] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:12 2019][1697126.357812] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697126.362959] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:12 2019][1697126.368714] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:12 2019][1697126.375423] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697126.380484] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.386759] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697126.393378] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.399644] scsi_eh_0 S ffffa07b7778c100 0 2262 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697126.406913] Call Trace: [Wed Feb 6 09:39:12 2019][1697126.409542] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697126.414695] [] scsi_error_handler+0xa3/0x8b0 [Wed Feb 6 09:39:12 2019][1697126.420794] [] ? scsi_eh_get_sense+0x250/0x250 [Wed Feb 6 09:39:12 2019][1697126.427066] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697126.432120] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.438385] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697126.444998] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.451263] scsi_tmf_0 S ffffa07b7778d140 0 2266 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697126.458538] Call Trace: [Wed Feb 6 09:39:12 2019][1697126.461171] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697126.466310] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:12 2019][1697126.472142] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:12 2019][1697126.478322] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697126.483376] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.489650] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697126.496264] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.502536] scsi_eh_1 S ffffa07b75d9e180 0 3704 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697126.509821] Call Trace: [Wed Feb 6 09:39:12 2019][1697126.512452] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697126.517592] [] scsi_error_handler+0xa3/0x8b0 [Wed Feb 6 09:39:12 2019][1697126.523693] [] ? scsi_eh_get_sense+0x250/0x250 [Wed Feb 6 09:39:12 2019][1697126.529959] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697126.535025] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.541295] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697126.547909] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.554183] scsi_tmf_1 S ffffa07b75ef0000 0 3706 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697126.561464] Call Trace: [Wed Feb 6 09:39:12 2019][1697126.564098] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697126.569244] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:12 2019][1697126.575078] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:12 2019][1697126.581257] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697126.586311] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.592585] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697126.599205] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.605472] poll_mpt3sas0_h S ffffa07b752dc100 0 5528 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697126.612743] Call Trace: [Wed Feb 6 09:39:12 2019][1697126.615370] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697126.620508] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:12 2019][1697126.626341] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:12 2019][1697126.632520] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697126.637575] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.643851] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697126.650469] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.656737] ata_sff S ffffa07b74eed140 0 7243 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697126.664024] Call Trace: [Wed Feb 6 09:39:12 2019][1697126.666649] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697126.671791] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:12 2019][1697126.677624] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:12 2019][1697126.683811] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697126.688864] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.695132] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697126.701751] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.708018] kworker/7:1 S ffffa08b77a5e180 0 7267 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697126.715298] Call Trace: [Wed Feb 6 09:39:12 2019][1697126.717934] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:12 2019][1697126.724119] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697126.729260] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:12 2019][1697126.735016] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:12 2019][1697126.741722] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697126.746774] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.753041] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697126.759654] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.765920] kworker/14:1 S ffffa07b74eee180 0 7269 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697126.773192] Call Trace: [Wed Feb 6 09:39:12 2019][1697126.775819] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:12 2019][1697126.782005] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697126.787143] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:12 2019][1697126.792893] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:12 2019][1697126.799601] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697126.804660] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.810929] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697126.817549] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.823816] kworker/15:1 S ffffa08b77a59040 0 7270 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697126.831097] Call Trace: [Wed Feb 6 09:39:12 2019][1697126.833728] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:12 2019][1697126.839908] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697126.845050] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:12 2019][1697126.850804] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:12 2019][1697126.857509] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697126.862563] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.868830] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697126.875441] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.881709] kworker/19:1 S ffffa08b77a5a080 0 7271 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697126.888980] Call Trace: [Wed Feb 6 09:39:12 2019][1697126.891605] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:12 2019][1697126.897787] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697126.902934] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:12 2019][1697126.908689] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:12 2019][1697126.915396] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697126.920447] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.926716] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697126.933338] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.939610] kworker/27:1 S ffffa08b77a5d140 0 7274 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697126.946886] Call Trace: [Wed Feb 6 09:39:12 2019][1697126.949517] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:12 2019][1697126.955696] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697126.960836] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:12 2019][1697126.966582] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:12 2019][1697126.973280] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697126.978336] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.984610] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697126.991232] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697126.997504] kworker/31:1 S ffffa08b77a5b0c0 0 7277 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697127.004777] Call Trace: [Wed Feb 6 09:39:12 2019][1697127.007402] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:12 2019][1697127.013582] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697127.018722] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:12 2019][1697127.024470] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:12 2019][1697127.031176] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697127.036229] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697127.042496] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697127.049116] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697127.055383] kworker/38:1 S ffffa07b741b6180 0 7280 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697127.062662] Call Trace: [Wed Feb 6 09:39:12 2019][1697127.065290] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:12 2019][1697127.071476] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697127.076619] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:12 2019][1697127.082370] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:12 2019][1697127.089068] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697127.094124] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697127.100396] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697127.107010] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697127.113277] kworker/47:1 S ffffa08b7582e180 0 7285 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697127.120548] Call Trace: [Wed Feb 6 09:39:12 2019][1697127.123182] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:12 2019][1697127.129362] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1697127.134502] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:12 2019][1697127.140255] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:12 2019][1697127.146955] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1697127.152007] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697127.158275] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1697127.164894] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1697127.171161] kworker/26:1 S ffffa07b7466c100 0 7286 2 0x00000000 [Wed Feb 6 09:39:12 2019][1697127.178435] Call Trace: [Wed Feb 6 09:39:12 2019][1697127.181067] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:13 2019][1697127.187246] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.192388] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:13 2019][1697127.198140] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:13 2019][1697127.204841] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.209901] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.216169] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697127.222788] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.229054] scsi_eh_2 S ffffa07b7466d140 0 7309 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697127.236323] Call Trace: [Wed Feb 6 09:39:13 2019][1697127.238955] [] ? __scsi_iterate_devices+0x68/0x90 [Wed Feb 6 09:39:13 2019][1697127.245487] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.250628] [] scsi_error_handler+0xa3/0x8b0 [Wed Feb 6 09:39:13 2019][1697127.256719] [] ? scsi_eh_get_sense+0x250/0x250 [Wed Feb 6 09:39:13 2019][1697127.262987] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.268047] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.274315] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697127.280933] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.287200] scsi_tmf_2 S ffffa07b7466e180 0 7310 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697127.294473] Call Trace: [Wed Feb 6 09:39:13 2019][1697127.297098] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.302237] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:13 2019][1697127.308073] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:13 2019][1697127.314259] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.319311] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.325577] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697127.332192] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.338465] mlx5_health0000 S ffffa07b74668000 0 7348 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697127.345737] Call Trace: [Wed Feb 6 09:39:13 2019][1697127.348361] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.353501] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:13 2019][1697127.359336] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:13 2019][1697127.365522] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.370575] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.376841] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697127.383453] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.389721] mlx5_cmd_0000:8 S ffffa07b7466b0c0 0 7362 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697127.396992] Call Trace: [Wed Feb 6 09:39:13 2019][1697127.399618] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.404757] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:13 2019][1697127.410592] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:13 2019][1697127.416778] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.421832] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.428106] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697127.434727] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.440993] mlx5_page_alloc S ffffa07b7466a080 0 7370 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697127.448264] Call Trace: [Wed Feb 6 09:39:13 2019][1697127.450890] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.456034] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:13 2019][1697127.461873] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:13 2019][1697127.468059] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.473114] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.479387] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697127.485999] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.492268] kworker/u261:0 S ffffa07b75efe180 0 7436 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697127.499546] Call Trace: [Wed Feb 6 09:39:13 2019][1697127.502171] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.507312] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:13 2019][1697127.513067] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:13 2019][1697127.519774] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.524827] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.531093] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697127.537715] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.543990] kworker/u262:0 S ffffa05b7c255140 0 7437 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697127.551271] Call Trace: [Wed Feb 6 09:39:13 2019][1697127.553905] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.559053] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:13 2019][1697127.564807] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:13 2019][1697127.571508] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.576568] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.582834] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697127.589458] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.595729] kworker/u263:0 S ffffa06b7b966180 0 7438 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697127.603003] Call Trace: [Wed Feb 6 09:39:13 2019][1697127.605637] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.610785] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:13 2019][1697127.616538] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:13 2019][1697127.623239] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.628301] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.634574] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697127.641188] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.647463] kworker/u265:0 S ffffa08b765d4100 0 7440 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697127.654753] Call Trace: [Wed Feb 6 09:39:13 2019][1697127.657384] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.662526] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:13 2019][1697127.668279] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:13 2019][1697127.674979] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.680036] []7.686306] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697127.692919] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.699185] mlx5_fc S ffffa07b74141040 0 7450 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697127.706456] Call Trace: [Wed Feb 6 09:39:13 2019][1697127.709081] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.714221] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:13 2019][1697127.720056] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:13 2019][1697127.726243] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.731296] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.737561] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697127.744174] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.750438] ib-comp-wq S ffffa08b76001040 0 7468 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697127.757713] Call Trace: [Wed Feb 6 09:39:13 2019][1697127.760347] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.765496] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:13 2019][1697127.771336] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:13 2019][1697127.777516] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.782576] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.788843] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697127.795457] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.801730] ib-comp-unb-wq S ffffa08b76005140 0 7469 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697127.809002] Call Trace: [Wed Feb 6 09:39:13 2019][1697127.811629] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.816775] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:13 2019][1697127.822610] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:13 2019][1697127.828799] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.833858] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.840134] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697127.846754] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.853021] ib_mcast S ffffa08b76004100 0 7470 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697127.860303] Call Trace: [Wed Feb 6 09:39:13 2019][1697127.862935] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.868075] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:13 2019][1697127.873906] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:13 2019][1697127.880085] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.885138] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.891405] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697127.898018] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.904284] ib_nl_sa_wq S ffffa08b760030c0 0 7471 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697127.911557] Call Trace: [Wed Feb 6 09:39:13 2019][1697127.914189] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.919330] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:13 2019][1697127.925173] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:13 2019][1697127.931359] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.936411] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.942680] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697127.949299] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.955564] mlx5_ib_sigerr_ S ffffa08b765d0000 0 7487 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697127.962838] Call Trace: [Wed Feb 6 09:39:13 2019][1697127.965470] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697127.970612] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:13 2019][1697127.976453] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:13 2019][1697127.982633] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697127.987693] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697127.993959] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697128.000571] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697128.006837] ib_mad1 S ffffa07b741430c0 0 7492 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697128.014111] Call Trace: [Wed Feb 6 09:39:13 2019][1697128.016745] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697128.021884] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:13 2019][1697128.027716] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:13 2019][1697128.033896] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697128.038947] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697128.045215] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697128.051828] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697128.058092] to_fifo S ffffa07b74144100 0 7493 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697128.065364] Call Trace: [Wed Feb 6 09:39:13 2019][1697128.067991] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697128.073132] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:13 2019][1697128.078973] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:13 2019][1697128.085160] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697128.090212] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697128.096479] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697128.103092] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697128.109356] mkey_cache S ffffa07b74145140 0 7496 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697128.116629] Call Trace: [Wed Feb 6 09:39:13 2019][1697128.119255] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697128.124396] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:13 2019][1697128.130235] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:13 2019][1697128.136414] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1697128.141468] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697128.147735] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1697128.154355] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1697128.160620] kworker/35:1H S ffffa08b765d1040 0 7575 2 0x00000000 [Wed Feb 6 09:39:13 2019][1697128.167894] Call Trace: [Wed Feb 6 09:39:13 2019][1697128.170528] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:13 2019][1697128.176706] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1697128.181845] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:14 2019][1697128.187593] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:14 2019][1697128.194301] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1697128.199362] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.205630] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1697128.212250] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.218525] poll_mpt3sas0_s S ffffa07b75efb0c0 0 7601 2 0x00000000 [Wed Feb 6 09:39:14 2019][1697128.225807] Call Trace: [Wed Feb 6 09:39:14 2019][1697128.228440] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1697128.233587] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:14 2019][1697128.239419] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:14 2019][1697128.245602] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1697128.250660] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.256928] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1697128.263549] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.269823] kworker/45:2 S ffffa06b7c0a9040 0 7817 2 0x00000000 [Wed Feb 6 09:39:14 2019][1697128.277097] Call Trace: [Wed Feb 6 09:39:14 2019][1697128.279730] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:14 2019][1697128.285916] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1697128.291055] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:14 2019][1697128.296803] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:14 2019][1697128.303501] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1697128.308554] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.314820] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1697128.321435] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.327709] kworker/0:1H S ffffa05b78b62080 0 7833 2 0x00000000 [Wed Feb 6 09:39:14 2019][1697128.334982] Call Trace: [Wed Feb 6 09:39:14 2019][1697128.337614] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:14 2019][1697128.343794] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1697128.348935] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:14 2019][1697128.354688] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:14 2019][1697128.361389] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1697128.366449] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.372716] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1697128.379338] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.385627] jbd2/sdk2-8 S ffffa06b7c0ab0c0 0 7868 2 0x00000000 [Wed Feb 6 09:39:14 2019][1697128.392897] Call Trace: [Wed Feb 6 09:39:14 2019][1697128.395526] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1697128.400697] [] kjournald2+0x203/0x260 [jbd2] [Wed Feb 6 09:39:14 2019][1697128.406795] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:14 2019][1697128.412814] [] ? commit_timeout+0x10/0x10 [jbd2] [Wed Feb 6 09:39:14 2019][1697128.419257] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1697128.424317] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.430582] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1697128.437196] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.443471] ext4-rsv-conver S ffffa06b7c0ac100 0 7869 2 0x00000000 [Wed Feb 6 09:39:14 2019][1697128.450743] Call Trace: [Wed Feb 6 09:39:14 2019][1697128.453377] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1697128.458525] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:14 2019][1697128.464357] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:14 2019][1697128.470537] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1697128.475599] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.481873] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1697128.488485] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.494751] systemd-journal R running task 0 7941 1 0x00000000 [Wed Feb 6 09:39:14 2019][1697128.502021] Call Trace: [Wed Feb 6 09:39:14 2019][1697128.504650] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1697128.509797] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:39:14 2019][1697128.517021] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Wed Feb 6 09:39:14 2019][1697128.523985] [] ? schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:14 2019][1697128.530684] [] ? ep_poll+0x23e/0x360 [Wed Feb 6 09:39:14 2019][1697128.536086] [] ? do_truncate+0x88/0xc0 [Wed Feb 6 09:39:14 2019][1697128.541665] [] ? __sb_start_write+0x58/0x110 [Wed Feb 6 09:39:14 2019][1697128.547758] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:14 2019][1697128.553504] [] ? SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:39:14 2019][1697128.559424] [] ? system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:14 2019][1697128.565783] systemd-udevd S ffffa05b72d48000 0 7984 1 0x00000000 [Wed Feb 6 09:39:14 2019][1697128.573052] Call Trace: [Wed Feb 6 09:39:14 2019][1697128.575698] [] ? kmem_cache_alloc+0x35/0x1f0 [Wed Feb 6 09:39:14 2019][1697128.581792] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1697128.586934] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:39:14 2019][1697128.594160] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Wed Feb 6 09:39:14 2019][1697128.601128] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:14 2019][1697128.607656] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:39:14 2019][1697128.612891] [] ? do_unlinkat+0xf1/0x2d0 [Wed Feb 6 09:39:14 2019][1697128.618557] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:14 2019][1697128.624303] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:39:14 2019][1697128.630049] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:14 2019][1697128.636229] kmpathd S ffffa072fc12a580 0 9274 2 0x00000000 [Wed Feb 6 09:39:14 2019][1697128.643510] Call Trace: [Wed Feb 6 09:39:14 2019][1697128.646145] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1697128.651292] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:14 2019][1697128.657131] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:14 2019][1697128.663311] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1697128.668365] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.674630] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1697128.681244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.687517] kmpath_handlerd S ffffa05b73612080 0 9276 2 0x00000000 [Wed Feb 6 09:39:14 2019][1697128.694791] Call Trace: [Wed Feb 6 09:39:14 2019][1697128.697424] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1697128.702565] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:14 2019][1697128.708406] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:14 2019][1697128.714594] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1697128.719645] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.725912] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1697128.732532] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1697128.738799] multipathd S ffffa08b76658000 0 9649 1 0x00000000 [Wed Feb 6 09:39:14 2019][1697128.746069] Call Trace: [Wed Feb 6 09:39:14 2019][1697128.748699] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1697128.753847] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:14 2019][1697128.760034] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:14 2019][1697128.765528] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:14 2019][1697128.770843] [] ? mntput+0x24/0x40 [Wed Feb 6 09:39:14 2019][1697128.775988] [] ? __fput+0x186/0x260 [Wed Feb 6 09:39:14 2019][1697128.781303] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:14 2019][1697128.786623] [] ? task_work_run+0xc0/0xe0 [Wed Feb 6 09:39:14 2019][1697128.792370] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:14 2019][1697128.798557] multipathd S ffffa06b76026180 0 9693 1 0x00000000 [Wed Feb 6 09:39:14 2019][1697128.805826] Call Trace: [Wed Feb 6 09:39:14 2019][1697128.808455] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1697128.813593] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:14 2019][1697128.819773] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:14 2019][1697128.825260] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:14 2019][1697128.830581] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:14 2019][1697128.835894] [] ? fd_install+0x25/0x30 [Wed Feb 6 09:39:14 2019][1697128.841387] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:14 2019][1697128.847567] multipathd S ffffa08b7c441900 0 9715 1 0x00000000 [Wed Feb 6 09:39:14 2019][1697128.854836] Call Trace: [Wed Feb 6 09:39:14 2019][1697128.857464] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1697128.862606] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:39:14 2019][1697128.869834] [] ? __slab_free+0x81/0x2f0 [Wed Feb 6 09:39:14 2019][1697128.875502] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:39:14 2019][1697128.881344] [] ? __pollwait+0x7f/0xf0 [Wed Feb 6 09:39:14 2019][1697128.886839] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:14 2019][1697128.893373] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:39:14 2019][1697128.899648] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:39:14 2019][1697128.905229] [] ? netlink_recvmsg+0x212/0x490 [Wed Feb 6 09:39:14 2019][1697128.911333] [] ? sock_recvmsg+0xc5/0x100 [Wed Feb 6 09:39:14 2019][1697128.917086] [] ? tracing_is_on+0x15/0x30 [Wed Feb 6 09:39:14 2019][1697128.922854] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:14 2019][1697128.928345] [] ? physflat_send_IPI_mask+0xe/0x10 [Wed Feb 6 09:39:14 2019][1697128.934791] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:39:14 2019][1697128.940804] [] ? wake_up_q+0x5b/0x80 [Wed Feb 6 09:39:14 2019][1697128.946204] [] ? futex_wake_op+0x4a5/0x610 [Wed Feb 6 09:39:14 2019][1697128.952123] [] ? do_futex+0x17a/0x5a0 [Wed Feb 6 09:39:14 2019][1697128.957611] [] ? plist_add+0x91/0xd0 [Wed Feb 6 09:39:14 2019][1697128.963020] [] ? __switch_to+0x151/0x580 [Wed Feb 6 09:39:14 2019][1697128.968773] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:39:14 2019][1697128.973999] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:14 2019][1697128.980186] multipathd S ffffa06b7610e180 0 9718 1 0x00000000 [Wed Feb 6 09:39:14 2019][1697128.987454] Call Trace: [Wed Feb 6 09:39:14 2019][1697128.990085] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:14 2019][1697128.996793] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1697129.001941] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:39:14 2019][1697129.009074] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:14 2019][1697129.015001] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:39:14 2019][1697129.022309] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:14 2019][1697129.028843] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:39:14 2019][1697129.035117] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:39:14 2019][1697129.040701] [] ? get_page_from_freelist+0x412/0xa70 [Wed Feb 6 09:39:14 2019][1697129.047408] [] ? ttwu_do_wakeup+0x19/0xe0 [Wed Feb 6 09:39:14 2019][1697129.053249] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:14 2019][1697129.058742] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:39:14 2019][1697129.065703] [] ? add_page_to_unevictable_list+0xc8/0x100 [Wed Feb 6 09:39:14 2019][1697129.072847] [] ? page_add_new_anon_rmap+0x12d/0x170 [Wed Feb 6 09:39:14 2019][1697129.079551] [] ? handle_pte_fault+0x5b8/0xd10 [Wed Feb 6 09:39:14 2019][1697129.085730] [] ? __kmalloc+0x55/0x230 [Wed Feb 6 09:39:14 2019][1697129.091219] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:14 2019][1697129.097317] [] ? __set_task_blocked+0x41/0xa0 [Wed Feb 6 09:39:14 2019][1697129.103498] [] SyS_ppoll+0xee/0x1f0 [Wed Feb 6 09:39:14 2019][1697129.108810] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:14 2019][1697129.114991] multipathd S ffffa06b7a118000 0 11652 1 0x00000000 [Wed Feb 6 09:39:14 2019][1697129.122267] Call Trace: [Wed Feb 6 09:39:14 2019][1697129.124897] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:14 2019][1697129.131605] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1697129.136753] [] do_nanosleep+0x86/0xf0 [Wed Feb 6 09:39:14 2019][1697129.142248] [] hrtimer_nanosleep+0xbb/0x180 [Wed Feb 6 09:39:14 2019][1697129.148262] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:14 2019][1697129.154180] [] ? do_nanosleep+0x72/0xf0 [Wed Feb 6 09:39:14 2019][1697129.159843] [] SyS_nanosleep+0x96/0xb0 [Wed Feb 6 09:39:14 2019][1697129.165422] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:14 2019][1697129.171610] multipathd S ffffa06b7a11b0c0 0 11653 1 0x00000000 [Wed Feb 6 09:39:14 2019][1697129.178877] Call Trace: [Wed Feb 6 09:39:14 2019][1697129.181508] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1697129.186654] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:15 2019][1697129.192835] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:15 2019][1697129.198320] [] ? mntput+0x24/0x40 [Wed Feb 6 09:39:15 2019][1697129.203461] [] ? path_openat+0x172/0x640 [Wed Feb 6 09:39:15 2019][1697129.209217] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:15 2019][1697129.214537] [] ? lockref_put_or_lock+0x48/0x80 [Wed Feb 6 09:39:15 2019][1697129.220812] [] ? mntput+0x24/0x40 [Wed Feb 6 09:39:15 2019][1697129.225960] [] ? __fput+0x186/0x260 [Wed Feb 6 09:39:15 2019][1697129.231281] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:15 2019][1697129.236604] [] ? task_work_run+0xc0/0xe0 [Wed Feb 6 09:39:15 2019][1697129.242357] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:15 2019][1697129.248543] multipathd S ffffa06b777ee180 0 16279 1 0x00000000 [Wed Feb 6 09:39:15 2019][1697129.255813] Call Trace: [Wed Feb 6 09:39:15 2019][1697129.258441] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1697129.263594] [] dm_wait_event+0xa1/0xf0 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.269952] [] ? strlcpy+0x42/0x60 [Wed Feb 6 09:39:15 2019][1697129.275187] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:15 2019][1697129.281206] [] dev_wait+0x50/0xd0 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.287133] [] ctl_ioctl+0x24e/0x550 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.293319] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:39:15 2019][1697129.300290] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.306738] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.312923] [] do_vfs_ioctl+0x3a0/0x5a0 [Wed Feb 6 09:39:15 2019][1697129.318588] [] ? __set_task_blocked+0x41/0xa0 [Wed Feb 6 09:39:15 2019][1697129.324768] [] ? __set_current_blocked+0x42/0x70 [Wed Feb 6 09:39:15 2019][1697129.331207] [] SyS_ioctl+0xa1/0xc0 [Wed Feb 6 09:39:15 2019][1697129.336436] [] ? SyS_rt_sigprocmask+0xc4/0x100 [Wed Feb 6 09:39:15 2019][1697129.342712] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:15 2019][1697129.348897] multipathd S ffffa06b777ed140 0 16358 1 0x00000000 [Wed Feb 6 09:39:15 2019][1697129.356167] Call Trace: [Wed Feb 6 09:39:15 2019][1697129.358795] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1697129.363939] [] dm_wait_event+0xa1/0xf0 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.370295] [] ? strlcpy+0x42/0x60 [Wed Feb 6 09:39:15 2019][1697129.375521] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:15 2019][1697129.381532] [] dev_wait+0x50/0xd0 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.387459] [] ctl_ioctl+0x24e/0x550 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.393643] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:39:15 2019][1697129.400606] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.407055] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.413240] [] do_vfs_ioctl+0x3a0/0x5a0 [Wed Feb 6 09:39:15 2019][1697129.418909] [] ? __set_task_blocked+0x41/0xa0 [Wed Feb 6 09:39:15 2019][1697129.425095] [] ? __set_current_blocked+0x42/0x70 [Wed Feb 6 09:39:15 2019][1697129.431535] [] SyS_ioctl+0xa1/0xc0 [Wed Feb 6 09:39:15 2019][1697129.436761] [] ? SyS_rt_sigprocmask+0xc4/0x100 [Wed Feb 6 09:39:15 2019][1697129.443028] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:15 2019][1697129.449215] multipathd S ffffa07b7a271040 0 16482 1 0x00000000 [Wed Feb 6 09:39:15 2019][1697129.456483] Call Trace: [Wed Feb 6 09:39:15 2019][1697129.459114] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1697129.464263] [] dm_wait_event+0xa1/0xf0 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.470623] [] ? strlcpy+0x42/0x60 [Wed Feb 6 09:39:15 2019][1697129.475858] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:15 2019][1697129.481875] [] dev_wait+0x50/0xd0 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.487804] [] ctl_ioctl+0x24e/0x550 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.493988] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:39:15 2019][1697129.500950] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.507399] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.513584] [] do_vfs_ioctl+0x3a0/0x5a0 [Wed Feb 6 09:39:15 2019][1697129.519252] [] ? __set_task_blocked+0x41/0xa0 [Wed Feb 6 09:39:15 2019][1697129.525441] [] ? __set_current_blocked+0x42/0x70 [Wed Feb 6 09:39:15 2019][1697129.531888] [] SyS_ioctl+0xa1/0xc0 [Wed Feb 6 09:39:15 2019][1697129.537122] [] ? SyS_rt_sigprocmask+0xc4/0x100 [Wed Feb 6 09:39:15 2019][1697129.543387] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:15 2019][1697129.549571] multipathd S ffffa07b75ef6180 0 17731 1 0x00000080 [Wed Feb 6 09:39:15 2019][1697129.556844] Call Trace: [Wed Feb 6 09:39:15 2019][1697129.559474] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1697129.564625] [] dm_wait_event+0xa1/0xf0 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.570985] [] ? strlcpy+0x42/0x60 [Wed Feb 6 09:39:15 2019][1697129.576240] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:15 2019][1697129.582255] [] dev_wait+0x50/0xd0 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.588179] [] ctl_ioctl+0x24e/0x550 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.594366] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:39:15 2019][1697129.601328] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.607777] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.613963] [] do_vfs_ioctl+0x3a0/0x5a0 [Wed Feb 6 09:39:15 2019][1697129.619630] [] ? __set_task_blocked+0x41/0xa0 [Wed Feb 6 09:39:15 2019][1697129.625810] [] ? __set_current_blocked+0x42/0x70 [Wed Feb 6 09:39:15 2019][1697129.632250] [] SyS_ioctl+0xa1/0xc0 [Wed Feb 6 09:39:15 2019][1697129.637483] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:15 2019][1697129.643662] multipathd S ffffa07b75ef1040 0 18675 1 0x00000080 [Wed Feb 6 09:39:15 2019][1697129.650933] Call Trace: [Wed Feb 6 09:39:15 2019][1697129.653561] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1697129.658711] [] dm_wait_event+0xa1/0xf0 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.665071] [] ? strlcpy+0x42/0x60 [Wed Feb 6 09:39:15 2019][1697129.670306] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:15 2019][1697129.676321] [] dev_wait+0x50/0xd0 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.682251] [] ctl_ioctl+0x24e/0x550 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.688436] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:39:15 2019][1697129.695397] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.701845] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [Wed Feb 6 09:39:15 2019][1697129.708032] [] do_vfs_ioctl+0x3a0/0x5a0 [Wed Feb 6 09:39:15 2019][1697129.713692] [] ? __set_task_blocked+0x41/0xa0 [Wed Feb 6 09:39:15 2019][1697129.719879] [] ? __set_current_blocked+0x42/0x70 [Wed Feb 6 09:39:15 2019][1697129.726320] [] SyS_ioctl+0xa1/0xc0 [Wed Feb 6 09:39:15 2019][1697129.731554] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:15 2019][1697129.737739] ccp-1-q2 S ffffa08b76628000 0 9977 2 0x00000000 [Wed Feb 6 09:39:15 2019][1697129.745009] Call Trace: [Wed Feb 6 09:39:15 2019][1697129.747639] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1697129.752780] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:15 2019][1697129.759574] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:15 2019][1697129.766283] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:15 2019][1697129.772988] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1697129.778046] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1697129.784316] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1697129.790929] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1697129.797195] ccp-1-q3 S ffffa08b772ee180 0 9979 2 0x00000000 [Wed Feb 6 09:39:15 2019][1697129.804463] Call Trace: [Wed Feb 6 09:39:15 2019][1697129.807094] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1697129.812244] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:15 2019][1697129.819039] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:15 2019][1697129.825744] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:15 2019][1697129.832451] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1697129.837504] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1697129.843770] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1697129.850384] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1697129.856658] ccp-1-q4 S ffffa08b772e9040 0 9981 2 0x00000000 [Wed Feb 6 09:39:15 2019][1697129.863927] Call Trace: [Wed Feb 6 09:39:15 2019][1697129.866557] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1697129.871706] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:15 2019][1697129.878502] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:15 2019][1697129.885209] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:15 2019][1697129.891914] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1697129.896968] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1697129.903243] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1697129.909862] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1697129.916128] ccp-2-q0 S ffffa08b772e8000 0 9990 2 0x00000000 [Wed Feb 6 09:39:15 2019][1697129.923399] Call Trace: [Wed Feb 6 09:39:15 2019][1697129.926026] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1697129.931169] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:15 2019][1697129.937964] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:15 2019][1697129.944672] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:15 2019][1697129.951377] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1697129.956431] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1697129.962705] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1697129.969327] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1697129.975601] ccp-2-q1 S ffffa08b772ea080 0 9991 2 0x00000000 [Wed Feb 6 09:39:15 2019][1697129.982877] Call Trace: [Wed Feb 6 09:39:15 2019][1697129.985508] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1697129.990659] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:15 2019][1697129.997452] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:15 2019][1697130.004159] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:15 2019][1697130.010865] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1697130.015918] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1697130.022184] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1697130.028797] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1697130.035062] ccp-2-q2 S ffffa08b772ed140 0 9992 2 0x00000000 [Wed Feb 6 09:39:15 2019][1697130.042331] Call Trace: [Wed Feb 6 09:39:15 2019][1697130.044960] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1697130.050103] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:15 2019][1697130.056898] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:15 2019][1697130.063604] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:15 2019][1697130.070311] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1697130.075364] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1697130.081647] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1697130.088259] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1697130.094526] ccp-2-q3 S ffffa08b772ec100 0 9993 2 0x00000000 [Wed Feb 6 09:39:15 2019][1697130.101795] Call Trace: [Wed Feb 6 09:39:15 2019][1697130.104425] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1697130.109576] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:15 2019][1697130.116369] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:15 2019][1697130.123078] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:15 2019][1697130.129781] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1697130.134835] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1697130.141101] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1697130.147713] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1697130.153980] ccp-2-q4 S ffffa08b76064100 0 9994 2 0x00000000 [Wed Feb 6 09:39:15 2019][1697130.161250] Call Trace: [Wed Feb 6 09:39:15 2019][1697130.163878] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1697130.169028] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:15 2019][1697130.175821] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:15 2019][1697130.182522] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697130.189227] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697130.194281] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.200547] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697130.207161] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.213435] ccp-3-q2 S ffffa08b76065140 0 9995 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697130.220703] Call Trace: [Wed Feb 6 09:39:16 2019][1697130.223331] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697130.228472] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.235267] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.241976] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697130.248684] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697130.253745] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.260017] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697130.266630] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.272898] ccp-3-q3 S ffffa08b760630c0 0 9996 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697130.280167] Call Trace: [Wed Feb 6 09:39:16 2019][1697130.282795] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697130.287937] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.294730] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.301439] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697130.308145] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697130.313206] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.319474] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697130.326093] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.332359] ccp-3-q4 S ffffa08b76061040 0 9997 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697130.339628] Call Trace: [Wed Feb 6 09:39:16 2019][1697130.342256] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697130.347399] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.354195] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.360901] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697130.367607] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697130.372662] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.378934] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697130.385546] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.391815] ccp-4-q0 S ffffa08b76066180 0 9998 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697130.399083] Call Trace: [Wed Feb 6 09:39:16 2019][1697130.401712] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697130.406855] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.413648] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.420356] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697130.427062] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697130.432116] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.438389] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697130.445001] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.451268] ccp-4-q1 S ffffa08b76062080 0 9999 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697130.458537] Call Trace: [Wed Feb 6 09:39:16 2019][1697130.461166] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697130.466317] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.473110] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.479816] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697130.486515] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697130.491568] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.497834] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697130.504447] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.510714] ccp-4-q2 S ffffa08b76060000 0 10000 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697130.517983] Call Trace: [Wed Feb 6 09:39:16 2019][1697130.520612] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697130.525762] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.532555] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.539262] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697130.545962] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697130.551022] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.557287] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697130.563901] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.570168] ccp-4-q3 S ffffa07c29758000 0 10001 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697130.577454] Call Trace: [Wed Feb 6 09:39:16 2019][1697130.580083] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697130.583067] LustreError: 17897:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549474632, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0003_UUID lock: ffffa0583a3f60c0/0x8c1de53949a12113 lrc: 3/0,1 mode: --/PW res: [0x280000e13:0xbc:0x0].0x0 bits 0x40/0x0 rrc: 126044 type: IBT flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 17897 timeout: 0 lvb_type: 0 [Wed Feb 6 09:39:16 2019][1697130.625126] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.631921] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.638628] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697130.645336] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697130.650396] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.656661] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697130.663275] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.669539] ccp-4-q4 S ffffa07c29759040 0 10002 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697130.676810] Call Trace: [Wed Feb 6 09:39:16 2019][1697130.679439] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697130.684587] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.691383] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.698092] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697130.704798] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697130.709858] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.716124] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697130.722738] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.729003] ccp-5-q2 S ffffa07c2975a080 0 10003 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697130.736272] Call Trace: [Wed Feb 6 09:39:16 2019][1697130.738900] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697130.744043] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.750838] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.757548] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697130.764254] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697130.769312] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.775578] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697130.782190] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.788458] ccp-5-q3 S ffffa07c2975b0c0 0 10004 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697130.795726] Call Trace: [Wed Feb 6 09:39:16 2019][1697130.798355] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697130.803497] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.810291] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.817001] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697130.823705] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697130.828760] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.835033] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697130.841645] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.847912] ccp-5-q4 S ffffa07c2975c100 0 10005 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697130.855179] Call Trace: [Wed Feb 6 09:39:16 2019][1697130.857809] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697130.862949] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.869746] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.876453] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697130.883158] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697130.888213] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.894478] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697130.901091] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.907358] ccp-6-q0 S ffffa07c2975d140 0 10006 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697130.914626] Call Trace: [Wed Feb 6 09:39:16 2019][1697130.917256] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697130.922403] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.929191] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.935899] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697130.942605] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697130.947658] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.953924] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697130.960537] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697130.966813] ccp-6-q1 S ffffa07c2975e180 0 10007 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697130.974088] Call Trace: [Wed Feb 6 09:39:16 2019][1697130.976717] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697130.981857] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.988653] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697130.995360] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697131.002061] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697131.007121] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697131.013387] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697131.020009] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697131.026284] ccp-6-q2 S ffffa08b79ec8000 0 10008 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697131.033560] Call Trace: [Wed Feb 6 09:39:16 2019][1697131.036190] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697131.041338] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697131.048134] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697131.054843] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697131.061547] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697131.066600] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697131.072869] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697131.079488] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697131.085754] ccp-6-q3 S ffffa08b79ec9040 0 10009 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697131.093021] Call Trace: [Wed Feb 6 09:39:16 2019][1697131.095651] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697131.100794] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697131.107587] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697131.114294] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697131.120993] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697131.126047] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697131.132320] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1697131.138933] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1697131.145199] ccp-6-q4 S ffffa08b79eca080 0 10010 2 0x00000000 [Wed Feb 6 09:39:16 2019][1697131.152475] Call Trace: [Wed Feb 6 09:39:16 2019][1697131.155105] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1697131.160245] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:16 2019][1697131.167032] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:16 2019][1697131.173742] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:16 2019][1697131.180449] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1697131.185509] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.191775] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697131.198387] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.204652] ccp-7-q2 S ffffa08b79ecb0c0 0 10011 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697131.211923] Call Trace: [Wed Feb 6 09:39:17 2019][1697131.214552] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:17 2019][1697131.220044] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697131.225188] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.231981] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.238688] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:17 2019][1697131.245388] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697131.250441] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.256705] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697131.263318] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.269583] ccp-7-q3 S ffffa08b79ecc100 0 10012 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697131.276855] Call Trace: [Wed Feb 6 09:39:17 2019][1697131.279482] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:17 2019][1697131.284968] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697131.290110] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.296903] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.303603] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:17 2019][1697131.310300] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697131.315355] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.321629] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697131.328243] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.334516] ccp-7-q4 S ffffa08b79ecd140 0 10013 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697131.341785] Call Trace: [Wed Feb 6 09:39:17 2019][1697131.344416] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697131.349565] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.356358] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.363067] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:17 2019][1697131.369772] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697131.374827] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.381101] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697131.387714] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.393990] ccp-8-q0 S ffffa08b79ece180 0 10014 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697131.401266] Call Trace: [Wed Feb 6 09:39:17 2019][1697131.403894] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697131.409037] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.415829] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.422539] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:17 2019][1697131.429244] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697131.434298] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.440564] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697131.447185] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.453449] ccp-8-q1 S ffffa08b784c0000 0 10015 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697131.460721] Call Trace: [Wed Feb 6 09:39:17 2019][1697131.463347] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697131.468490] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.475284] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.481992] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:17 2019][1697131.488697] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697131.493752] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.500028] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697131.506646] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.512914] ccp-8-q2 S ffffa08b784c1040 0 10016 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697131.520183] Call Trace: [Wed Feb 6 09:39:17 2019][1697131.522811] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697131.527954] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.534746] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.541456] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:17 2019][1697131.548160] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697131.553214] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.559481] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697131.566092] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.572358] ccp-8-q3 S ffffa08b784c2080 0 10017 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697131.579627] Call Trace: [Wed Feb 6 09:39:17 2019][1697131.582257] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697131.587405] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.594201] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.600925] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:17 2019][1697131.607624] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697131.612676] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.618942] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697131.625557] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.631830] ccp-8-q4 S ffffa08b784c30c0 0 10018 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697131.639100] Call Trace: [Wed Feb 6 09:39:17 2019][1697131.641728] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697131.646878] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.653672] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:39:17 2019][1697131.660380] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:39:17 2019][1697131.667088] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697131.672148] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.678415] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697131.685037] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.691310] kvm-irqfd-clean S ffffa08b7662d140 0 14538 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697131.698585] Call Trace: [Wed Feb 6 09:39:17 2019][1697131.701218] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697131.706366] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:17 2019][1697131.712206] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:17 2019][1697131.718387] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697131.723447] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.729714] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697131.736336] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.742608] kworker/5:1H S ffffa06b7ebea080 0 16026 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697131.749882] Call Trace: [Wed Feb 6 09:39:17 2019][1697131.752517] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:17 2019][1697131.758704] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697131.763852] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:17 2019][1697131.769606] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:17 2019][1697131.776305] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697131.781360] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.787633] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697131.794246] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.800513] kworker/32:1H S ffffa05b785fb0c0 0 16048 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697131.807793] Call Trace: [Wed Feb 6 09:39:17 2019][1697131.810425] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:17 2019][1697131.816606] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697131.821746] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:17 2019][1697131.827490] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:17 2019][1697131.834191] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697131.839243] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.845509] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697131.852123] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.858389] kworker/16:1H S ffffa05b785fd140 0 16049 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697131.865662] Call Trace: [Wed Feb 6 09:39:17 2019][1697131.868295] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:17 2019][1697131.874473] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697131.879613] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:17 2019][1697131.885360] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:17 2019][1697131.892070] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697131.897129] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.903397] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697131.910016] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.916282] kworker/9:1H S ffffa06b7ebee180 0 16051 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697131.923556] Call Trace: [Wed Feb 6 09:39:17 2019][1697131.926188] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:17 2019][1697131.932368] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697131.937508] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:17 2019][1697131.943253] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:17 2019][1697131.949952] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697131.955005] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.961271] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697131.967885] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697131.974150] kworker/38:1H S ffffa07b7415d140 0 16092 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697131.981423] Call Trace: [Wed Feb 6 09:39:17 2019][1697131.984048] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:17 2019][1697131.990228] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697131.995369] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:17 2019][1697132.001121] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:17 2019][1697132.007822] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697132.012882] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697132.019148] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697132.025761] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697132.032025] kworker/22:1H S ffffa07b74158000 0 16193 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697132.039301] Call Trace: [Wed Feb 6 09:39:17 2019][1697132.041934] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:17 2019][1697132.048113] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697132.053251] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:17 2019][1697132.059000] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:17 2019][1697132.065707] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697132.070761] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697132.077033] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697132.083647] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697132.089911] kdmflush S ffffa06b7a11e180 0 16262 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697132.097185] Call Trace: [Wed Feb 6 09:39:17 2019][1697132.099819] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697132.104958] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:17 2019][1697132.110790] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:17 2019][1697132.116971] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697132.122024] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697132.128300] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697132.134920] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697132.141196] bioset S ffffa06b7ba28000 0 16277 2 0x00000000 [Wed Feb 6 09:39:17 2019][1697132.148477] Call Trace: [Wed Feb 6 09:39:17 2019][1697132.151110] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1697132.156255] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:17 2019][1697132.162089] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:17 2019][1697132.168269] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1697132.173320] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1697132.179588] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1697132.186200] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.192467] kdmwork-253:0 S ffffa06b7ba2b0c0 0 16278 2 0x00000000 [Wed Feb 6 09:39:18 2019][1697132.199736] Call Trace: [Wed Feb 6 09:39:18 2019][1697132.202363] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697132.207504] [] kthread_worker_fn+0xf7/0x180 [Wed Feb 6 09:39:18 2019][1697132.213509] [] ? kthread_stop+0xf0/0xf0 [Wed Feb 6 09:39:18 2019][1697132.219170] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1697132.224222] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.230488] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1697132.237100] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.243366] kdmflush S ffffa06b7ba2d140 0 16353 2 0x00000000 [Wed Feb 6 09:39:18 2019][1697132.250639] Call Trace: [Wed Feb 6 09:39:18 2019][1697132.253274] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697132.258420] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:18 2019][1697132.264255] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:18 2019][1697132.270442] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1697132.275495] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.281761] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1697132.288373] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.294640] bioset S ffffa06b7ba2e180 0 16355 2 0x00000000 [Wed Feb 6 09:39:18 2019][1697132.301912] Call Trace: [Wed Feb 6 09:39:18 2019][1697132.304543] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697132.309686] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:18 2019][1697132.315527] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:18 2019][1697132.321714] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1697132.326770] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.333042] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1697132.339656] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.345928] kdmwork-253:1 S ffffa06b7ba29040 0 16356 2 0x00000000 [Wed Feb 6 09:39:18 2019][1697132.353197] Call Trace: [Wed Feb 6 09:39:18 2019][1697132.355827] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697132.360967] [] kthread_worker_fn+0xf7/0x180 [Wed Feb 6 09:39:18 2019][1697132.366972] [] ? kthread_stop+0xf0/0xf0 [Wed Feb 6 09:39:18 2019][1697132.372630] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1697132.377687] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.383960] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1697132.390571] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.396838] kworker/17:1H S ffffa06b777ea080 0 16409 2 0x00000000 [Wed Feb 6 09:39:18 2019][1697132.404111] Call Trace: [Wed Feb 6 09:39:18 2019][1697132.406745] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:18 2019][1697132.412926] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697132.418077] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:18 2019][1697132.423826] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:18 2019][1697132.430525] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1697132.435578] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.441846] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1697132.448466] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.454733] kworker/21:1H S ffffa06b777e8000 0 16411 2 0x00000000 [Wed Feb 6 09:39:18 2019][1697132.462004] Call Trace: [Wed Feb 6 09:39:18 2019][1697132.464640] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:18 2019][1697132.470827] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697132.475975] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:18 2019][1697132.481719] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:18 2019][1697132.488418] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1697132.493473] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.499738] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1697132.506352] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.512619] kworker/14:1H S ffffa07b7a2730c0 0 16416 2 0x00000000 [Wed Feb 6 09:39:18 2019][1697132.519900] Call Trace: [Wed Feb 6 09:39:18 2019][1697132.522532] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:18 2019][1697132.528713] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697132.533860] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:18 2019][1697132.539606] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:18 2019][1697132.546306] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1697132.551358] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.557625] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1697132.564245] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.570510] kdmflush S ffffa06b7a11d140 0 16477 2 0x00000000 [Wed Feb 6 09:39:18 2019][1697132.577801] Call Trace: [Wed Feb 6 09:39:18 2019][1697132.580427] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697132.585575] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:18 2019][1697132.591408] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:18 2019][1697132.597585] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1697132.602640] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.608907] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1697132.615526] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.621793] bioset S ffffa06b7a11c100 0 16479 2 0x00000000 [Wed Feb 6 09:39:18 2019][1697132.629072] Call Trace: [Wed Feb 6 09:39:18 2019][1697132.631699] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697132.636839] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:18 2019][1697132.642672] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:18 2019][1697132.648857] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1697132.653912] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.660178] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1697132.666791] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.673056] kdmwork-253:2 S ffffa07b593bb0c0 0 16480 2 0x00000000 [Wed Feb 6 09:39:18 2019][1697132.680325] Call Trace: [Wed Feb 6 09:39:18 2019][1697132.682955] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697132.688094] [] kthread_worker_fn+0xf7/0x180 [Wed Feb 6 09:39:18 2019][1697132.694100] [] ? kthread_stop+0xf0/0xf0 [Wed Feb 6 09:39:18 2019][1697132.699758] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1697132.704812] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.711079] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1697132.717693] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.723965] kworker/39:1H S ffffa08b76ccb0c0 0 16540 2 0x00000000 [Wed Feb 6 09:39:18 2019][1697132.731238] Call Trace: [Wed Feb 6 09:39:18 2019][1697132.733872] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:18 2019][1697132.740051] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697132.745190] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:18 2019][1697132.750938] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:18 2019][1697132.757646] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1697132.762707] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.768972] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1697132.775584] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.781851] kworker/18:1H S ffffa07b593b9040 0 16554 2 0x00000000 [Wed Feb 6 09:39:18 2019][1697132.789125] Call Trace: [Wed Feb 6 09:39:18 2019][1697132.791758] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:18 2019][1697132.797945] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697132.803084] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:18 2019][1697132.808832] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:18 2019][1697132.815538] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1697132.820592] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.826857] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1697132.833472] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697132.839744] auditd S ffffa07b74626180 0 16727 1 0x00000000 [Wed Feb 6 09:39:18 2019][1697132.847014] Call Trace: [Wed Feb 6 09:39:18 2019][1697132.849643] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:18 2019][1697132.856342] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697132.861483] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:39:18 2019][1697132.868624] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:18 2019][1697132.874552] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:39:18 2019][1697132.881868] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:18 2019][1697132.888404] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:39:18 2019][1697132.893637] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:18 2019][1697132.899389] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:39:18 2019][1697132.905138] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:18 2019][1697132.911324] auditd S ffffa07b775c8000 0 16728 1 0x00000000 [Wed Feb 6 09:39:18 2019][1697132.918593] Call Trace: [Wed Feb 6 09:39:18 2019][1697132.921223] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697132.926370] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:18 2019][1697132.932548] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:18 2019][1697132.938035] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:18 2019][1697132.943694] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:18 2019][1697132.949021] [] ? jbd2_complete_transaction+0x52/0xa0 [jbd2] [Wed Feb 6 09:39:18 2019][1697132.956418] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:18 2019][1697132.961732] [] ? do_fsync+0x78/0xb0 [Wed Feb 6 09:39:18 2019][1697132.967052] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:18 2019][1697132.973239] rpciod S ffffa07b593be180 0 16731 2 0x00000080 [Wed Feb 6 09:39:18 2019][1697132.980514] Call Trace: [Wed Feb 6 09:39:18 2019][1697132.983145] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697132.988286] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:18 2019][1697132.994127] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:18 2019][1697133.000312] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1697133.005366] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697133.011634] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1697133.018255] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697133.024519] xprtiod S ffffa07b593bd140 0 16732 2 0x00000080 [Wed Feb 6 09:39:18 2019][1697133.031792] Call Trace: [Wed Feb 6 09:39:18 2019][1697133.034418] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697133.039559] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:18 2019][1697133.045398] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:18 2019][1697133.051578] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1697133.056630] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697133.062897] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1697133.069510] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1697133.075775] sssd S ffffa05b7b3fc100 0 16762 1 0x00000080 [Wed Feb 6 09:39:18 2019][1697133.083045] Call Trace: [Wed Feb 6 09:39:18 2019][1697133.085673] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:18 2019][1697133.092373] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1697133.097513] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:39:18 2019][1697133.104656] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:18 2019][1697133.110582] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:39:18 2019][1697133.117888] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:18 2019][1697133.124416] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:39:18 2019][1697133.130690] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:39:18 2019][1697133.136273] [] ? sock_recvmsg+0xc5/0x100 [Wed Feb 6 09:39:18 2019][1697133.142024] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:18 2019][1697133.147510] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:18 2019][1697133.152998] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:18 2019][1697133.158483] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:18 2019][1697133.163970] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:18 2019][1697133.169465] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:18 2019][1697133.174960] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:18 2019][1697133.180455] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:18 2019][1697133.185949] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:39:19 2019][1697133.191791] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:39:19 2019][1697133.197025] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:19 2019][1697133.203204] dbus-daemon S ffffa05b7b3fd140 0 16765 1 0x00000080 [Wed Feb 6 09:39:19 2019][1697133.210474] Call Trace: [Wed Feb 6 09:39:19 2019][1697133.213101] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1697133.218241] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:39:19 2019][1697133.225461] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Wed Feb 6 09:39:19 2019][1697133.232430] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:19 2019][1697133.238964] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:39:19 2019][1697133.244197] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:19 2019][1697133.249944] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:39:19 2019][1697133.255690] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:19 2019][1697133.261867] smartd S ffffa06b7a8de180 0 16782 1 0x00000080 [Wed Feb 6 09:39:19 2019][1697133.269136] Call Trace: [Wed Feb 6 09:39:19 2019][1697133.271769] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:19 2019][1697133.278474] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1697133.283615] [] do_nanosleep+0x86/0xf0 [Wed Feb 6 09:39:19 2019][1697133.289109] [] hrtimer_nanosleep+0xbb/0x180 [Wed Feb 6 09:39:19 2019][1697133.295116] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:19 2019][1697133.301042] [] ? do_nanosleep+0x72/0xf0 [Wed Feb 6 09:39:19 2019][1697133.306701] [] SyS_nanosleep+0x96/0xb0 [Wed Feb 6 09:39:19 2019][1697133.312273] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:19 2019][1697133.318455] irqbalance S ffffa06b7a8dd140 0 16783 1 0x00000080 [Wed Feb 6 09:39:19 2019][1697133.325725] Call Trace: [Wed Feb 6 09:39:19 2019][1697133.328353] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:19 2019][1697133.335062] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1697133.340210] [] do_nanosleep+0x86/0xf0 [Wed Feb 6 09:39:19 2019][1697133.345703] [] hrtimer_nanosleep+0xbb/0x180 [Wed Feb 6 09:39:19 2019][1697133.351709] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:19 2019][1697133.357630] [] ? do_nanosleep+0x72/0xf0 [Wed Feb 6 09:39:19 2019][1697133.363297] [] SyS_nanosleep+0x96/0xb0 [Wed Feb 6 09:39:19 2019][1697133.368871] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:19 2019][1697133.375057] polkitd S ffffa06b7a8d8000 0 16785 1 0x00000080 [Wed Feb 6 09:39:19 2019][1697133.382325] Call Trace: [Wed Feb 6 09:39:19 2019][1697133.384955] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1697133.390101] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:39:19 2019][1697133.397323] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:39:19 2019][1697133.403164] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:19 2019][1697133.409689] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:39:19 2019][1697133.415957] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:39:19 2019][1697133.421539] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:19 2019][1697133.427815] [] ? __switch_to+0x151/0x580 [Wed Feb 6 09:39:19 2019][1697133.433566] [] ? update_curr+0x14c/0x1e0 [Wed Feb 6 09:39:19 2019][1697133.439312] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:19 2019][1697133.444798] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:19 2019][1697133.450285] [] ? ttwu_do_activate+0x6f/0x80 [Wed Feb 6 09:39:19 2019][1697133.456290] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:39:19 2019][1697133.462298] [] ? inotify_free_event+0xe/0x10 [Wed Feb 6 09:39:19 2019][1697133.468400] [] ? kfree+0x106/0x140 [Wed Feb 6 09:39:19 2019][1697133.473636] [] ? list_del+0xd/0x30 [Wed Feb 6 09:39:19 2019][1697133.478869] [] ? inotify_read+0x2ef/0x420 [Wed Feb 6 09:39:19 2019][1697133.484710] [] ? eventfd_ctx_read+0x67/0x1e0 [Wed Feb 6 09:39:19 2019][1697133.490812] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:39:19 2019][1697133.496047] [] ? __audit_syscall_exit+0x1e6/0x280 [Wed Feb 6 09:39:19 2019][1697133.502581] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:19 2019][1697133.508769] gmain S ffffa05b6e311040 0 16798 1 0x00000080 [Wed Feb 6 09:39:19 2019][1697133.516045] Call Trace: [Wed Feb 6 09:39:19 2019][1697133.518676] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1697133.523824] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:39:19 2019][1697133.531053] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:39:19 2019][1697133.536891] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:19 2019][1697133.543418] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:39:19 2019][1697133.549686] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:39:19 2019][1697133.555267] [] ? radix_tree_lookup_slot+0x22/0x50 [Wed Feb 6 09:39:19 2019][1697133.561802] [] ? __find_get_page+0x1e/0xa0 [Wed Feb 6 09:39:19 2019][1697133.567730] [] ? filemap_fault+0x17e/0x490 [Wed Feb 6 09:39:19 2019][1697133.573658] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:19 2019][1697133.579166] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:19 2019][1697133.584655] [] ? do_read_fault.isra.61+0x139/0x1b0 [Wed Feb 6 09:39:19 2019][1697133.591266] [] ? handle_pte_fault+0x2f4/0xd10 [Wed Feb 6 09:39:19 2019][1697133.597447] [] ? eventfd_ctx_read+0x67/0x1e0 [Wed Feb 6 09:39:19 2019][1697133.603541] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:39:19 2019][1697133.608775] [] ? __audit_syscall_exit+0x1e6/0x280 [Wed Feb 6 09:39:19 2019][1697133.615301] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:19 2019][1697133.621490] gdbus S ffffa08b765d2080 0 16800 1 0x00000080 [Wed Feb 6 09:39:19 2019][1697133.628766] Call Trace: [Wed Feb 6 09:39:19 2019][1697133.631396] [] ? __kmalloc_reserve.isra.32+0x31/0x90 [Wed Feb 6 09:39:19 2019][1697133.638191] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1697133.643336] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:39:19 2019][1697133.650558] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:39:19 2019][1697133.656415] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:39:19 2019][1697133.662250] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:19 2019][1697133.668782] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:39:19 2019][1697133.675049] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:39:19 2019][1697133.680623] [] ? sock_sendmsg+0xb6/0xf0 [Wed Feb 6 09:39:19 2019][1697133.686291] [] ? sock_recvmsg+0xc5/0x100 [Wed Feb 6 09:39:19 2019][1697133.692045] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:19 2019][1697133.697704] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:19 2019][1697133.703188] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:19 2019][1697133.708678] [] ? ___sys_sendmsg+0x3e9/0x400 [Wed Feb 6 09:39:19 2019][1697133.714691] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:19 2019][1697133.721131] [] ? pollwake+0x73/0x90 [Wed Feb 6 09:39:19 2019][1697133.726443] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:19 2019][1697133.732189] [] ? eventfd_ctx_read+0x67/0x1e0 [Wed Feb 6 09:39:19 2019][1697133.738283] [] ? __sys_sendmsg+0x85/0x90 [Wed Feb 6 09:39:19 2019][1697133.744036] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:39:19 2019][1697133.749263] [] ? __audit_syscall_exit+0x1e6/0x280 [Wed Feb 6 09:39:19 2019][1697133.755789] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:19 2019][1697133.761967] polkitd S ffffa06b79e4e180 0 16801 1 0x00000080 [Wed Feb 6 09:39:19 2019][1697133.769237] Call Trace: [Wed Feb 6 09:39:19 2019][1697133.771865] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1697133.777006] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:39:19 2019][1697133.784234] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:39:19 2019][1697133.790073] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:19 2019][1697133.796601] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:39:19 2019][1697133.802868] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:39:19 2019][1697133.808448] [] ? __enqueue_entity+0x78/0x80 [Wed Feb 6 09:39:19 2019][1697133.814464] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:19 2019][1697133.819959] [] ? resched_curr+0xae/0xc0 [Wed Feb 6 09:39:19 2019][1697133.825628] [] ? check_preempt_curr+0x80/0xa0 [Wed Feb 6 09:39:19 2019][1697133.831814] [] ? ttwu_do_wakeup+0x19/0xe0 [Wed Feb 6 09:39:19 2019][1697133.837646] [] ? ttwu_do_activate+0x6f/0x80 [Wed Feb 6 09:39:19 2019][1697133.843653] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:39:19 2019][1697133.849659] [] ? wake_up_q+0x5b/0x80 [Wed Feb 6 09:39:19 2019][1697133.855059] [] ? futex_wake+0x90/0x180 [Wed Feb 6 09:39:19 2019][1697133.860637] [] ? eventfd_ctx_read+0x67/0x1e0 [Wed Feb 6 09:39:19 2019][1697133.866734] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:39:19 2019][1697133.871968] [] ? __audit_syscall_exit+0x1e6/0x280 [Wed Feb 6 09:39:19 2019][1697133.878501] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:19 2019][1697133.884683] JS GC Helper S ffffa06b79e4d140 0 16803 1 0x00000080 [Wed Feb 6 09:39:19 2019][1697133.891957] Call Trace: [Wed Feb 6 09:39:19 2019][1697133.894586] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1697133.899726] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:19 2019][1697133.905906] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:19 2019][1697133.911392] [] ? handle_pte_fault+0x2f4/0xd10 [Wed Feb 6 09:39:19 2019][1697133.917581] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:19 2019][1697133.922907] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:19 2019][1697133.928224] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:19 2019][1697133.934410] JS Sour~ Thread S ffffa06b79e4c100 0 16806 1 0x00000080 [Wed Feb 6 09:39:19 2019][1697133.941680] Call Trace: [Wed Feb 6 09:39:19 2019][1697133.944309] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1697133.949457] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:19 2019][1697133.955637] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:19 2019][1697133.961133] [] ? lru_cache_add+0xe/0x10 [Wed Feb 6 09:39:19 2019][1697133.966800] [] ? page_add_new_anon_rmap+0xb8/0x170 [Wed Feb 6 09:39:19 2019][1697133.973420] [] ? get_futex_key+0x1c8/0x2c0 [Wed Feb 6 09:39:19 2019][1697133.979340] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:19 2019][1697133.984654] [] ? yama_task_prctl+0x24/0x110 [Wed Feb 6 09:39:19 2019][1697133.990668] [] ? strlcpy+0x42/0x60 [Wed Feb 6 09:39:19 2019][1697133.995903] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:19 2019][1697134.001224] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:19 2019][1697134.007412] polkitd S ffffa06b79e48000 0 16810 1 0x00000080 [Wed Feb 6 09:39:19 2019][1697134.014680] Call Trace: [Wed Feb 6 09:39:19 2019][1697134.017308] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1697134.022449] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:39:19 2019][1697134.029678] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:19 2019][1697134.035604] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:39:19 2019][1697134.041436] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:19 2019][1697134.047966] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:39:19 2019][1697134.054240] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:39:19 2019][1697134.059821] [] ? __alloc_pages_nodemask+0x176/0x420 [Wed Feb 6 09:39:19 2019][1697134.066526] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:19 2019][1697134.072016] [] ? physflat_send_IPI_mask+0xe/0x10 [Wed Feb 6 09:39:19 2019][1697134.078463] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:39:19 2019][1697134.084475] [] ? wake_up_q+0x5b/0x80 [Wed Feb 6 09:39:19 2019][1697134.089876] [] ? futex_wake+0x90/0x180 [Wed Feb 6 09:39:19 2019][1697134.095457] [] ? eventfd_ctx_read+0x67/0x1e0 [Wed Feb 6 09:39:19 2019][1697134.101550] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:39:19 2019][1697134.106775] [] ? __audit_syscall_exit+0x1e6/0x280 [Wed Feb 6 09:39:19 2019][1697134.113302] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:19 2019][1697134.119482] mdadm S ffffa07b74ee9040 0 16792 1 0x00000080 [Wed Feb 6 09:39:19 2019][1697134.126750] Call Trace: [Wed Feb 6 09:39:19 2019][1697134.129380] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:19 2019][1697134.136091] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1697134.141236] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:39:19 2019][1697134.148379] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:19 2019][1697134.154305] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:39:19 2019][1697134.161612] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:19 2019][1697134.168147] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:39:19 2019][1697134.174420] [] do_select+0x6d1/0x7c0 [Wed Feb 6 09:39:19 2019][1697134.179819] [] ? __alloc_pages_nodemask+0x176/0x420 [Wed Feb 6 09:39:19 2019][1697134.186520] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:39:20 2019][1697134.193488] [] ? lru_cache_add+0xe/0x10 [Wed Feb 6 09:39:20 2019][1697134.199146] [] ? page_add_new_anon_rmap+0xb8/0x170 [Wed Feb 6 09:39:20 2019][1697134.205758] [] ? handle_pte_fault+0x5b8/0xd10 [Wed Feb 6 09:39:20 2019][1697134.211940] [] ? __mem_cgroup_uncharge_common+0xcd/0x2f0 [Wed Feb 6 09:39:20 2019][1697134.219082] [] ? free_hot_cold_page_list+0x46/0xa0 [Wed Feb 6 09:39:20 2019][1697134.225701] [] ? release_pages+0x24e/0x430 [Wed Feb 6 09:39:20 2019][1697134.231638] [] core_sys_select+0x1ff/0x340 [Wed Feb 6 09:39:20 2019][1697134.237558] [] ? evict+0x115/0x180 [Wed Feb 6 09:39:20 2019][1697134.242784] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:39:20 2019][1697134.248618] [] ? d_free+0x4f/0x70 [Wed Feb 6 09:39:20 2019][1697134.253766] [] ? __dentry_kill+0x128/0x180 [Wed Feb 6 09:39:20 2019][1697134.259695] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:39:20 2019][1697134.265533] [] SyS_select+0xca/0x120 [Wed Feb 6 09:39:20 2019][1697134.270935] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:20 2019][1697134.277122] gssproxy S ffffa08b7778c100 0 16802 1 0x00000080 [Wed Feb 6 09:39:20 2019][1697134.284389] Call Trace: [Wed Feb 6 09:39:20 2019][1697134.287020] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1697134.292168] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:39:20 2019][1697134.299397] [] ? __enqueue_entity+0x78/0x80 [Wed Feb 6 09:39:20 2019][1697134.305411] [] ? enqueue_entity+0x2ef/0xbe0 [Wed Feb 6 09:39:20 2019][1697134.311424] [] ? cpumask_next_and+0x35/0x50 [Wed Feb 6 09:39:20 2019][1697134.317432] [] ? security_inode_alloc+0x24/0x30 [Wed Feb 6 09:39:20 2019][1697134.323792] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:20 2019][1697134.330319] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:39:20 2019][1697134.335554] [] ? SyS_epoll_ctl+0x130/0xc80 [Wed Feb 6 09:39:20 2019][1697134.341483] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:20 2019][1697134.347236] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:39:20 2019][1697134.352981] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:20 2019][1697134.359162] gssproxy S ffffa07b7b0e2080 0 16814 1 0x00000080 [Wed Feb 6 09:39:20 2019][1697134.366430] Call Trace: [Wed Feb 6 09:39:20 2019][1697134.369058] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1697134.374212] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:20 2019][1697134.380395] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:20 2019][1697134.385883] [] ? handle_pte_fault+0x2f4/0xd10 [Wed Feb 6 09:39:20 2019][1697134.392067] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:20 2019][1697134.397384] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:20 2019][1697134.402703] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:20 2019][1697134.408885] gssproxy S ffffa07b7b0e1040 0 16815 1 0x00000080 [Wed Feb 6 09:39:20 2019][1697134.416159] Call Trace: [Wed Feb 6 09:39:20 2019][1697134.418789] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1697134.423930] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:20 2019][1697134.430116] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:20 2019][1697134.435602] [] ? file_free+0x40/0x40 [Wed Feb 6 09:39:20 2019][1697134.441002] [] ? physflat_send_IPI_mask+0xe/0x10 [Wed Feb 6 09:39:20 2019][1697134.447441] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:39:20 2019][1697134.453448] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:20 2019][1697134.458770] [] ? call_rwsem_wake+0x1b/0x30 [Wed Feb 6 09:39:20 2019][1697134.464697] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:20 2019][1697134.470019] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:20 2019][1697134.476206] gssproxy S ffffa07b7b0e30c0 0 16816 1 0x00000080 [Wed Feb 6 09:39:20 2019][1697134.483474] Call Trace: [Wed Feb 6 09:39:20 2019][1697134.486101] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1697134.491244] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:20 2019][1697134.497433] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:20 2019][1697134.502927] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:20 2019][1697134.509202] [] ? sched_clock_cpu+0x85/0xc0 [Wed Feb 6 09:39:20 2019][1697134.515128] [] ? idle_balance+0x7f/0x250 [Wed Feb 6 09:39:20 2019][1697134.520874] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:20 2019][1697134.526189] [] ? schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1697134.531509] [] ? do_exit+0x6bb/0xa40 [Wed Feb 6 09:39:20 2019][1697134.536909] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:20 2019][1697134.542231] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:20 2019][1697134.548418] gssproxy S ffffa07b7b0e4100 0 16818 1 0x00000080 [Wed Feb 6 09:39:20 2019][1697134.555684] Call Trace: [Wed Feb 6 09:39:20 2019][1697134.558315] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1697134.563456] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:20 2019][1697134.569643] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:20 2019][1697134.575130] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:20 2019][1697134.581402] [] ? sched_clock_cpu+0x85/0xc0 [Wed Feb 6 09:39:20 2019][1697134.587342] [] ? idle_balance+0x7f/0x250 [Wed Feb 6 09:39:20 2019][1697134.593097] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:20 2019][1697134.598415] [] ? schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1697134.603729] [] ? do_exit+0x6bb/0xa40 [Wed Feb 6 09:39:20 2019][1697134.609129] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:20 2019][1697134.614450] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:20 2019][1697134.620628] gssproxy S ffffa07b7b0e5140 0 16819 1 0x00000080 [Wed Feb 6 09:39:20 2019][1697134.627897] Call Trace: [Wed Feb 6 09:39:20 2019][1697134.630526] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1697134.635666] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:20 2019][1697134.641855] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:20 2019][1697134.647349] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:20 2019][1697134.652662] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:20 2019][1697134.657974] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:20 2019][1697134.664155] rpcbind S ffffa08b76076180 0 16809 1 0x00000080 [Wed Feb 6 09:39:20 2019][1697134.671431] Call Trace: [Wed Feb 6 09:39:20 2019][1697134.674059] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:20 2019][1697134.680760] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1697134.685907] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:39:20 2019][1697134.693042] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:20 2019][1697134.698967] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:39:20 2019][1697134.706275] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:20 2019][1697134.712801] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:39:20 2019][1697134.719076] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:39:20 2019][1697134.724651] [] ? radix_tree_lookup_slot+0x22/0x50 [Wed Feb 6 09:39:20 2019][1697134.731183] [] ? __find_get_page+0x1e/0xa0 [Wed Feb 6 09:39:20 2019][1697134.737111] [] ? filemap_fault+0x17e/0x490 [Wed Feb 6 09:39:20 2019][1697134.743030] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:20 2019][1697134.748517] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:20 2019][1697134.754012] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:20 2019][1697134.759504] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:20 2019][1697134.764992] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:20 2019][1697134.770487] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:20 2019][1697134.775973] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:20 2019][1697134.781467] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:39:20 2019][1697134.787300] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:39:20 2019][1697134.792526] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:20 2019][1697134.798705] chronyd S ffffa072fc09c100 0 16811 1 0x00000080 [Wed Feb 6 09:39:20 2019][1697134.805974] Call Trace: [Wed Feb 6 09:39:20 2019][1697134.808605] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:20 2019][1697134.815312] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1697134.820457] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:39:20 2019][1697134.827592] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:20 2019][1697134.833512] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:39:20 2019][1697134.840825] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:20 2019][1697134.847354] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:39:20 2019][1697134.853628] [] do_select+0x6d1/0x7c0 [Wed Feb 6 09:39:20 2019][1697134.859036] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:20 2019][1697134.864529] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:20 2019][1697134.870015] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:20 2019][1697134.875501] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:20 2019][1697134.880988] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:20 2019][1697134.886656] [] ? schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1697134.891969] [] ? futex_wait_queue_me+0xd5/0x130 [Wed Feb 6 09:39:20 2019][1697134.898323] [] ? __check_object_size+0x1ca/0x250 [Wed Feb 6 09:39:20 2019][1697134.904770] [] core_sys_select+0x1ff/0x340 [Wed Feb 6 09:39:20 2019][1697134.910698] [] ? __d_free+0x35/0x40 [Wed Feb 6 09:39:20 2019][1697134.916019] [] ? d_free+0x60/0x70 [Wed Feb 6 09:39:20 2019][1697134.921161] [] ? __dentry_kill+0x128/0x180 [Wed Feb 6 09:39:20 2019][1697134.927089] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:39:20 2019][1697134.932929] [] SyS_select+0xca/0x120 [Wed Feb 6 09:39:20 2019][1697134.938336] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:20 2019][1697134.944515] sssd_be S ffffa07b781e6180 0 16813 16762 0x00000080 [Wed Feb 6 09:39:20 2019][1697134.951782] Call Trace: [Wed Feb 6 09:39:20 2019][1697134.954415] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:20 2019][1697134.961122] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1697134.966270] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:39:20 2019][1697134.973411] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:20 2019][1697134.979329] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:39:20 2019][1697134.986636] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:20 2019][1697134.993164] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:39:20 2019][1697134.998399] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:20 2019][1697135.004500] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:20 2019][1697135.010254] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:39:20 2019][1697135.016010] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:20 2019][1697135.022194] sssd_nss S ffffa05b7b3fa080 0 16832 16762 0x00000080 [Wed Feb 6 09:39:20 2019][1697135.029464] Call Trace: [Wed Feb 6 09:39:20 2019][1697135.032095] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:20 2019][1697135.038801] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1697135.043940] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:39:20 2019][1697135.051076] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:20 2019][1697135.057001] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:39:20 2019][1697135.064307] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:20 2019][1697135.070833] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:39:20 2019][1697135.076060] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:20 2019][1697135.082152] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:20 2019][1697135.087899] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:39:20 2019][1697135.093646] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:20 2019][1697135.099833] sssd_pam S ffffa05b7b3fb0c0 0 16833 16762 0x00000080 [Wed Feb 6 09:39:20 2019][1697135.107102] Call Trace: [Wed Feb 6 09:39:20 2019][1697135.109731] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:20 2019][1697135.116430] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1697135.121577] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:39:20 2019][1697135.128711] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:20 2019][1697135.134631] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:39:20 2019][1697135.141944] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:20 2019][1697135.148472] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:39:20 2019][1697135.153705] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:20 2019][1697135.159798] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:20 2019][1697135.165544] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:39:20 2019][1697135.171290] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:20 2019][1697135.177469] systemd-logind S ffffa05b79734100 0 16849 1 0x00000080 [Wed Feb 6 09:39:20 2019][1697135.184737] Call Trace: [Wed Feb 6 09:39:20 2019][1697135.187368] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697135.192505] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:39:21 2019][1697135.199727] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Wed Feb 6 09:39:21 2019][1697135.206695] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:21 2019][1697135.213228] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:39:21 2019][1697135.218455] [] ? SyS_epoll_ctl+0x24d/0xc80 [Wed Feb 6 09:39:21 2019][1697135.224373] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:21 2019][1697135.230123] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:39:21 2019][1697135.235875] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:21 2019][1697135.242054] mlx4 S ffffa06b777eb0c0 0 16966 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697135.249329] Call Trace: [Wed Feb 6 09:39:21 2019][1697135.251961] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697135.257102] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:21 2019][1697135.262943] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:21 2019][1697135.269130] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697135.274190] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.280457] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697135.287085] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.293353] mlx4_ib S ffffa06b777ec100 0 16981 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697135.300626] Call Trace: [Wed Feb 6 09:39:21 2019][1697135.303261] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697135.308409] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:21 2019][1697135.314249] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:21 2019][1697135.320427] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697135.325480] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.331746] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697135.338358] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.344626] mlx4_ib_mcg S ffffa06b777e9040 0 16982 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697135.351898] Call Trace: [Wed Feb 6 09:39:21 2019][1697135.354533] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697135.359679] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:21 2019][1697135.365511] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:21 2019][1697135.371690] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697135.376745] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.383012] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697135.389631] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.395897] ipoib_flush S ffffa07b7b0e6180 0 17079 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697135.403171] Call Trace: [Wed Feb 6 09:39:21 2019][1697135.405805] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697135.410945] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:21 2019][1697135.416786] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:21 2019][1697135.422973] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697135.428025] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.434292] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697135.440906] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.447170] mlx5i S ffffa07b7b0e0000 0 17081 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697135.454443] Call Trace: [Wed Feb 6 09:39:21 2019][1697135.457069] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697135.462206] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:21 2019][1697135.468042] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:21 2019][1697135.474229] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697135.479282] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.485558] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697135.492179] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.498451] ipoib_wq S ffffa06b79749040 0 17093 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697135.505726] Call Trace: [Wed Feb 6 09:39:21 2019][1697135.508359] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697135.513507] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:21 2019][1697135.519347] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:21 2019][1697135.525529] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697135.530588] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.536854] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697135.543467] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.549733] kworker/8:1H S ffffa05b785fe180 0 17097 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697135.557007] Call Trace: [Wed Feb 6 09:39:21 2019][1697135.559639] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:21 2019][1697135.565819] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697135.570959] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:21 2019][1697135.576704] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:21 2019][1697135.583405] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697135.588467] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.594739] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697135.601353] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.607618] rdma_cm S ffffa06b7ebe8000 0 17508 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697135.614892] Call Trace: [Wed Feb 6 09:39:21 2019][1697135.617526] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697135.622665] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:21 2019][1697135.628508] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:21 2019][1697135.634694] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697135.639747] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.646012] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697135.652626] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.658894] kdmflush S ffffa08b760bd140 0 17726 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697135.666173] Call Trace: [Wed Feb 6 09:39:21 2019][1697135.668806] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697135.673946] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:21 2019][1697135.679779] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:21 2019][1697135.685958] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697135.691011] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.697278] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697135.703898] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.710163] bioset S ffffa08b760bc100 0 17729 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697135.717438] Call Trace: [Wed Feb 6 09:39:21 2019][1697135.720072] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697135.725219] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:21 2019][1697135.731060] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:21 2019][1697135.737241] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697135.742299] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.748569] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697135.755189] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.761463] kdmwork-253:3 S ffffa08b760ba080 0 17730 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697135.768731] Call Trace: [Wed Feb 6 09:39:21 2019][1697135.771361] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697135.776500] [] kthread_worker_fn+0xf7/0x180 [Wed Feb 6 09:39:21 2019][1697135.782505] [] ? kthread_stop+0xf0/0xf0 [Wed Feb 6 09:39:21 2019][1697135.788164] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697135.793217] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.799485] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697135.806105] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.812369] kworker/12:1H S ffffa05b6fadc100 0 17833 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697135.819643] Call Trace: [Wed Feb 6 09:39:21 2019][1697135.822270] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:21 2019][1697135.828458] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697135.833598] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:21 2019][1697135.839352] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:21 2019][1697135.846050] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697135.851103] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.857369] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697135.863981] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.870249] kworker/19:1H S ffffa08b7662c100 0 17852 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697135.877530] Call Trace: [Wed Feb 6 09:39:21 2019][1697135.880163] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:21 2019][1697135.886344] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697135.891491] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:21 2019][1697135.897236] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:21 2019][1697135.903936] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697135.908989] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.915255] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697135.921867] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.928132] kworker/28:1H S ffffa05b6fade180 0 18016 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697135.935405] Call Trace: [Wed Feb 6 09:39:21 2019][1697135.938031] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:21 2019][1697135.944210] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697135.949351] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:21 2019][1697135.955100] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:21 2019][1697135.961804] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697135.966858] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.973132] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697135.979746] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697135.986018] kworker/1:1H S ffffa06b7a348000 0 18109 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697135.993292] Call Trace: [Wed Feb 6 09:39:21 2019][1697135.995927] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:21 2019][1697136.002113] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697136.007254] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:21 2019][1697136.013007] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:21 2019][1697136.019707] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697136.024761] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697136.031035] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697136.037655] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697136.043921] kworker/20:1H S ffffa05b78991040 0 18110 2 0x00000080 [Wed Feb 6 09:39:21 2019][1697136.051194] Call Trace: [Wed Feb 6 09:39:21 2019][1697136.053830] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:21 2019][1697136.060015] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697136.065157] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:21 2019][1697136.070911] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:21 2019][1697136.077619] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1697136.082680] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697136.088946] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1697136.095560] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1697136.101835] sshd S ffffa06b7f4ae180 0 18216 1 0x00000080 [Wed Feb 6 09:39:21 2019][1697136.109112] Call Trace: [Wed Feb 6 09:39:21 2019][1697136.111739] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1697136.116880] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:39:21 2019][1697136.124109] [] ? dequeue_entity+0x11c/0x5e0 [Wed Feb 6 09:39:21 2019][1697136.130121] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:39:21 2019][1697136.135956] [] ? __pollwait+0x7f/0xf0 [Wed Feb 6 09:39:21 2019][1697136.141451] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:21 2019][1697136.147983] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:39:21 2019][1697136.154252] [] do_select+0x6d1/0x7c0 [Wed Feb 6 09:39:21 2019][1697136.159660] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:21 2019][1697136.165152] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:21 2019][1697136.170654] [] ? __alloc_skb+0xa1/0x2d0 [Wed Feb 6 09:39:21 2019][1697136.176317] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:39:21 2019][1697136.183286] [] ? cpumask_any_but+0x39/0x50 [Wed Feb 6 09:39:22 2019][1697136.189214] [] ? flush_tlb_page+0x38/0xa0 [Wed Feb 6 09:39:22 2019][1697136.195054] [] ? lru_cache_add+0xe/0x10 [Wed Feb 6 09:39:22 2019][1697136.200724] [] ? page_add_new_anon_rmap+0xb8/0x170 [Wed Feb 6 09:39:22 2019][1697136.207345] [] ? wp_page_copy.isra.71+0x3ab/0x560 [Wed Feb 6 09:39:22 2019][1697136.213878] [] ? unlock_page+0x2b/0x30 [Wed Feb 6 09:39:22 2019][1697136.219451] [] ? do_wp_page+0x19e/0x720 [Wed Feb 6 09:39:22 2019][1697136.225120] [] core_sys_select+0x1ff/0x340 [Wed Feb 6 09:39:22 2019][1697136.231046] [] ? handle_mm_fault+0x39d/0x9b0 [Wed Feb 6 09:39:22 2019][1697136.237139] [] SyS_select+0xca/0x120 [Wed Feb 6 09:39:22 2019][1697136.242540] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:22 2019][1697136.248728] collectd S ffffa06b7f4ad140 0 18219 1 0x00000080 [Wed Feb 6 09:39:22 2019][1697136.255994] Call Trace: [Wed Feb 6 09:39:22 2019][1697136.258624] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:22 2019][1697136.265322] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1697136.270464] [] do_nanosleep+0x86/0xf0 [Wed Feb 6 09:39:22 2019][1697136.275959] [] hrtimer_nanosleep+0xbb/0x180 [Wed Feb 6 09:39:22 2019][1697136.281972] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:22 2019][1697136.287891] [] ? do_nanosleep+0x72/0xf0 [Wed Feb 6 09:39:22 2019][1697136.293551] [] SyS_nanosleep+0x96/0xb0 [Wed Feb 6 09:39:22 2019][1697136.299126] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:22 2019][1697136.305312] writer#0 S ffffa07b74ace180 0 18226 1 0x00000080 [Wed Feb 6 09:39:22 2019][1697136.312581] Call Trace: [Wed Feb 6 09:39:22 2019][1697136.315210] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1697136.320351] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:22 2019][1697136.326536] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:22 2019][1697136.332026] [] ? physflat_send_IPI_mask+0xe/0x10 [Wed Feb 6 09:39:22 2019][1697136.338471] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:39:22 2019][1697136.344479] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:22 2019][1697136.349799] [] ? mntput+0x24/0x40 [Wed Feb 6 09:39:22 2019][1697136.354939] [] ? __fput+0x186/0x260 [Wed Feb 6 09:39:22 2019][1697136.360259] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:22 2019][1697136.365573] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:22 2019][1697136.371774] writer#1 S ffffa07b74aca080 0 18227 1 0x00000080 [Wed Feb 6 09:39:22 2019][1697136.379047] Call Trace: [Wed Feb 6 09:39:22 2019][1697136.381674] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1697136.386813] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:22 2019][1697136.392994] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:22 2019][1697136.398481] [] ? ttwu_do_wakeup+0xb5/0xe0 [Wed Feb 6 09:39:22 2019][1697136.404321] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:39:22 2019][1697136.410328] [] ? get_futex_key+0x1c8/0x2c0 [Wed Feb 6 09:39:22 2019][1697136.416255] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:22 2019][1697136.421569] [] ? mntput+0x24/0x40 [Wed Feb 6 09:39:22 2019][1697136.426715] [] ? __fput+0x186/0x260 [Wed Feb 6 09:39:22 2019][1697136.432030] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:22 2019][1697136.437350] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:22 2019][1697136.443539] writer#2 S ffffa07b74ac8000 0 18228 1 0x00000080 [Wed Feb 6 09:39:22 2019][1697136.450808] Call Trace: [Wed Feb 6 09:39:22 2019][1697136.453435] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1697136.458577] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:22 2019][1697136.464764] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:22 2019][1697136.470249] [] ? ttwu_do_wakeup+0xb5/0xe0 [Wed Feb 6 09:39:22 2019][1697136.476084] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:39:22 2019][1697136.482097] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:22 2019][1697136.487417] [] ? mntput+0x24/0x40 [Wed Feb 6 09:39:22 2019][1697136.492558] [] ? __fput+0x186/0x260 [Wed Feb 6 09:39:22 2019][1697136.497870] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:22 2019][1697136.503185] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:22 2019][1697136.509385] writer#3 S ffffa07b74acd140 0 18229 1 0x00000080 [Wed Feb 6 09:39:22 2019][1697136.516658] Call Trace: [Wed Feb 6 09:39:22 2019][1697136.519288] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1697136.524434] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:22 2019][1697136.530613] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:22 2019][1697136.536105] [] ? ttwu_do_wakeup+0xb5/0xe0 [Wed Feb 6 09:39:22 2019][1697136.541942] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:39:22 2019][1697136.547957] [] ? get_futex_key+0x1c8/0x2c0 [Wed Feb 6 09:39:22 2019][1697136.553882] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:22 2019][1697136.559198] [] ? mntput+0x24/0x40 [Wed Feb 6 09:39:22 2019][1697136.564344] [] ? __fput+0x186/0x260 [Wed Feb 6 09:39:22 2019][1697136.569658] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:22 2019][1697136.574971] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:22 2019][1697136.581171] writer#4 S ffffa07b74ac9040 0 18230 1 0x00000080 [Wed Feb 6 09:39:22 2019][1697136.588445] Call Trace: [Wed Feb 6 09:39:22 2019][1697136.591074] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1697136.596222] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:22 2019][1697136.602411] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:22 2019][1697136.607906] [] ? ttwu_do_wakeup+0xb5/0xe0 [Wed Feb 6 09:39:22 2019][1697136.613747] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:39:22 2019][1697136.619760] [] ? get_futex_key+0x1c8/0x2c0 [Wed Feb 6 09:39:22 2019][1697136.625681] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:22 2019][1697136.631000] [] ? mntput+0x24/0x40 [Wed Feb 6 09:39:22 2019][1697136.636140] [] ? __fput+0x186/0x260 [Wed Feb 6 09:39:22 2019][1697136.641453] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:22 2019][1697136.646768] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:22 2019][1697136.652953] reader#0 S ffffa08b78670000 0 18231 1 0x00000080 [Wed Feb 6 09:39:22 2019][1697136.660224] Call Trace: [Wed Feb 6 09:39:22 2019][1697136.662852] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:22 2019][1697136.669559] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1697136.674699] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:22 2019][1697136.680880] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:22 2019][1697136.686374] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:22 2019][1697136.692302] [] ? futex_wait_queue_me+0xa4/0x130 [Wed Feb 6 09:39:22 2019][1697136.698664] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:22 2019][1697136.703986] [] ? do_munmap+0x327/0x480 [Wed Feb 6 09:39:22 2019][1697136.709565] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:22 2019][1697136.714880] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:22 2019][1697136.721066] reader#1 S ffffa08b78672080 0 18232 1 0x00000080 [Wed Feb 6 09:39:22 2019][1697136.728333] Call Trace: [Wed Feb 6 09:39:22 2019][1697136.730962] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:22 2019][1697136.737679] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1697136.742822] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:22 2019][1697136.749010] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:22 2019][1697136.754503] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:22 2019][1697136.760430] [] ? futex_wait_queue_me+0xa4/0x130 [Wed Feb 6 09:39:22 2019][1697136.766783] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:22 2019][1697136.772096] [] ? mntput+0x24/0x40 [Wed Feb 6 09:39:22 2019][1697136.777236] [] ? __fput+0x186/0x260 [Wed Feb 6 09:39:22 2019][1697136.782549] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:22 2019][1697136.787870] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:22 2019][1697136.794059] reader#2 S ffffa08b78671040 0 18233 1 0x00000080 [Wed Feb 6 09:39:22 2019][1697136.801334] Call Trace: [Wed Feb 6 09:39:22 2019][1697136.803963] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:22 2019][1697136.810665] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1697136.815813] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:22 2019][1697136.822000] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:22 2019][1697136.827485] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:22 2019][1697136.833405] [] ? futex_wait_queue_me+0xa4/0x130 [Wed Feb 6 09:39:22 2019][1697136.839758] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:22 2019][1697136.845071] [] ? do_munmap+0x327/0x480 [Wed Feb 6 09:39:22 2019][1697136.850654] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:22 2019][1697136.855974] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:22 2019][1697136.862161] reader#3 S ffffa08b786730c0 0 18234 1 0x00000080 [Wed Feb 6 09:39:22 2019][1697136.869429] Call Trace: [Wed Feb 6 09:39:22 2019][1697136.872060] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:22 2019][1697136.878766] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1697136.883908] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:22 2019][1697136.890094] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:22 2019][1697136.895580] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:22 2019][1697136.901500] [] ? futex_wait_queue_me+0xa4/0x130 [Wed Feb 6 09:39:22 2019][1697136.907860] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:22 2019][1697136.913175] [] ? do_munmap+0x327/0x480 [Wed Feb 6 09:39:22 2019][1697136.918755] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:22 2019][1697136.924069] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:22 2019][1697136.930254] reader#4 S ffffa08b78674100 0 18236 1 0x00000080 [Wed Feb 6 09:39:22 2019][1697136.937524] Call Trace: [Wed Feb 6 09:39:22 2019][1697136.940155] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1697136.945302] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:22 2019][1697136.951481] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:22 2019][1697136.956967] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:22 2019][1697136.962280] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:22 2019][1697136.967594] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:22 2019][1697136.973780] rsyslogd S ffffa06b7f4a8000 0 18221 1 0x00000080 [Wed Feb 6 09:39:22 2019][1697136.981050] Call Trace: [Wed Feb 6 09:39:22 2019][1697136.983679] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:22 2019][1697136.990385] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1697136.995526] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:39:22 2019][1697137.002660] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:22 2019][1697137.008586] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:39:22 2019][1697137.015892] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:22 2019][1697137.022419] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:39:22 2019][1697137.028685] [] do_select+0x6d1/0x7c0 [Wed Feb 6 09:39:22 2019][1697137.034086] [] ? _raw_read_unlock_bh+0x1b/0x20 [Wed Feb 6 09:39:22 2019][1697137.040360] [] ? __skb_try_recv_datagram+0x172/0x300 [Wed Feb 6 09:39:22 2019][1697137.047153] [] ? task_scan_min+0x3e/0x60 [Wed Feb 6 09:39:22 2019][1697137.052898] [] ? task_scan_max+0x27/0x40 [Wed Feb 6 09:39:22 2019][1697137.058645] [] ? task_numa_fault+0x8d0/0xbb0 [Wed Feb 6 09:39:22 2019][1697137.064739] [] ? do_numa_page+0x1be/0x250 [Wed Feb 6 09:39:22 2019][1697137.070581] [] ? handle_pte_fault+0x316/0xd10 [Wed Feb 6 09:39:22 2019][1697137.076769] [] ? should_numa_migrate_memory+0x5b/0x150 [Wed Feb 6 09:39:22 2019][1697137.083737] [] ? mpol_misplaced+0x18c/0x1e0 [Wed Feb 6 09:39:22 2019][1697137.089752] [] core_sys_select+0x1ff/0x340 [Wed Feb 6 09:39:22 2019][1697137.095679] [] ? handle_mm_fault+0x39d/0x9b0 [Wed Feb 6 09:39:22 2019][1697137.101773] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:39:22 2019][1697137.107612] [] SyS_select+0xca/0x120 [Wed Feb 6 09:39:22 2019][1697137.113013] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:22 2019][1697137.119199] in:imjournal S ffffa07b78f6d140 0 18241 1 0x00000080 [Wed Feb 6 09:39:22 2019][1697137.126469] Call Trace: [Wed Feb 6 09:39:22 2019][1697137.129099] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:22 2019][1697137.135809] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1697137.140953] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:39:22 2019][1697137.148086] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:22 2019][1697137.154007] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:39:22 2019][1697137.161320] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:22 2019][1697137.167846] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:39:22 2019][1697137.174115] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:39:22 2019][1697137.179711] [] ? jbd2_journal_dirty_metadata+0xd1/0x260 [jbd2] [Wed Feb 6 09:39:22 2019][1697137.187367] [] ? security_inode_permission+0x22/0x30 [Wed Feb 6 09:39:23 2019][1697137.194160] [] ? __enqueue_entity+0x78/0x80 [Wed Feb 6 09:39:23 2019][1697137.200167] [] ? link_path_walk+0x81/0x8b0 [Wed Feb 6 09:39:23 2019][1697137.206093] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:39:23 2019][1697137.211580] [] ? complete_walk+0x50/0xe0 [Wed Feb 6 09:39:23 2019][1697137.217324] [] ? path_lookupat+0x92/0x8b0 [Wed Feb 6 09:39:23 2019][1697137.223160] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:39:23 2019][1697137.229001] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:39:23 2019][1697137.235015] [] ? wake_up_q+0x5b/0x80 [Wed Feb 6 09:39:23 2019][1697137.240413] [] ? futex_wake_op+0x4a5/0x610 [Wed Feb 6 09:39:23 2019][1697137.246335] [] ? do_futex+0x17a/0x5a0 [Wed Feb 6 09:39:23 2019][1697137.251829] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:39:23 2019][1697137.257669] [] SyS_ppoll+0x1d3/0x1f0 [Wed Feb 6 09:39:23 2019][1697137.263072] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:23 2019][1697137.269258] rs:main Q:Reg S ffffa05b753c4100 0 18242 1 0x00000080 [Wed Feb 6 09:39:23 2019][1697137.276533] Call Trace: [Wed Feb 6 09:39:23 2019][1697137.279165] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1697137.284313] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:23 2019][1697137.290501] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:23 2019][1697137.295997] [] ? generic_file_aio_write+0x77/0xa0 [Wed Feb 6 09:39:23 2019][1697137.302537] [] ? ext4_file_write+0xd2/0x1e0 [ext4] [Wed Feb 6 09:39:23 2019][1697137.309152] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:23 2019][1697137.314472] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:23 2019][1697137.319785] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:23 2019][1697137.325972] nfsiod S ffffa06b79748000 0 18306 2 0x00000080 [Wed Feb 6 09:39:23 2019][1697137.333246] Call Trace: [Wed Feb 6 09:39:23 2019][1697137.335876] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1697137.341021] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:23 2019][1697137.346861] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:23 2019][1697137.353048] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:23 2019][1697137.358110] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1697137.364384] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:23 2019][1697137.370995] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1697137.377261] kworker/10:1H S ffffa07b74ee8000 0 18372 2 0x00000080 [Wed Feb 6 09:39:23 2019][1697137.384536] Call Trace: [Wed Feb 6 09:39:23 2019][1697137.387169] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:23 2019][1697137.393346] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1697137.398489] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:23 2019][1697137.404242] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:23 2019][1697137.410941] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:23 2019][1697137.415994] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1697137.422262] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:23 2019][1697137.428882] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1697137.435148] kworker/2:1H S ffffa07b74eeb0c0 0 18373 2 0x00000080 [Wed Feb 6 09:39:23 2019][1697137.442420] Call Trace: [Wed Feb 6 09:39:23 2019][1697137.445053] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:23 2019][1697137.451233] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1697137.456373] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:23 2019][1697137.462118] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:23 2019][1697137.468819] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:23 2019][1697137.473880] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1697137.480147] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:23 2019][1697137.486767] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1697137.493033] nfsv4.1-svc S ffffa07b74eec100 0 18374 2 0x00000080 [Wed Feb 6 09:39:23 2019][1697137.500301] Call Trace: [Wed Feb 6 09:39:23 2019][1697137.502929] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1697137.508080] [] nfs41_callback_svc+0x15d/0x1c0 [nfsv4] [Wed Feb 6 09:39:23 2019][1697137.514962] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:23 2019][1697137.520981] [] ? nfs4_callback_svc+0x60/0x60 [nfsv4] [Wed Feb 6 09:39:23 2019][1697137.527772] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:23 2019][1697137.532831] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1697137.539108] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:23 2019][1697137.545726] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1697137.551994] crond S ffffa06b7a8da080 0 18411 1 0x00000080 [Wed Feb 6 09:39:23 2019][1697137.559261] Call Trace: [Wed Feb 6 09:39:23 2019][1697137.561892] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:23 2019][1697137.568600] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1697137.573749] [] do_nanosleep+0x86/0xf0 [Wed Feb 6 09:39:23 2019][1697137.579256] [] hrtimer_nanosleep+0xbb/0x180 [Wed Feb 6 09:39:23 2019][1697137.585264] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:23 2019][1697137.591184] [] ? do_nanosleep+0x72/0xf0 [Wed Feb 6 09:39:23 2019][1697137.596846] [] SyS_nanosleep+0x96/0xb0 [Wed Feb 6 09:39:23 2019][1697137.602425] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:23 2019][1697137.608604] agetty S ffffa06b76384100 0 18466 1 0x00000080 [Wed Feb 6 09:39:23 2019][1697137.615873] Call Trace: [Wed Feb 6 09:39:23 2019][1697137.618505] [] ? __do_fault.isra.59+0x8a/0x100 [Wed Feb 6 09:39:23 2019][1697137.624778] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1697137.629924] [] schedule_timeout+0x221/0x2d0 [Wed Feb 6 09:39:23 2019][1697137.635945] [] ? get_work_pool+0x3e/0x70 [Wed Feb 6 09:39:23 2019][1697137.641694] [] ? flush_work+0x13b/0x190 [Wed Feb 6 09:39:23 2019][1697137.647356] [] ? do_read_fault.isra.61+0x139/0x1b0 [Wed Feb 6 09:39:23 2019][1697137.653977] [] ? serial8250_start_tx+0xa5/0xe0 [Wed Feb 6 09:39:23 2019][1697137.660253] [] ? set_room+0x68/0xc0 [Wed Feb 6 09:39:23 2019][1697137.665573] [] n_tty_read+0x27c/0x930 [Wed Feb 6 09:39:23 2019][1697137.671067] [] ? ldsem_down_read+0x3b/0x220 [Wed Feb 6 09:39:23 2019][1697137.677080] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:23 2019][1697137.682827] [] tty_read+0x90/0x100 [Wed Feb 6 09:39:23 2019][1697137.688053] [] vfs_read+0x9f/0x170 [Wed Feb 6 09:39:23 2019][1697137.693279] [] SyS_read+0x7f/0xf0 [Wed Feb 6 09:39:23 2019][1697137.698420] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:23 2019][1697137.704607] agetty S ffffa06b76382080 0 18468 1 0x00000080 [Wed Feb 6 09:39:23 2019][1697137.711876] Call Trace: [Wed Feb 6 09:39:23 2019][1697137.714507] [] ? bit_cursor+0x663/0x6a0 [Wed Feb 6 09:39:23 2019][1697137.720171] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1697137.725311] [] schedule_timeout+0x221/0x2d0 [Wed Feb 6 09:39:23 2019][1697137.731319] [] ? get_work_pool+0x3e/0x70 [Wed Feb 6 09:39:23 2019][1697137.737073] [] ? flush_work+0x13b/0x190 [Wed Feb 6 09:39:23 2019][1697137.742745] [] ? set_room+0x68/0xc0 [Wed Feb 6 09:39:23 2019][1697137.748061] [] n_tty_read+0x27c/0x930 [Wed Feb 6 09:39:23 2019][1697137.753550] [] ? ldsem_down_read+0x3b/0x220 [Wed Feb 6 09:39:23 2019][1697137.759564] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:23 2019][1697137.765319] [] tty_read+0x90/0x100 [Wed Feb 6 09:39:23 2019][1697137.770554] [] vfs_read+0x9f/0x170 [Wed Feb 6 09:39:23 2019][1697137.775787] [] SyS_read+0x7f/0xf0 [Wed Feb 6 09:39:23 2019][1697137.780935] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:23 2019][1697137.787125] master S ffffa07b749a4100 0 18601 1 0x00000080 [Wed Feb 6 09:39:23 2019][1697137.794425] Call Trace: [Wed Feb 6 09:39:23 2019][1697137.797055] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:23 2019][1697137.803758] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1697137.808906] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:39:23 2019][1697137.816046] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:23 2019][1697137.821974] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:39:23 2019][1697137.829287] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:23 2019][1697137.835818] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:39:23 2019][1697137.841052] [] ? hrtimer_start+0x12/0x20 [Wed Feb 6 09:39:23 2019][1697137.846803] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:23 2019][1697137.852548] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:39:23 2019][1697137.858296] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:23 2019][1697137.864483] slapd S ffffa08b78675140 0 18606 1 0x00000080 [Wed Feb 6 09:39:23 2019][1697137.871752] Call Trace: [Wed Feb 6 09:39:23 2019][1697137.874380] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1697137.879521] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:23 2019][1697137.885706] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:23 2019][1697137.891196] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:23 2019][1697137.896690] [] ? sched_slice.isra.58+0x66/0xc0 [Wed Feb 6 09:39:23 2019][1697137.902965] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:23 2019][1697137.908286] [] ? resched_curr+0xae/0xc0 [Wed Feb 6 09:39:23 2019][1697137.913953] [] ? check_preempt_curr+0x80/0xa0 [Wed Feb 6 09:39:23 2019][1697137.920145] [] ? wake_up_new_task+0x131/0x1a0 [Wed Feb 6 09:39:23 2019][1697137.926327] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:23 2019][1697137.931646] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:23 2019][1697137.937831] slapd S ffffa08b77eee180 0 18607 1 0x00000080 [Wed Feb 6 09:39:23 2019][1697137.945108] Call Trace: [Wed Feb 6 09:39:23 2019][1697137.947737] [] ? pipe_read+0x2e5/0x400 [Wed Feb 6 09:39:23 2019][1697137.953319] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1697137.958467] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:39:23 2019][1697137.965694] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Wed Feb 6 09:39:23 2019][1697137.972663] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:23 2019][1697137.979197] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:39:23 2019][1697137.984431] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:23 2019][1697137.990180] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:39:23 2019][1697137.995931] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:23 2019][1697138.002117] slapd S ffffa05b7389c100 0 18608 1 0x00000080 [Wed Feb 6 09:39:23 2019][1697138.009390] Call Trace: [Wed Feb 6 09:39:23 2019][1697138.012019] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1697138.017165] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:39:23 2019][1697138.023347] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:39:23 2019][1697138.028842] [] ? __sb_end_write+0x31/0x60 [Wed Feb 6 09:39:23 2019][1697138.034683] [] ? pipe_write+0x48f/0x540 [Wed Feb 6 09:39:23 2019][1697138.040350] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:39:23 2019][1697138.045677] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:39:23 2019][1697138.050995] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:23 2019][1697138.057182] qmgr S ffffa08b7864e180 0 18610 18601 0x00000080 [Wed Feb 6 09:39:23 2019][1697138.064457] Call Trace: [Wed Feb 6 09:39:23 2019][1697138.067087] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:23 2019][1697138.073794] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1697138.078945] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:39:23 2019][1697138.086087] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:23 2019][1697138.092013] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:39:23 2019][1697138.099329] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:23 2019][1697138.105863] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:39:23 2019][1697138.111099] [] ? mntput+0x24/0x40 [Wed Feb 6 09:39:23 2019][1697138.116246] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:23 2019][1697138.122000] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:39:23 2019][1697138.127756] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:23 2019][1697138.133944] kdmflush S ffffa08b79e7b0c0 0 18670 2 0x00000080 [Wed Feb 6 09:39:23 2019][1697138.141249] Call Trace: [Wed Feb 6 09:39:23 2019][1697138.143883] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1697138.149025] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:23 2019][1697138.154866] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:23 2019][1697138.161053] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:23 2019][1697138.166119] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1697138.172389] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:23 2019][1697138.179012] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1697138.185285] bioset S ffffa08b79e79040 0 18672 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697138.192565] Call Trace: [Wed Feb 6 09:39:24 2019][1697138.195200] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697138.200348] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:24 2019][1697138.206189] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:24 2019][1697138.212376] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697138.217438] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.223713] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697138.230335] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.236610] kdmwork-253:4 S ffffa08b79e7c100 0 18673 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697138.243886] Call Trace: [Wed Feb 6 09:39:24 2019][1697138.246516] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697138.251666] [] kthread_worker_fn+0xf7/0x180 [Wed Feb 6 09:39:24 2019][1697138.257678] [] ? kthread_stop+0xf0/0xf0 [Wed Feb 6 09:39:24 2019][1697138.263347] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697138.268409] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.274682] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697138.281296] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.287568] kworker/36:1H S ffffa05b72d4b0c0 0 18810 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697138.294845] Call Trace: [Wed Feb 6 09:39:24 2019][1697138.297476] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697138.303664] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697138.308813] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697138.314569] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697138.321276] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697138.326338] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.332614] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697138.339233] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.345506] kworker/37:1H S ffffa06b79fc1040 0 18813 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697138.352781] Call Trace: [Wed Feb 6 09:39:24 2019][1697138.355412] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697138.361601] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697138.366748] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697138.372497] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697138.379203] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697138.384265] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.390540] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697138.397163] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.403435] kworker/11:1H S ffffa08b79e7e180 0 18821 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697138.410716] Call Trace: [Wed Feb 6 09:39:24 2019][1697138.413351] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697138.419539] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697138.424687] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697138.430445] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697138.437149] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697138.442204] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.448477] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697138.455101] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.461373] kworker/26:1H S ffffa07b5739e180 0 18900 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697138.468645] Call Trace: [Wed Feb 6 09:39:24 2019][1697138.471279] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697138.477469] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697138.482618] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697138.488370] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697138.495079] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697138.500138] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.506408] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697138.513026] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.519302] kworker/33:1H S ffffa06b79fc5140 0 18959 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697138.526574] Call Trace: [Wed Feb 6 09:39:24 2019][1697138.529212] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697138.535396] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697138.540543] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697138.546291] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697138.552997] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697138.558053] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.564326] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697138.570951] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.577223] kworker/34:1H S ffffa07b5739a080 0 18978 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697138.584504] Call Trace: [Wed Feb 6 09:39:24 2019][1697138.587138] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697138.593324] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697138.598465] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697138.604218] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697138.610919] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697138.615981] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.622254] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697138.628876] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.635150] kworker/44:1H S ffffa05b72d4d140 0 19058 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697138.642432] Call Trace: [Wed Feb 6 09:39:24 2019][1697138.645067] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697138.651254] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697138.656402] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697138.662156] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697138.668866] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697138.673950] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.680217] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697138.686831] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.693107] kworker/6:1H S ffffa07b5739d140 0 19274 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697138.700397] Call Trace: [Wed Feb 6 09:39:24 2019][1697138.703031] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697138.709218] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697138.714365] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697138.720122] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697138.726828] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697138.731890] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.738164] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697138.744787] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.751062] kworker/3:1H S ffffa08b79e78000 0 19395 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697138.758341] Call Trace: [Wed Feb 6 09:39:24 2019][1697138.760976] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697138.767162] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697138.772302] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697138.778049] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697138.784757] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697138.789818] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.796095] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697138.802715] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.808988] kworker/4:1H S ffffa04ca972d140 0 19440 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697138.816262] Call Trace: [Wed Feb 6 09:39:24 2019][1697138.818895] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697138.825084] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697138.830235] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697138.835985] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697138.842694] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697138.847755] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.854032] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697138.860653] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.866928] kworker/42:1H S ffffa07b7a125140 0 20448 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697138.874207] Call Trace: [Wed Feb 6 09:39:24 2019][1697138.876842] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697138.883031] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697138.888176] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697138.893925] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697138.900630] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697138.905684] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.911961] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697138.918595] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.924865] kworker/40:1H S ffffa05b78994100 0 39599 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697138.932144] Call Trace: [Wed Feb 6 09:39:24 2019][1697138.934778] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697138.940967] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697138.946119] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697138.951876] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697138.958579] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697138.963639] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.969912] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697138.976528] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697138.982801] kworker/24:1H S ffffa05b78996180 0 39600 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697138.990082] Call Trace: [Wed Feb 6 09:39:24 2019][1697138.992718] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697138.998904] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697139.004055] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697139.009809] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697139.016515] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697139.021578] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697139.027852] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697139.034474] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697139.040747] kworker/13:1H S ffffa06b7974a080 0 39650 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697139.048028] Call Trace: [Wed Feb 6 09:39:24 2019][1697139.050662] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697139.056850] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697139.062000] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697139.067752] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697139.074452] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697139.079513] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697139.085781] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697139.092403] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697139.098674] kworker/46:1H S ffffa07b57231040 0 40156 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697139.105949] Call Trace: [Wed Feb 6 09:39:24 2019][1697139.108583] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697139.114769] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697139.119910] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697139.125667] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697139.132372] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1697139.137438] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697139.143710] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1697139.150330] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1697139.156603] kworker/27:1H S ffffa08b78676180 0 40527 2 0x00000080 [Wed Feb 6 09:39:24 2019][1697139.163879] Call Trace: [Wed Feb 6 09:39:24 2019][1697139.166511] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:24 2019][1697139.172698] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1697139.177837] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:24 2019][1697139.183584] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:24 2019][1697139.190283] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697139.195336] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.201602] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697139.208215] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.214481] kworker/43:1H S ffffa08b79f28000 0 40567 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697139.221754] Call Trace: [Wed Feb 6 09:39:25 2019][1697139.224388] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:25 2019][1697139.230575] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697139.235713] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:25 2019][1697139.241461] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:25 2019][1697139.248168] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697139.253221] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.259490] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697139.266112] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.272385] kworker/30:1H S ffffa07b75d9b0c0 0 40711 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697139.279665] Call Trace: [Wed Feb 6 09:39:25 2019][1697139.282299] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:25 2019][1697139.288489] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697139.293635] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:25 2019][1697139.299390] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:25 2019][1697139.306100] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697139.311160] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.317435] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697139.324056] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.330329] kworker/23:1H S ffffa08b76002080 0 43868 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697139.337604] Call Trace: [Wed Feb 6 09:39:25 2019][1697139.340237] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:25 2019][1697139.346427] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697139.351576] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:25 2019][1697139.357326] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:25 2019][1697139.364027] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697139.369086] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.375353] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697139.381966] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.388232] kworker/7:1H S ffffa08b7662b0c0 0 44035 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697139.395505] Call Trace: [Wed Feb 6 09:39:25 2019][1697139.398139] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:25 2019][1697139.404327] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697139.409475] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:25 2019][1697139.415222] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:25 2019][1697139.421928] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697139.426983] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.433257] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697139.439879] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.446153] kworker/15:1H S ffffa08b7662a080 0 44173 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697139.453432] Call Trace: [Wed Feb 6 09:39:25 2019][1697139.456057] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:25 2019][1697139.462239] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697139.467385] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:25 2019][1697139.473132] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:25 2019][1697139.479832] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697139.484894] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.491169] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697139.497787] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.504055] kworker/25:1H S ffffa06b78078000 0 46340 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697139.511330] Call Trace: [Wed Feb 6 09:39:25 2019][1697139.513961] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:25 2019][1697139.520140] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697139.525280] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:25 2019][1697139.531026] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:25 2019][1697139.537725] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697139.542778] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.549045] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697139.555657] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.561923] kworker/29:1H S ffffa06b7807d140 0 46362 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697139.569195] Call Trace: [Wed Feb 6 09:39:25 2019][1697139.571821] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:25 2019][1697139.578009] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697139.583147] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:25 2019][1697139.588894] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:25 2019][1697139.595594] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697139.600647] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.606921] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697139.613535] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.619808] kworker/47:1H S ffffa08b772eb0c0 0 46363 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697139.627081] Call Trace: [Wed Feb 6 09:39:25 2019][1697139.629715] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:25 2019][1697139.635904] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697139.641051] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:25 2019][1697139.646798] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:25 2019][1697139.653504] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697139.658557] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.664824] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697139.671445] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.677711] kworker/31:1H S ffffa08b765d6180 0 46420 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697139.684985] Call Trace: [Wed Feb 6 09:39:25 2019][1697139.687618] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:25 2019][1697139.693798] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697139.698945] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:25 2019][1697139.704691] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:25 2019][1697139.711390] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697139.716443] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.722709] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697139.729322] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.735590] kworker/41:1H S ffffa06b776a6180 0 49497 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697139.742873] Call Trace: [Wed Feb 6 09:39:25 2019][1697139.745504] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:25 2019][1697139.751692] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697139.756839] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:25 2019][1697139.762585] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:25 2019][1697139.769283] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697139.774338] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.780602] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697139.787215] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.793483] kworker/6:2 S ffffa07b76b4a080 0 72368 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697139.800779] Call Trace: [Wed Feb 6 09:39:25 2019][1697139.803407] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:25 2019][1697139.809592] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697139.814733] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:25 2019][1697139.820487] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:25 2019][1697139.827187] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697139.832240] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.838507] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697139.845132] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.851402] kworker/45:1H S ffffa06b77cbc100 0 117225 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697139.858762] Call Trace: [Wed Feb 6 09:39:25 2019][1697139.861396] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:25 2019][1697139.867584] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697139.872732] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:25 2019][1697139.878486] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:25 2019][1697139.885185] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697139.890239] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.896512] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697139.903132] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.909401] kworker/36:1 S ffffa05b7b3fe180 0 124458 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697139.916768] Call Trace: [Wed Feb 6 09:39:25 2019][1697139.919402] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:25 2019][1697139.925581] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697139.930728] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:25 2019][1697139.936476] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:25 2019][1697139.943198] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697139.948253] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.954518] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697139.961138] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697139.967406] cfs_rh_00 S ffffa07b75ef4100 0 126358 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697139.974762] Call Trace: [Wed Feb 6 09:39:25 2019][1697139.977392] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697139.982554] [] cfs_wi_scheduler+0x30c/0x460 [libcfs] [Wed Feb 6 09:39:25 2019][1697139.989344] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:25 2019][1697139.995365] [] ? cfs_wi_sched_create+0x680/0x680 [libcfs] [Wed Feb 6 09:39:25 2019][1697140.002588] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697140.007649] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697140.013924] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697140.020543] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697140.026821] cfs_rh_01 S ffffa07b75ef30c0 0 126359 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697140.034192] Call Trace: [Wed Feb 6 09:39:25 2019][1697140.036819] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697140.041964] [] cfs_wi_scheduler+0x30c/0x460 [libcfs] [Wed Feb 6 09:39:25 2019][1697140.048752] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:25 2019][1697140.054766] [] ? cfs_wi_sched_create+0x680/0x680 [libcfs] [Wed Feb 6 09:39:25 2019][1697140.061990] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697140.067048] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697140.073314] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697140.079929] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697140.086203] cfs_rh_02 S ffffa07b75ef5140 0 126360 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697140.093566] Call Trace: [Wed Feb 6 09:39:25 2019][1697140.096194] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697140.101340] [] cfs_wi_scheduler+0x30c/0x460 [libcfs] [Wed Feb 6 09:39:25 2019][1697140.108134] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:25 2019][1697140.114151] [] ? cfs_wi_sched_create+0x680/0x680 [libcfs] [Wed Feb 6 09:39:25 2019][1697140.121373] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1697140.126433] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697140.132701] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1697140.139321] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1697140.145587] cfs_rh_03 S ffffa08b784c5140 0 126361 2 0x00000080 [Wed Feb 6 09:39:25 2019][1697140.152942] Call Trace: [Wed Feb 6 09:39:25 2019][1697140.155573] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1697140.160724] [] cfs_wi_scheduler+0x30c/0x460 [libcfs] [Wed Feb 6 09:39:25 2019][1697140.167515] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:26 2019][1697140.173534] [] ? cfs_wi_sched_create+0x680/0x680 [libcfs] [Wed Feb 6 09:39:26 2019][1697140.180756] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697140.185811] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.192086] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697140.198706] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.204973] clio S ffffa06b76108000 0 126374 2 0x00000080 [Wed Feb 6 09:39:26 2019][1697140.212341] Call Trace: [Wed Feb 6 09:39:26 2019][1697140.214973] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1697140.220112] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:26 2019][1697140.225946] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:26 2019][1697140.232137] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697140.237195] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.243462] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697140.250083] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.256349] ptlrpc_hr00_000 S ffffa06b7610d140 0 126379 2 0x00000080 [Wed Feb 6 09:39:26 2019][1697140.263704] Call Trace: [Wed Feb 6 09:39:26 2019][1697140.266334] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1697140.272436] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1697140.277631] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.284248] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1697140.290033] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.296997] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697140.302056] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.308324] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697140.314947] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.321220] ptlrpc_hr00_001 S ffffa06b7610b0c0 0 126380 2 0x00000080 [Wed Feb 6 09:39:26 2019][1697140.328575] Call Trace: [Wed Feb 6 09:39:26 2019][1697140.331202] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1697140.337296] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1697140.342469] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.349083] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1697140.354862] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.361825] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697140.366886] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.373159] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697140.379772] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.386036] ptlrpc_hr00_002 S ffffa06b76109040 0 126381 2 0x00000080 [Wed Feb 6 09:39:26 2019][1697140.393393] Call Trace: [Wed Feb 6 09:39:26 2019][1697140.396023] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1697140.402123] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1697140.407293] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.413911] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1697140.419688] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.426653] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697140.431714] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.437993] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697140.444611] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.450885] ptlrpc_hr00_003 S ffffa08b7582b0c0 0 126382 2 0x00000080 [Wed Feb 6 09:39:26 2019][1697140.458248] Call Trace: [Wed Feb 6 09:39:26 2019][1697140.460878] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1697140.466976] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1697140.472150] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.478772] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1697140.484552] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.491516] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697140.496575] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.502852] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697140.509470] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.515738] ptlrpc_hr00_004 S ffffa08b7582a080 0 126383 2 0x00000080 [Wed Feb 6 09:39:26 2019][1697140.523103] Call Trace: [Wed Feb 6 09:39:26 2019][1697140.525731] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1697140.531830] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1697140.537001] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.543619] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1697140.549403] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.556368] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697140.561430] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.567706] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697140.574324] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.580592] ptlrpc_hr00_005 S ffffa08b7582c100 0 126384 2 0x00000080 [Wed Feb 6 09:39:26 2019][1697140.587955] Call Trace: [Wed Feb 6 09:39:26 2019][1697140.590583] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1697140.596686] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1697140.601863] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.608481] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1697140.614267] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.621228] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697140.626281] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.632556] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697140.639169] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.645443] ptlrpc_hr01_000 S ffffa08b7582d140 0 126386 2 0x00000080 [Wed Feb 6 09:39:26 2019][1697140.652799] Call Trace: [Wed Feb 6 09:39:26 2019][1697140.655428] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1697140.661531] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1697140.666706] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.673326] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1697140.679110] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.686075] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697140.691138] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.697412] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697140.704032] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.710306] ptlrpc_hr01_001 S ffffa08b7864c100 0 126387 2 0x00000080 [Wed Feb 6 09:39:26 2019][1697140.717660] Call Trace: [Wed Feb 6 09:39:26 2019][1697140.720292] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1697140.726393] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1697140.731568] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.738190] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1697140.743972] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.750939] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697140.756000] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.762273] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697140.768897] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.775169] ptlrpc_hr01_002 S ffffa08b78648000 0 126388 2 0x00000080 [Wed Feb 6 09:39:26 2019][1697140.782532] Call Trace: [Wed Feb 6 09:39:26 2019][1697140.785163] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1697140.791265] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1697140.796442] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.803062] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1697140.808841] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.815808] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697140.820867] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.827141] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697140.833755] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.840022] ptlrpc_hr01_003 S ffffa08b7864b0c0 0 126389 2 0x00000080 [Wed Feb 6 09:39:26 2019][1697140.847385] Call Trace: [Wed Feb 6 09:39:26 2019][1697140.850017] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1697140.856116] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1697140.861286] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.867902] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1697140.873677] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.880642] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697140.885697] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.891973] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697140.898594] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.904867] ptlrpc_hr01_004 S ffffa08b78649040 0 126390 2 0x00000080 [Wed Feb 6 09:39:26 2019][1697140.912239] Call Trace: [Wed Feb 6 09:39:26 2019][1697140.914868] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1697140.920961] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1697140.926142] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.932766] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1697140.938548] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.945515] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697140.950577] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.956849] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697140.963463] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697140.969737] ptlrpc_hr01_005 S ffffa08b7864d140 0 126391 2 0x00000080 [Wed Feb 6 09:39:26 2019][1697140.977093] Call Trace: [Wed Feb 6 09:39:26 2019][1697140.979722] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1697140.985824] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1697140.991001] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697140.997620] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1697141.003402] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697141.010367] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697141.015429] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697141.021695] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697141.028317] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697141.034591] ptlrpc_hr02_000 S ffffa08b7864a080 0 126392 2 0x00000080 [Wed Feb 6 09:39:26 2019][1697141.041955] Call Trace: [Wed Feb 6 09:39:26 2019][1697141.044584] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1697141.050686] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1697141.055877] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697141.062498] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1697141.068273] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697141.075237] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697141.080291] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697141.086566] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697141.093187] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697141.099451] ptlrpc_hr02_001 S ffffa08b7665c100 0 126393 2 0x00000080 [Wed Feb 6 09:39:26 2019][1697141.106809] Call Trace: [Wed Feb 6 09:39:26 2019][1697141.109438] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1697141.115539] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1697141.120710] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697141.127325] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1697141.133109] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:26 2019][1697141.140073] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1697141.145128] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697141.151400] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1697141.158014] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1697141.164281] ptlrpc_hr02_002 S ffffa08b7665b0c0 0 126394 2 0x00000080 [Wed Feb 6 09:39:27 2019][1697141.171636] Call Trace: [Wed Feb 6 09:39:27 2019][1697141.174266] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:27 2019][1697141.180366] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1697141.185533] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.192154] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1697141.197938] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.204905] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1697141.209962] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.216230] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1697141.222853] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.229128] ptlrpc_hr02_003 S ffffa08b76659040 0 126395 2 0x00000080 [Wed Feb 6 09:39:27 2019][1697141.236492] Call Trace: [Wed Feb 6 09:39:27 2019][1697141.239120] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:27 2019][1697141.245220] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1697141.250400] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.257015] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1697141.262800] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.269765] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1697141.274826] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.281101] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1697141.287723] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.293997] ptlrpc_hr02_004 S ffffa08b7665a080 0 126396 2 0x00000080 [Wed Feb 6 09:39:27 2019][1697141.301368] Call Trace: [Wed Feb 6 09:39:27 2019][1697141.303999] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:27 2019][1697141.310101] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1697141.315279] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.321897] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1697141.327677] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.334645] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1697141.339706] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.345983] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1697141.352602] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.358877] ptlrpc_hr02_005 S ffffa08b7665d140 0 126397 2 0x00000080 [Wed Feb 6 09:39:27 2019][1697141.366240] Call Trace: [Wed Feb 6 09:39:27 2019][1697141.368869] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:27 2019][1697141.374970] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1697141.380141] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.386756] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1697141.392535] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.399497] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1697141.404553] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.410827] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1697141.417445] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.423712] ptlrpc_hr03_000 S ffffa08b7665e180 0 126398 2 0x00000080 [Wed Feb 6 09:39:27 2019][1697141.431067] Call Trace: [Wed Feb 6 09:39:27 2019][1697141.433696] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:27 2019][1697141.439791] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1697141.444969] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.451586] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1697141.457368] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.464336] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1697141.469395] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.475670] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1697141.482290] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.488566] ptlrpc_hr03_001 S ffffa08b77788000 0 126400 2 0x00000080 [Wed Feb 6 09:39:27 2019][1697141.495948] Call Trace: [Wed Feb 6 09:39:27 2019][1697141.498576] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:27 2019][1697141.504678] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1697141.509854] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.516476] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1697141.522259] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.529225] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1697141.534283] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.540558] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1697141.547179] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.553452] ptlrpc_hr03_002 S ffffa08b7778e180 0 126401 2 0x00000080 [Wed Feb 6 09:39:27 2019][1697141.560815] Call Trace: [Wed Feb 6 09:39:27 2019][1697141.563447] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:27 2019][1697141.569547] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1697141.574735] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.581355] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1697141.587134] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.594101] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1697141.599155] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.605431] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1697141.612051] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.618325] ptlrpc_hr03_003 S ffffa08b7778b0c0 0 126402 2 0x00000080 [Wed Feb 6 09:39:27 2019][1697141.625687] Call Trace: [Wed Feb 6 09:39:27 2019][1697141.628320] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:27 2019][1697141.634417] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1697141.639589] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.646208] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1697141.651990] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.658955] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1697141.664017] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.670293] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1697141.676914] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.683188] ptlrpc_hr03_004 S ffffa08b77789040 0 126403 2 0x00000080 [Wed Feb 6 09:39:27 2019][1697141.690558] Call Trace: [Wed Feb 6 09:39:27 2019][1697141.693191] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:27 2019][1697141.699291] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1697141.704468] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.711086] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1697141.716870] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.723834] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1697141.728898] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.735172] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1697141.741792] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.748068] ptlrpc_hr03_005 S ffffa08b7778d140 0 126404 2 0x00000080 [Wed Feb 6 09:39:27 2019][1697141.755431] Call Trace: [Wed Feb 6 09:39:27 2019][1697141.758062] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:27 2019][1697141.764163] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1697141.769347] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.775967] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1697141.781750] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1697141.788715] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1697141.793774] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.800043] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1697141.806663] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.812939] kiblnd_connd S ffffa08b760be180 0 126419 2 0x00000080 [Wed Feb 6 09:39:27 2019][1697141.820300] Call Trace: [Wed Feb 6 09:39:27 2019][1697141.822930] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1697141.828077] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:27 2019][1697141.834086] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:27 2019][1697141.840628] [] kiblnd_connd+0x60c/0xa00 [ko2iblnd] [Wed Feb 6 09:39:27 2019][1697141.847250] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1697141.853026] [] ? kiblnd_check_conns+0x870/0x870 [ko2iblnd] [Wed Feb 6 09:39:27 2019][1697141.860336] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1697141.865399] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.871672] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1697141.878295] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.884568] kiblnd_sd_00_00 S ffffa08b760bb0c0 0 126421 2 0x00000080 [Wed Feb 6 09:39:27 2019][1697141.891932] Call Trace: [Wed Feb 6 09:39:27 2019][1697141.894562] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1697141.899713] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:27 2019][1697141.906766] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:27 2019][1697141.913043] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:27 2019][1697141.918709] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1697141.924466] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:27 2019][1697141.931342] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1697141.936393] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.942662] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1697141.949282] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697141.955548] kiblnd_sd_00_01 S ffffa08b760b9040 0 126422 2 0x00000080 [Wed Feb 6 09:39:27 2019][1697141.962903] Call Trace: [Wed Feb 6 09:39:27 2019][1697141.965532] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1697141.970676] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:27 2019][1697141.977728] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:27 2019][1697141.984000] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:27 2019][1697141.989661] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1697141.995410] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:27 2019][1697142.002288] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1697142.007340] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697142.013607] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1697142.020219] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697142.026484] kiblnd_sd_00_02 S ffffa08b76000000 0 126423 2 0x00000080 [Wed Feb 6 09:39:27 2019][1697142.033841] Call Trace: [Wed Feb 6 09:39:27 2019][1697142.036469] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1697142.041613] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:27 2019][1697142.048666] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:27 2019][1697142.054939] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:27 2019][1697142.060599] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1697142.066362] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:27 2019][1697142.073233] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1697142.078287] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697142.084553] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1697142.091166] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697142.097430] kiblnd_sd_00_03 S ffffa08b76006180 0 126424 2 0x00000080 [Wed Feb 6 09:39:27 2019][1697142.104789] Call Trace: [Wed Feb 6 09:39:27 2019][1697142.107415] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1697142.112561] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:27 2019][1697142.119611] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:27 2019][1697142.125886] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:27 2019][1697142.131553] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1697142.137306] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:27 2019][1697142.144180] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1697142.149234] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697142.155508] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1697142.162119] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1697142.168387] kiblnd_sd_01_00 S ffffa08b77a5c100 0 126425 2 0x00000080 [Wed Feb 6 09:39:28 2019][1697142.175741] Call Trace: [Wed Feb 6 09:39:28 2019][1697142.178372] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1697142.183523] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.190575] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:28 2019][1697142.196847] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:28 2019][1697142.202509] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1697142.208283] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.215162] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1697142.220215] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.226490] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1697142.233112] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.239385] kiblnd_sd_01_01 S ffffa07c294fa080 0 126426 2 0x00000080 [Wed Feb 6 09:39:28 2019][1697142.246741] Call Trace: [Wed Feb 6 09:39:28 2019][1697142.249369] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1697142.254514] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.261563] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:28 2019][1697142.267831] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:28 2019][1697142.273499] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1697142.279255] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.286133] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1697142.291185] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.297454] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1697142.304074] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.310341] kiblnd_sd_01_02 S ffffa07c29442080 0 126427 2 0x00000080 [Wed Feb 6 09:39:28 2019][1697142.317697] Call Trace: [Wed Feb 6 09:39:28 2019][1697142.320324] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1697142.325466] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.332518] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:28 2019][1697142.338786] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:28 2019][1697142.344453] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1697142.350203] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.357079] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1697142.362132] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.368399] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1697142.375011] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.381278] kiblnd_sd_01_03 S ffffa08b7a82e180 0 126428 2 0x00000080 [Wed Feb 6 09:39:28 2019][1697142.388633] Call Trace: [Wed Feb 6 09:39:28 2019][1697142.391264] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1697142.396414] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.403466] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:28 2019][1697142.409741] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:28 2019][1697142.415407] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1697142.421158] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.428034] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1697142.433088] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.439352] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1697142.445967] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.452233] kiblnd_sd_02_00 S ffffa08b7a828000 0 126429 2 0x00000080 [Wed Feb 6 09:39:28 2019][1697142.459587] Call Trace: [Wed Feb 6 09:39:28 2019][1697142.462217] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1697142.467361] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.474410] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:28 2019][1697142.480676] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:28 2019][1697142.486335] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1697142.492085] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.498963] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1697142.504016] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.510284] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1697142.516904] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.523170] kiblnd_sd_02_01 S ffffa08b7a829040 0 126430 2 0x00000080 [Wed Feb 6 09:39:28 2019][1697142.530525] Call Trace: [Wed Feb 6 09:39:28 2019][1697142.533155] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1697142.538297] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.545350] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:28 2019][1697142.551622] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:28 2019][1697142.557282] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1697142.563040] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.569919] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1697142.574970] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.581254] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1697142.587867] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.594134] kiblnd_sd_02_02 S ffffa08b7a82a080 0 126431 2 0x00000080 [Wed Feb 6 09:39:28 2019][1697142.601489] Call Trace: [Wed Feb 6 09:39:28 2019][1697142.604117] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1697142.609260] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.616313] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:28 2019][1697142.622586] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:28 2019][1697142.628247] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1697142.633994] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.640872] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1697142.645927] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.652202] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1697142.658823] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.665097] kiblnd_sd_02_03 S ffffa08b7a82b0c0 0 126432 2 0x00000080 [Wed Feb 6 09:39:28 2019][1697142.672452] Call Trace: [Wed Feb 6 09:39:28 2019][1697142.675082] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1697142.680233] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.687284] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:28 2019][1697142.693559] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:28 2019][1697142.699219] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1697142.704967] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.711845] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1697142.716900] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.723174] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1697142.729796] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.736084] kiblnd_sd_03_00 S ffffa08b7a82c100 0 126433 2 0x00000080 [Wed Feb 6 09:39:28 2019][1697142.743443] Call Trace: [Wed Feb 6 09:39:28 2019][1697142.746072] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1697142.751223] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.758275] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:28 2019][1697142.764547] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:28 2019][1697142.770209] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1697142.775966] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.782845] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1697142.787905] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.794171] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1697142.800785] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.807050] kiblnd_sd_03_01 S ffffa08b7662e180 0 126434 2 0x00000080 [Wed Feb 6 09:39:28 2019][1697142.814407] Call Trace: [Wed Feb 6 09:39:28 2019][1697142.817036] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1697142.822186] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.829238] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:28 2019][1697142.835512] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:28 2019][1697142.841174] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1697142.846931] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.853807] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1697142.858862] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.865134] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1697142.871748] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.878015] kiblnd_sd_03_02 S ffffa08b76629040 0 126435 2 0x00000080 [Wed Feb 6 09:39:28 2019][1697142.885379] Call Trace: [Wed Feb 6 09:39:28 2019][1697142.888006] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1697142.893154] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.900208] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:28 2019][1697142.906488] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:28 2019][1697142.912157] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1697142.917911] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.924789] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1697142.929841] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.936106] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1697142.942721] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697142.948995] kiblnd_sd_03_03 S ffffa05b78992080 0 126436 2 0x00000080 [Wed Feb 6 09:39:28 2019][1697142.956350] Call Trace: [Wed Feb 6 09:39:28 2019][1697142.958979] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1697142.964121] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.971173] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:28 2019][1697142.977447] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:39:28 2019][1697142.983109] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1697142.988866] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:39:28 2019][1697142.995742] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1697143.000797] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697143.007070] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1697143.013683] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697143.019948] monitor_thread S ffffa05b78995140 0 126437 2 0x00000080 [Wed Feb 6 09:39:28 2019][1697143.027306] Call Trace: [Wed Feb 6 09:39:28 2019][1697143.029933] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1697143.035074] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:28 2019][1697143.041090] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:28 2019][1697143.047623] [] ? prepare_to_wait+0x56/0x90 [Wed Feb 6 09:39:28 2019][1697143.053556] [] lnet_monitor_thread+0x492/0x560 [lnet] [Wed Feb 6 09:39:28 2019][1697143.060433] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:28 2019][1697143.066445] [] ? lnet_recover_peer_nis+0x670/0x670 [lnet] [Wed Feb 6 09:39:28 2019][1697143.073666] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1697143.078720] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697143.084996] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1697143.091616] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697143.097882] lnet_discovery S ffffa05b7b3f9040 0 126439 2 0x00000080 [Wed Feb 6 09:39:28 2019][1697143.105245] Call Trace: [Wed Feb 6 09:39:28 2019][1697143.107874] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1697143.113013] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:28 2019][1697143.119021] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:28 2019][1697143.125554] [] ? prepare_to_wait+0x56/0x90 [Wed Feb 6 09:39:28 2019][1697143.131485] [] lnet_peer_discovery+0xdf/0x1150 [lnet] [Wed Feb 6 09:39:28 2019][1697143.138364] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:28 2019][1697143.144381] [] ? lnet_peer_merge_data+0xde0/0xde0 [lnet] [Wed Feb 6 09:39:28 2019][1697143.151520] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1697143.156572] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1697143.162840] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1697143.169461] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.175733] ptlrpcd_rcv S ffffa05b7c10d140 0 126440 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697143.183090] Call Trace: [Wed Feb 6 09:39:29 2019][1697143.185749] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697143.192453] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697143.197640] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.203651] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1697143.209428] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.216130] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1697143.221184] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.227450] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1697143.234064] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.240337] ptlrpcd_00_00 S ffffa05b7c10a080 0 126441 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697143.247694] Call Trace: [Wed Feb 6 09:39:29 2019][1697143.250341] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697143.257046] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697143.262217] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.268228] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1697143.274010] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.280716] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1697143.285770] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.292043] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1697143.298658] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.304932] ptlrpcd_00_01 S ffffa05b7c108000 0 126442 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697143.312295] Call Trace: [Wed Feb 6 09:39:29 2019][1697143.314942] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697143.321648] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697143.326821] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.332829] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1697143.338605] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.345310] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1697143.350363] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.356638] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1697143.363259] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.369533] ptlrpcd_00_02 S ffffa04ca9672080 0 126443 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697143.376888] Call Trace: [Wed Feb 6 09:39:29 2019][1697143.379538] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697143.386242] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697143.391413] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.397423] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1697143.403209] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.409913] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1697143.414972] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.421239] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1697143.427853] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.434125] ptlrpcd_00_03 S ffffa05b7c5ea080 0 126444 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697143.441481] Call Trace: [Wed Feb 6 09:39:29 2019][1697143.444129] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697143.450837] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697143.456014] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.462024] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1697143.467799] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.474505] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1697143.479558] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.485825] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1697143.492444] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.498711] ptlrpcd_00_04 S ffffa05b7c5ec100 0 126445 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697143.506067] Call Trace: [Wed Feb 6 09:39:29 2019][1697143.508714] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697143.515422] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697143.520597] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.526610] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1697143.532387] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.539089] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1697143.544142] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.550408] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1697143.557021] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.563287] ptlrpcd_00_05 S ffffa05b7c5ed140 0 126446 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697143.570643] Call Trace: [Wed Feb 6 09:39:29 2019][1697143.573292] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697143.580012] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697143.585186] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.591195] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1697143.596979] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.603682] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1697143.608737] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.615012] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1697143.621631] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.627896] ptlrpcd_00_06 S ffffa05b7c5eb0c0 0 126447 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697143.635255] Call Trace: [Wed Feb 6 09:39:29 2019][1697143.637900] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697143.644607] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697143.649779] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.655788] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1697143.661564] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.668267] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1697143.673319] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.679588] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1697143.686208] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.692475] ptlrpcd_00_07 S ffffa05c29a8e180 0 126448 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697143.699830] Call Trace: [Wed Feb 6 09:39:29 2019][1697143.702478] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697143.709185] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697143.714361] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.720373] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1697143.726149] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.732852] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1697143.737907] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.744201] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1697143.750819] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.757084] ptlrpcd_00_08 S ffffa05c298a30c0 0 126449 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697143.764440] Call Trace: [Wed Feb 6 09:39:29 2019][1697143.767087] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697143.773785] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697143.778955] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.784969] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1697143.790750] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.797456] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1697143.802517] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.808792] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1697143.815413] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.821679] ptlrpcd_00_09 S ffffa06b7807c100 0 126450 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697143.829035] Call Trace: [Wed Feb 6 09:39:29 2019][1697143.831682] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697143.838390] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697143.843564] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.849576] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1697143.855352] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.862059] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1697143.867119] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.873384] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1697143.879999] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.886272] ptlrpcd_00_10 S ffffa06b78079040 0 126451 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697143.893628] Call Trace: [Wed Feb 6 09:39:29 2019][1697143.896274] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697143.902981] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697143.908151] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.914166] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1697143.919959] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.926659] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1697143.931711] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.937980] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1697143.944600] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697143.950865] ptlrpcd_00_11 S ffffa06b7807a080 0 126452 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697143.958220] Call Trace: [Wed Feb 6 09:39:29 2019][1697143.960867] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697143.967577] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697143.972752] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.978764] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1697143.984539] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697143.991244] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1697143.996311] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697144.002582] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1697144.009203] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697144.015477] ptlrpcd_01_00 S ffffa06b77cbb0c0 0 126453 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697144.022857] Call Trace: [Wed Feb 6 09:39:29 2019][1697144.025505] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697144.032211] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697144.037382] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697144.043393] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1697144.049179] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697144.055881] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1697144.060933] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697144.067201] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1697144.073822] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697144.080096] ptlrpcd_01_01 S ffffa06b77cba080 0 126454 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697144.087451] Call Trace: [Wed Feb 6 09:39:29 2019][1697144.090101] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697144.096807] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697144.101983] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697144.107994] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1697144.113772] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697144.120475] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1697144.125536] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697144.131804] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1697144.138422] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1697144.144690] ptlrpcd_01_02 S ffffa06b77cbd140 0 126455 2 0x00000080 [Wed Feb 6 09:39:29 2019][1697144.152047] Call Trace: [Wed Feb 6 09:39:29 2019][1697144.154692] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:29 2019][1697144.161399] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1697144.166568] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:29 2019][1697144.172580] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697144.178363] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.185068] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1697144.190122] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.196397] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1697144.203016] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.209284] ptlrpcd_01_03 S ffffa06b77cbe180 0 126456 2 0x00000080 [Wed Feb 6 09:39:30 2019][1697144.216638] Call Trace: [Wed Feb 6 09:39:30 2019][1697144.219284] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:30 2019][1697144.225983] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1697144.231153] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.237167] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697144.242948] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.249653] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1697144.254706] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.260973] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1697144.267591] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.273859] ptlrpcd_01_04 S ffffa06b77cb9040 0 126457 2 0x00000080 [Wed Feb 6 09:39:30 2019][1697144.281216] Call Trace: [Wed Feb 6 09:39:30 2019][1697144.283861] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:30 2019][1697144.290569] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1697144.295738] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.301750] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697144.307532] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.314238] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1697144.319292] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.325566] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1697144.332185] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.338451] ptlrpcd_01_05 S ffffa06b77cb8000 0 126458 2 0x00000080 [Wed Feb 6 09:39:30 2019][1697144.345808] Call Trace: [Wed Feb 6 09:39:30 2019][1697144.348455] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:30 2019][1697144.355167] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1697144.360338] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.366352] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697144.372134] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.378838] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1697144.383891] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.390158] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1697144.396770] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.403038] ptlrpcd_01_06 S ffffa07b7ed8d140 0 126459 2 0x00000080 [Wed Feb 6 09:39:30 2019][1697144.410392] Call Trace: [Wed Feb 6 09:39:30 2019][1697144.413040] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:30 2019][1697144.419748] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1697144.424923] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.430935] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697144.436710] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.443417] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1697144.448476] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.454744] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1697144.461364] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.467629] ptlrpcd_01_07 S ffffa07b7ed8a080 0 126460 2 0x00000080 [Wed Feb 6 09:39:30 2019][1697144.474987] Call Trace: [Wed Feb 6 09:39:30 2019][1697144.477633] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:30 2019][1697144.484342] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1697144.489518] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.495529] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697144.501303] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.508008] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1697144.513062] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.519330] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1697144.525952] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.532224] ptlrpcd_01_08 S ffffa07b7ed8b0c0 0 126461 2 0x00000080 [Wed Feb 6 09:39:30 2019][1697144.539583] Call Trace: [Wed Feb 6 09:39:30 2019][1697144.542226] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:30 2019][1697144.548925] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1697144.554093] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.560107] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697144.565888] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.572594] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1697144.577646] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.583916] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1697144.590535] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.596809] ptlrpcd_01_09 S ffffa07b7ed89040 0 126462 2 0x00000080 [Wed Feb 6 09:39:30 2019][1697144.604165] Call Trace: [Wed Feb 6 09:39:30 2019][1697144.606810] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:30 2019][1697144.613511] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1697144.618677] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.624692] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697144.630471] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.637173] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1697144.642231] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.648497] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1697144.655112] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.661385] ptlrpcd_01_10 S ffffa07b7ed88000 0 126463 2 0x00000080 [Wed Feb 6 09:39:30 2019][1697144.668742] Call Trace: [Wed Feb 6 09:39:30 2019][1697144.671385] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:30 2019][1697144.678085] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1697144.683254] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.689266] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697144.695042] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.701746] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1697144.706802] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.713073] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1697144.719687] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.725953] ptlrpcd_01_11 S ffffa07b7ed8e180 0 126464 2 0x00000080 [Wed Feb 6 09:39:30 2019][1697144.733307] Call Trace: [Wed Feb 6 09:39:30 2019][1697144.735954] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:30 2019][1697144.742670] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1697144.747842] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.753851] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697144.759625] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.766331] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1697144.771384] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.777651] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1697144.784264] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.790537] ptlrpcd_02_00 S ffffa07b7e88b0c0 0 126465 2 0x00000080 [Wed Feb 6 09:39:30 2019][1697144.797892] Call Trace: [Wed Feb 6 09:39:30 2019][1697144.800538] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:30 2019][1697144.807239] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1697144.812407] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.818420] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697144.824205] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.830908] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1697144.835960] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.842249] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1697144.848867] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.855139] ptlrpcd_02_01 S ffffa07b7e88a080 0 126466 2 0x00000080 [Wed Feb 6 09:39:30 2019][1697144.862495] Call Trace: [Wed Feb 6 09:39:30 2019][1697144.865142] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:30 2019][1697144.871849] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1697144.877018] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.883030] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697144.888814] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.895517] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1697144.900571] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.906839] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1697144.913458] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.919723] ptlrpcd_02_02 S ffffa07b7e88c100 0 126467 2 0x00000080 [Wed Feb 6 09:39:30 2019][1697144.927081] Call Trace: [Wed Feb 6 09:39:30 2019][1697144.929729] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:30 2019][1697144.936435] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1697144.941603] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.947617] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697144.953399] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697144.960104] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1697144.965163] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.971438] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1697144.978053] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697144.984317] ptlrpcd_02_03 S ffffa07b7e889040 0 126468 2 0x00000080 [Wed Feb 6 09:39:30 2019][1697144.991674] Call Trace: [Wed Feb 6 09:39:30 2019][1697144.994322] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:30 2019][1697145.001030] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1697145.006207] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697145.012218] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697145.017999] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697145.024706] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1697145.029767] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697145.036034] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1697145.042656] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697145.048930] ptlrpcd_02_04 S ffffa07b7e88d140 0 126469 2 0x00000080 [Wed Feb 6 09:39:30 2019][1697145.056285] Call Trace: [Wed Feb 6 09:39:30 2019][1697145.058945] [] ? ptlrpc_check_set.part.23+0x11bd/0x1df0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697145.066773] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1697145.071915] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:30 2019][1697145.077930] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:30 2019][1697145.084493] [] ptlrpcd+0x278/0x550 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697145.090505] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697145.096287] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697145.102993] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1697145.108054] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697145.114336] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1697145.120950] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1697145.127218] ptlrpcd_02_05 S ffffa07b7e888000 0 126470 2 0x00000080 [Wed Feb 6 09:39:30 2019][1697145.134581] Call Trace: [Wed Feb 6 09:39:30 2019][1697145.137239] [] ? ptlrpc_check_set.part.23+0x11bd/0x1df0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697145.145070] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1697145.150210] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:30 2019][1697145.156224] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:30 2019][1697145.162779] [] ptlrpcd+0x278/0x550 [ptlrpc] [Wed Feb 6 09:39:30 2019][1697145.168793] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1697145.174573] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.181279] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1697145.186335] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.192609] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1697145.199229] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.205493] ptlrpcd_02_06 S ffffa07b7e88e180 0 126471 2 0x00000080 [Wed Feb 6 09:39:31 2019][1697145.212850] Call Trace: [Wed Feb 6 09:39:31 2019][1697145.215497] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:31 2019][1697145.222205] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1697145.227382] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.233392] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1697145.239168] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.245873] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1697145.250928] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.257200] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1697145.263814] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.270088] ptlrpcd_02_07 S ffffa07b49ee6180 0 126472 2 0x00000080 [Wed Feb 6 09:39:31 2019][1697145.277443] Call Trace: [Wed Feb 6 09:39:31 2019][1697145.280092] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:31 2019][1697145.286800] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1697145.291976] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.297988] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1697145.303768] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.310476] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1697145.315539] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.321813] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1697145.328434] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.334707] ptlrpcd_02_08 S ffffa07b49ee5140 0 126473 2 0x00000080 [Wed Feb 6 09:39:31 2019][1697145.342063] Call Trace: [Wed Feb 6 09:39:31 2019][1697145.344710] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:31 2019][1697145.351417] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1697145.356585] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.362598] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1697145.368381] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.375086] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1697145.380141] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.386414] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1697145.393035] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.399314] ptlrpcd_02_09 S ffffa07b49ee1040 0 126474 2 0x00000080 [Wed Feb 6 09:39:31 2019][1697145.406674] Call Trace: [Wed Feb 6 09:39:31 2019][1697145.409320] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:31 2019][1697145.416018] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1697145.421189] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.427198] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1697145.432975] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.439680] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1697145.444743] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.451017] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1697145.457637] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.463905] ptlrpcd_02_10 S ffffa07b49ee0000 0 126475 2 0x00000080 [Wed Feb 6 09:39:31 2019][1697145.471267] Call Trace: [Wed Feb 6 09:39:31 2019][1697145.473913] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:31 2019][1697145.480612] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1697145.485781] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.491794] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1697145.497575] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.504283] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1697145.509342] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.515610] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1697145.522231] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.528497] ptlrpcd_02_11 S ffffa07b49ee4100 0 126476 2 0x00000080 [Wed Feb 6 09:39:31 2019][1697145.535853] Call Trace: [Wed Feb 6 09:39:31 2019][1697145.538499] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:31 2019][1697145.545205] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1697145.550375] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.556388] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1697145.562169] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.568875] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1697145.573928] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.580203] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1697145.586827] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.593098] ptlrpcd_03_00 S ffffa07b49ee30c0 0 126477 2 0x00000080 [Wed Feb 6 09:39:31 2019][1697145.600453] Call Trace: [Wed Feb 6 09:39:31 2019][1697145.603102] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:31 2019][1697145.609810] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1697145.614986] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.620997] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1697145.626772] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.633479] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1697145.638538] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.644805] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1697145.651419] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.657693] ptlrpcd_03_01 S ffffa07b49ee2080 0 126478 2 0x00000080 [Wed Feb 6 09:39:31 2019][1697145.665055] Call Trace: [Wed Feb 6 09:39:31 2019][1697145.667701] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:31 2019][1697145.674401] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1697145.679569] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.685584] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1697145.691364] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.698071] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1697145.703134] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.709408] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1697145.716026] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.722295] ptlrpcd_03_02 S ffffa07b57399040 0 126479 2 0x00000080 [Wed Feb 6 09:39:31 2019][1697145.729649] Call Trace: [Wed Feb 6 09:39:31 2019][1697145.732296] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:31 2019][1697145.739006] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1697145.744182] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.750192] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1697145.755969] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.762672] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1697145.767724] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.773992] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1697145.780615] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.786887] ptlrpcd_03_03 S ffffa07b5739c100 0 126480 2 0x00000080 [Wed Feb 6 09:39:31 2019][1697145.794244] Call Trace: [Wed Feb 6 09:39:31 2019][1697145.796889] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:31 2019][1697145.803589] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1697145.808758] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.814770] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1697145.820554] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.827258] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1697145.832313] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.838587] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1697145.845209] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.851482] ptlrpcd_03_04 S ffffa07b5739b0c0 0 126481 2 0x00000080 [Wed Feb 6 09:39:31 2019][1697145.858845] Call Trace: [Wed Feb 6 09:39:31 2019][1697145.861504] [] ? ptlrpc_check_set.part.23+0x11bd/0x1df0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.869337] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1697145.874483] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:31 2019][1697145.880492] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:31 2019][1697145.887054] [] ptlrpcd+0x278/0x550 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.893066] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1697145.898849] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.905554] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1697145.910615] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.916889] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1697145.923502] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.929770] ptlrpcd_03_05 S ffffa07b57398000 0 126482 2 0x00000080 [Wed Feb 6 09:39:31 2019][1697145.937132] Call Trace: [Wed Feb 6 09:39:31 2019][1697145.939780] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:31 2019][1697145.946487] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1697145.951663] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.957675] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1697145.963453] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697145.970157] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1697145.975218] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.981493] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1697145.988114] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697145.994389] ptlrpcd_03_06 S ffffa07b781e0000 0 126483 2 0x00000080 [Wed Feb 6 09:39:31 2019][1697146.001753] Call Trace: [Wed Feb 6 09:39:31 2019][1697146.004411] [] ? ptlrpc_check_set.part.23+0x11bd/0x1df0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697146.012244] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1697146.017389] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:31 2019][1697146.023396] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:31 2019][1697146.029951] [] ptlrpcd+0x278/0x550 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697146.035963] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1697146.041737] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697146.048443] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1697146.053504] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697146.059770] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1697146.066383] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697146.072650] ptlrpcd_03_07 S ffffa07b781e4100 0 126484 2 0x00000080 [Wed Feb 6 09:39:31 2019][1697146.080013] Call Trace: [Wed Feb 6 09:39:31 2019][1697146.082661] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:31 2019][1697146.089369] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1697146.094544] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697146.100556] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1697146.106340] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697146.113046] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1697146.118107] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697146.124374] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1697146.130996] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1697146.137268] ptlrpcd_03_08 S ffffa07b781e5140 0 126485 2 0x00000080 [Wed Feb 6 09:39:31 2019][1697146.144623] Call Trace: [Wed Feb 6 09:39:31 2019][1697146.147270] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:31 2019][1697146.153980] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1697146.159158] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697146.165166] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1697146.170942] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:31 2019][1697146.177646] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697146.182702] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.188973] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697146.195589] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.201862] ptlrpcd_03_09 S ffffa07b781e2080 0 126486 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697146.209217] Call Trace: [Wed Feb 6 09:39:32 2019][1697146.211864] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:32 2019][1697146.218572] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697146.223751] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697146.229760] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1697146.235536] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697146.242242] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697146.247301] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.253569] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697146.260189] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.266454] ptlrpcd_03_10 S ffffa07b781e30c0 0 126487 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697146.273819] Call Trace: [Wed Feb 6 09:39:32 2019][1697146.276464] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:32 2019][1697146.283167] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697146.288342] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697146.294354] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1697146.300138] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697146.306842] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697146.311895] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.318164] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697146.324782] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.331049] ptlrpcd_03_11 S ffffa07b781e1040 0 126488 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697146.338402] Call Trace: [Wed Feb 6 09:39:32 2019][1697146.341049] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:39:32 2019][1697146.347749] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697146.352920] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697146.358933] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1697146.364714] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697146.371420] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697146.376481] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.382755] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697146.389368] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.395634] kmmpd-dm-1 S ffffa07b7b2b8000 0 126574 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697146.402988] Call Trace: [Wed Feb 6 09:39:32 2019][1697146.405618] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697146.410781] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:32 2019][1697146.416789] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:32 2019][1697146.423327] [] ? write_mmp_block+0x12a/0x160 [ldiskfs] [Wed Feb 6 09:39:32 2019][1697146.430292] [] schedule_timeout_interruptible+0x1e/0x20 [Wed Feb 6 09:39:32 2019][1697146.437348] [] kmmpd+0x216/0x430 [ldiskfs] [Wed Feb 6 09:39:32 2019][1697146.443284] [] ? __dump_mmp_msg+0x70/0x70 [ldiskfs] [Wed Feb 6 09:39:32 2019][1697146.449990] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697146.455044] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.461312] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697146.467933] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.474208] jbd2/dm-1-8 S ffffa07b7b2ba080 0 126575 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697146.481573] Call Trace: [Wed Feb 6 09:39:32 2019][1697146.484200] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697146.489367] [] kjournald2+0x203/0x260 [jbd2] [Wed Feb 6 09:39:32 2019][1697146.495467] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:32 2019][1697146.501487] [] ? commit_timeout+0x10/0x10 [jbd2] [Wed Feb 6 09:39:32 2019][1697146.507928] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697146.512982] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.519247] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697146.525861] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.532126] kmmpd-dm-2 S ffffa07b7b2be180 0 126576 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697146.539482] Call Trace: [Wed Feb 6 09:39:32 2019][1697146.542108] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697146.547249] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:32 2019][1697146.553258] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:32 2019][1697146.559801] [] ? write_mmp_block+0x12a/0x160 [ldiskfs] [Wed Feb 6 09:39:32 2019][1697146.566769] [] schedule_timeout_interruptible+0x1e/0x20 [Wed Feb 6 09:39:32 2019][1697146.573831] [] kmmpd+0x216/0x430 [ldiskfs] [Wed Feb 6 09:39:32 2019][1697146.579758] [] ? __dump_mmp_msg+0x70/0x70 [ldiskfs] [Wed Feb 6 09:39:32 2019][1697146.586459] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697146.591519] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.597786] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697146.604399] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.610665] jbd2/dm-2-8 S ffffa07b7b2b9040 0 126577 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697146.618020] Call Trace: [Wed Feb 6 09:39:32 2019][1697146.620651] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697146.625802] [] kjournald2+0x203/0x260 [jbd2] [Wed Feb 6 09:39:32 2019][1697146.631899] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:32 2019][1697146.637916] [] ? commit_timeout+0x10/0x10 [jbd2] [Wed Feb 6 09:39:32 2019][1697146.644362] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697146.649414] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.655681] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697146.662303] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.668578] ldiskfs-rsv-con S ffffa07b7b2bb0c0 0 126578 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697146.675947] Call Trace: [Wed Feb 6 09:39:32 2019][1697146.678577] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697146.683718] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:32 2019][1697146.689560] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:32 2019][1697146.695747] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697146.700800] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.707064] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697146.713678] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.719945] ldiskfs-rsv-con S ffffa07b7b2bd140 0 126581 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697146.727305] Call Trace: [Wed Feb 6 09:39:32 2019][1697146.729936] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697146.735077] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:39:32 2019][1697146.740910] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:39:32 2019][1697146.747096] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697146.752151] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.758426] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697146.765047] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.771320] ldlm_cb00_000 S ffffa07b797cb0c0 0 126585 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697146.778675] Call Trace: [Wed Feb 6 09:39:32 2019][1697146.781306] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697146.786487] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697146.793361] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1697146.799137] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697146.805612] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697146.813185] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697146.818244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.824519] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697146.831139] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.837407] lc_watchdogd S ffffa07b797cd140 0 126586 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697146.844769] Call Trace: [Wed Feb 6 09:39:32 2019][1697146.847398] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697146.852544] [] lcw_dispatch_main+0x749/0x940 [libcfs] [Wed Feb 6 09:39:32 2019][1697146.859419] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:39:32 2019][1697146.865433] [] ? lc_watchdog_add+0x790/0x790 [libcfs] [Wed Feb 6 09:39:32 2019][1697146.872309] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697146.877367] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.883635] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697146.890255] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.896536] ldlm_cb00_001 S ffffa07b797ce180 0 126587 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697146.903894] Call Trace: [Wed Feb 6 09:39:32 2019][1697146.906522] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697146.911694] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697146.918570] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1697146.924355] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697146.930829] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697146.938398] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697146.943453] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.949727] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697146.956350] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697146.962623] ldlm_cb01_000 S ffffa07b797c8000 0 126588 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697146.969978] Call Trace: [Wed Feb 6 09:39:32 2019][1697146.972607] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697146.977788] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697146.984662] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1697146.990436] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697146.996914] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697147.004484] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697147.009537] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697147.015812] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697147.022432] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697147.028698] ldlm_cb01_001 S ffffa07b797c9040 0 126589 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697147.036055] Call Trace: [Wed Feb 6 09:39:32 2019][1697147.038681] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697147.043850] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697147.050729] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1697147.056504] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697147.062978] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697147.070549] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697147.075602] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697147.081870] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697147.088492] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697147.094765] ldlm_cb02_000 S ffffa07b797cc100 0 126590 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697147.102121] Call Trace: [Wed Feb 6 09:39:32 2019][1697147.104750] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697147.109928] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697147.116804] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1697147.122577] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697147.129055] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1697147.136628] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1697147.141687] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697147.147953] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1697147.154569] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1697147.160842] ldlm_cb02_001 S ffffa07b593ba080 0 126591 2 0x00000080 [Wed Feb 6 09:39:32 2019][1697147.168204] Call Trace: [Wed Feb 6 09:39:32 2019][1697147.170833] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1697147.176002] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.182881] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1697147.188663] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.195139] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.202709] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1697147.207762] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.214029] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1697147.220642] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.226908] ldlm_cb03_000 S ffffa07b593bc100 0 126593 2 0x00000080 [Wed Feb 6 09:39:33 2019][1697147.234262] Call Trace: [Wed Feb 6 09:39:33 2019][1697147.236892] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1697147.242060] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.248939] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1697147.254724] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.261202] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.268777] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1697147.273838] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.280106] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1697147.286726] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.292993] ldlm_cb03_001 S ffffa07b593b8000 0 126594 2 0x00000080 [Wed Feb 6 09:39:33 2019][1697147.300356] Call Trace: [Wed Feb 6 09:39:33 2019][1697147.302984] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1697147.308151] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.315033] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1697147.320816] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.327289] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.334861] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1697147.339924] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.346199] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1697147.352818] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.359086] ldlm_cn00_000 S ffffa07b74669040 0 126595 2 0x00000080 [Wed Feb 6 09:39:33 2019][1697147.366449] Call Trace: [Wed Feb 6 09:39:33 2019][1697147.369076] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1697147.374244] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.381124] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1697147.386899] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.393372] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.400947] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1697147.406008] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.412281] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1697147.418895] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.425170] ldlm_cn00_001 S ffffa07b749a5140 0 126596 2 0x00000080 [Wed Feb 6 09:39:33 2019][1697147.432533] Call Trace: [Wed Feb 6 09:39:33 2019][1697147.435163] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1697147.440339] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.447218] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1697147.452999] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.459476] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.467046] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1697147.472099] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.478365] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1697147.484978] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.491244] ldlm_cn00_002 S ffffa07b749a30c0 0 126597 2 0x00000080 [Wed Feb 6 09:39:33 2019][1697147.498600] Call Trace: [Wed Feb 6 09:39:33 2019][1697147.501228] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1697147.506399] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.513276] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1697147.519057] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.525531] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.533107] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1697147.538166] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.544432] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1697147.551045] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.557312] ldlm_cn01_000 S ffffa07b749a0000 0 126598 2 0x00000080 [Wed Feb 6 09:39:33 2019][1697147.564675] Call Trace: [Wed Feb 6 09:39:33 2019][1697147.567303] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1697147.572472] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.579352] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1697147.585134] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.591608] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.599181] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1697147.604243] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.610518] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1697147.617138] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.623411] ldlm_cn01_001 S ffffa07b749a1040 0 126599 2 0x00000080 [Wed Feb 6 09:39:33 2019][1697147.630767] Call Trace: [Wed Feb 6 09:39:33 2019][1697147.633397] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1697147.638567] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.645444] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1697147.651221] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.657693] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.665265] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1697147.670318] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.676584] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1697147.683206] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.689480] ldlm_cn01_002 S ffffa07b749a6180 0 126600 2 0x00000080 [Wed Feb 6 09:39:33 2019][1697147.696843] Call Trace: [Wed Feb 6 09:39:33 2019][1697147.699471] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1697147.704657] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.711536] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1697147.717311] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.723786] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.731358] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1697147.736432] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.742704] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1697147.749324] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.755600] ldlm_cn02_000 S ffffa07b749a2080 0 126601 2 0x00000080 [Wed Feb 6 09:39:33 2019][1697147.762962] Call Trace: [Wed Feb 6 09:39:33 2019][1697147.765590] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1697147.770760] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.777639] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1697147.783420] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.789896] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.797470] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1697147.802528] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.808796] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1697147.815417] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.821681] ldlm_cn02_001 S ffffa08b5d580000 0 126602 2 0x00000080 [Wed Feb 6 09:39:33 2019][1697147.829038] Call Trace: [Wed Feb 6 09:39:33 2019][1697147.831667] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1697147.836846] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.843723] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1697147.849511] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.855979] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.863551] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1697147.868605] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.874881] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1697147.881501] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.887768] ldlm_cn02_002 S ffffa08b5d581040 0 126603 2 0x00000080 [Wed Feb 6 09:39:33 2019][1697147.895121] Call Trace: [Wed Feb 6 09:39:33 2019][1697147.897750] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1697147.902920] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.909798] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1697147.915582] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.922055] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.929628] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1697147.934690] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.940962] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1697147.947576] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697147.953843] ldlm_cn03_000 S ffffa08b5d582080 0 126604 2 0x00000080 [Wed Feb 6 09:39:33 2019][1697147.961198] Call Trace: [Wed Feb 6 09:39:33 2019][1697147.963826] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1697147.968993] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.975875] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1697147.981656] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.988132] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697147.995703] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1697148.000766] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697148.007039] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1697148.013660] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697148.019926] ldlm_cn03_001 S ffffa08b5d5830c0 0 126605 2 0x00000080 [Wed Feb 6 09:39:33 2019][1697148.027282] Call Trace: [Wed Feb 6 09:39:33 2019][1697148.029910] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1697148.035078] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697148.041958] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1697148.047739] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697148.054218] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697148.061786] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1697148.066839] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697148.073107] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1697148.079729] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697148.086003] ldlm_cn03_002 S ffffa08b5d584100 0 126606 2 0x00000080 [Wed Feb 6 09:39:33 2019][1697148.093366] Call Trace: [Wed Feb 6 09:39:33 2019][1697148.095994] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1697148.101164] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697148.108043] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1697148.113824] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697148.120300] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697148.127873] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1697148.132931] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697148.139202] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1697148.145818] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1697148.152085] ldlm_bl_01 R running task 0 126607 2 0x00000080 [Wed Feb 6 09:39:33 2019][1697148.159443] Call Trace: [Wed Feb 6 09:39:33 2019][1697148.162097] [] ? ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697148.170735] [] ? ldlm_cancel_locks_for_export_cb+0x3f/0x50 [ptlrpc] [Wed Feb 6 09:39:33 2019][1697148.178830] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [Wed Feb 6 09:39:34 2019][1697148.186424] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.195057] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.203678] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [Wed Feb 6 09:39:34 2019][1697148.211183] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.218779] [] ? ldlm_bl_thread_main+0x4c8/0x700 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.226006] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1697148.231786] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.239351] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1697148.244579] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.250853] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1697148.257647] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.263915] ldlm_bl_02 S ffffa08b5d586180 0 126608 2 0x00000080 [Wed Feb 6 09:39:34 2019][1697148.271277] Call Trace: [Wed Feb 6 09:39:34 2019][1697148.273909] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:39:34 2019][1697148.279774] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:39:34 2019][1697148.287167] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1697148.292343] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.299396] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1697148.305179] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.312750] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1697148.317804] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.324070] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1697148.330682] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.336949] ldlm_elt S ffffa08b5ea58000 0 126609 2 0x00000080 [Wed Feb 6 09:39:34 2019][1697148.344305] Call Trace: [Wed Feb 6 09:39:34 2019][1697148.346932] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:34 2019][1697148.353024] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1697148.358197] [] expired_lock_main+0x1c2/0xc00 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.365071] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1697148.370846] [] ? ldlm_server_glimpse_ast+0x3f0/0x3f0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.378419] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1697148.383472] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.389737] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1697148.396351] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.402619] ll_cfg_requeue S ffffa08b5ea59040 0 126610 2 0x00000080 [Wed Feb 6 09:39:34 2019][1697148.409982] Call Trace: [Wed Feb 6 09:39:34 2019][1697148.412610] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:34 2019][1697148.418710] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1697148.423854] [] mgc_requeue_thread+0x5f5/0x870 [mgc] [Wed Feb 6 09:39:34 2019][1697148.430560] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1697148.436316] [] ? mgc_process_config+0x13f0/0x13f0 [mgc] [Wed Feb 6 09:39:34 2019][1697148.443369] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1697148.448429] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.454697] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1697148.461317] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.467582] mdt00_000 S ffffa08b5ea5a080 0 126623 2 0x00000080 [Wed Feb 6 09:39:34 2019][1697148.474938] Call Trace: [Wed Feb 6 09:39:34 2019][1697148.477568] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1697148.482745] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.489623] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1697148.495405] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.501881] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.509451] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1697148.514505] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.520781] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1697148.527402] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.533677] mdt00_001 S ffffa08b5ea5b0c0 0 126624 2 0x00000080 [Wed Feb 6 09:39:34 2019][1697148.541040] Call Trace: [Wed Feb 6 09:39:34 2019][1697148.543668] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1697148.548838] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.555716] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1697148.561500] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.567971] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.575546] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1697148.580620] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.586891] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1697148.593512] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.599787] mdt00_002 S ffffa08b5ea5c100 0 126625 2 0x00000080 [Wed Feb 6 09:39:34 2019][1697148.607148] Call Trace: [Wed Feb 6 09:39:34 2019][1697148.609779] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1697148.614952] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.621827] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1697148.627612] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.634085] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.641655] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1697148.646709] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.652982] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1697148.659596] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.665871] mdt01_000 S ffffa08b5ea5d140 0 126626 2 0x00000080 [Wed Feb 6 09:39:34 2019][1697148.673235] Call Trace: [Wed Feb 6 09:39:34 2019][1697148.675863] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1697148.681032] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.687912] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1697148.693692] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.700167] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.707740] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1697148.712802] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.719077] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1697148.725697] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.731971] mdt01_001 S ffffa08b5ea5e180 0 126627 2 0x00000080 [Wed Feb 6 09:39:34 2019][1697148.739327] Call Trace: [Wed Feb 6 09:39:34 2019][1697148.741954] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1697148.747126] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.754003] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1697148.759786] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.766259] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.773833] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1697148.778894] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.785160] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1697148.791774] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.798046] mdt01_002 S ffffa08b5f280000 0 126628 2 0x00000080 [Wed Feb 6 09:39:34 2019][1697148.805404] Call Trace: [Wed Feb 6 09:39:34 2019][1697148.808052] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1697148.813228] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.820102] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1697148.825877] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.832354] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.839924] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1697148.844978] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.851244] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1697148.857871] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.864141] mdt02_000 S ffffa08b5f281040 0 126629 2 0x00000080 [Wed Feb 6 09:39:34 2019][1697148.871504] Call Trace: [Wed Feb 6 09:39:34 2019][1697148.874133] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1697148.879300] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.886180] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1697148.891965] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.898440] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.906009] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1697148.911062] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.917329] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1697148.923965] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.930232] mdt02_001 S ffffa08b5f282080 0 126630 2 0x00000080 [Wed Feb 6 09:39:34 2019][1697148.937587] Call Trace: [Wed Feb 6 09:39:34 2019][1697148.940217] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1697148.945384] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.952265] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1697148.958047] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.964521] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697148.972093] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1697148.977148] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.983423] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1697148.990044] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697148.996319] mdt02_002 S ffffa08b5f2830c0 0 126631 2 0x00000080 [Wed Feb 6 09:39:34 2019][1697149.003681] Call Trace: [Wed Feb 6 09:39:34 2019][1697149.006309] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1697149.011477] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697149.018358] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1697149.024139] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697149.030614] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697149.038186] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1697149.043240] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697149.049529] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1697149.056143] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697149.062409] mdt03_000 S ffffa08b5f284100 0 126632 2 0x00000080 [Wed Feb 6 09:39:34 2019][1697149.069766] Call Trace: [Wed Feb 6 09:39:34 2019][1697149.072393] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1697149.077562] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697149.084441] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1697149.090225] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697149.096697] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697149.104271] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1697149.109325] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697149.115599] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1697149.122218] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697149.128483] mdt03_001 S ffffa08b5f285140 0 126633 2 0x00000080 [Wed Feb 6 09:39:34 2019][1697149.135842] Call Trace: [Wed Feb 6 09:39:34 2019][1697149.138469] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1697149.143638] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697149.150518] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1697149.156299] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697149.162772] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1697149.170345] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1697149.175398] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1697149.181664] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697149.188275] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.194545] mdt03_002 S ffffa08b5f286180 0 126634 2 0x00000080 [Wed Feb 6 09:39:35 2019][1697149.201907] Call Trace: [Wed Feb 6 09:39:35 2019][1697149.204537] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1697149.209712] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.216593] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1697149.222375] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.228850] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.236422] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1697149.241483] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.247759] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697149.254378] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.260645] mdt_rdpg00_000 S ffffa08b70be0000 0 126635 2 0x00000080 [Wed Feb 6 09:39:35 2019][1697149.268001] Call Trace: [Wed Feb 6 09:39:35 2019][1697149.270630] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1697149.275808] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.282684] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1697149.288458] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.294935] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.302505] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1697149.307557] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.313824] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697149.320438] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.326712] mdt_rdpg00_001 S ffffa08b70be1040 0 126636 2 0x00000080 [Wed Feb 6 09:39:35 2019][1697149.334066] Call Trace: [Wed Feb 6 09:39:35 2019][1697149.336695] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1697149.341865] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.348742] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1697149.354520] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.360990] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.368565] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1697149.373625] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.379893] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697149.386513] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.392779] mdt_rdpg01_000 S ffffa08b70be2080 0 126637 2 0x00000080 [Wed Feb 6 09:39:35 2019][1697149.400135] Call Trace: [Wed Feb 6 09:39:35 2019][1697149.402763] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1697149.407933] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.414809] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1697149.420584] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.427059] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.434631] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1697149.439692] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.445958] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697149.452580] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.458853] mdt_rdpg01_001 S ffffa08b70be30c0 0 126638 2 0x00000080 [Wed Feb 6 09:39:35 2019][1697149.466209] Call Trace: [Wed Feb 6 09:39:35 2019][1697149.468837] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1697149.474006] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.480888] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1697149.486667] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.493143] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.500715] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1697149.505767] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.512036] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697149.518654] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.524921] mdt_rdpg02_000 S ffffa08b70be4100 0 126639 2 0x00000080 [Wed Feb 6 09:39:35 2019][1697149.532276] Call Trace: [Wed Feb 6 09:39:35 2019][1697149.534906] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1697149.540082] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.546960] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1697149.552734] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.559212] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.566780] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1697149.571834] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.578099] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697149.584713] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.590979] mdt_rdpg02_001 S ffffa08b70be5140 0 126640 2 0x00000080 [Wed Feb 6 09:39:35 2019][1697149.598333] Call Trace: [Wed Feb 6 09:39:35 2019][1697149.600964] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1697149.606140] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.613019] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1697149.618793] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.625267] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.632840] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1697149.637902] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.644177] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697149.650797] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.657064] mdt_rdpg03_000 S ffffa08b70be6180 0 126641 2 0x00000080 [Wed Feb 6 09:39:35 2019][1697149.664419] Call Trace: [Wed Feb 6 09:39:35 2019][1697149.667047] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1697149.672218] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.679093] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1697149.684870] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.691341] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.698914] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1697149.703970] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.710244] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697149.716864] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.723132] mdt_rdpg03_001 S ffffa08b79ab8000 0 126642 2 0x00000080 [Wed Feb 6 09:39:35 2019][1697149.730494] Call Trace: [Wed Feb 6 09:39:35 2019][1697149.733123] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1697149.738292] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.745168] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1697149.750944] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.757418] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.764990] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1697149.770043] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.776312] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697149.782931] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.789197] mdt_attr00_000 S ffffa08b79ab9040 0 126643 2 0x00000080 [Wed Feb 6 09:39:35 2019][1697149.796553] Call Trace: [Wed Feb 6 09:39:35 2019][1697149.799181] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1697149.804350] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.811228] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1697149.817003] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.823478] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.831050] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1697149.836110] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.842378] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697149.848998] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.855274] mdt_attr00_001 S ffffa08b79aba080 0 126644 2 0x00000080 [Wed Feb 6 09:39:35 2019][1697149.862636] Call Trace: [Wed Feb 6 09:39:35 2019][1697149.865263] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1697149.870435] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.877312] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1697149.883087] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.889561] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.897155] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1697149.902214] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.908487] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697149.915098] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.921366] mdt_attr01_000 S ffffa08b79abb0c0 0 126645 2 0x00000080 [Wed Feb 6 09:39:35 2019][1697149.928730] Call Trace: [Wed Feb 6 09:39:35 2019][1697149.931357] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1697149.936527] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.943404] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1697149.949181] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.955656] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697149.963227] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1697149.968288] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.974564] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697149.981183] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697149.987451] mdt_attr01_001 S ffffa08b79abc100 0 126646 2 0x00000080 [Wed Feb 6 09:39:35 2019][1697149.994813] Call Trace: [Wed Feb 6 09:39:35 2019][1697149.997441] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1697150.002610] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697150.009491] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1697150.015273] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697150.021745] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697150.029320] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1697150.034382] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697150.040657] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697150.047274] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697150.053543] mdt_attr02_000 S ffffa08b79abd140 0 126647 2 0x00000080 [Wed Feb 6 09:39:35 2019][1697150.060897] Call Trace: [Wed Feb 6 09:39:35 2019][1697150.063527] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1697150.068696] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697150.075574] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1697150.081359] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697150.087832] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697150.095402] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1697150.100455] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697150.106722] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697150.113344] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697150.119616] mdt_attr02_001 S ffffa08b79abe180 0 126648 2 0x00000080 [Wed Feb 6 09:39:35 2019][1697150.126974] Call Trace: [Wed Feb 6 09:39:35 2019][1697150.129601] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1697150.134772] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697150.141649] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1697150.147425] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697150.153897] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1697150.161470] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1697150.166531] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1697150.172800] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1697150.179419] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.185695] mdt_attr03_000 S ffffa08b5d458000 0 126649 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697150.193057] Call Trace: [Wed Feb 6 09:39:36 2019][1697150.195686] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1697150.200857] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.207732] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1697150.213507] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.219982] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.227553] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1697150.232607] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.238872] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1697150.245485] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.251751] mdt_attr03_001 S ffffa08b5d459040 0 126650 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697150.259108] Call Trace: [Wed Feb 6 09:39:36 2019][1697150.261736] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1697150.266914] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.273791] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1697150.279565] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.286039] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.293611] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1697150.298665] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.304931] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1697150.311544] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.317819] mdt_out00_000 S ffffa05b7c0b92c0 0 126651 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697150.325175] Call Trace: [Wed Feb 6 09:39:36 2019][1697150.327802] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1697150.332971] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.339850] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1697150.345633] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.352110] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.359680] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1697150.364756] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.371023] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1697150.377637] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.383904] mdt_out00_001 S ffffa08b5d45b0c0 0 126652 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697150.391266] Call Trace: [Wed Feb 6 09:39:36 2019][1697150.393893] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1697150.399063] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.405943] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1697150.411725] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.418199] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.425774] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1697150.430834] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.437108] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1697150.443730] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.450005] mdt_out01_000 S ffffa08b5d45c100 0 126653 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697150.457368] Call Trace: [Wed Feb 6 09:39:36 2019][1697150.459997] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1697150.465166] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.472044] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1697150.477826] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.484299] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.491875] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1697150.496935] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.503200] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1697150.509814] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.516088] mdt_out01_001 S ffffa08b5d45d140 0 126654 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697150.523443] Call Trace: [Wed Feb 6 09:39:36 2019][1697150.526072] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1697150.531243] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.538121] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1697150.543904] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.550378] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.557950] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1697150.563010] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.569279] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1697150.575898] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.582179] mdt_out02_000 S ffffa08b5d45e180 0 126655 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697150.589537] Call Trace: [Wed Feb 6 09:39:36 2019][1697150.592165] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1697150.597334] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.604218] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1697150.609995] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.616470] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.624040] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1697150.629095] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.635370] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1697150.641991] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.648280] mdt_out02_001 S ffffa08b5eb48000 0 126656 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697150.655639] Call Trace: [Wed Feb 6 09:39:36 2019][1697150.658267] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1697150.663435] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.670314] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1697150.676095] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.682570] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.690144] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1697150.695197] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.701472] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1697150.708094] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.714368] mdt_out03_000 S ffffa08b5eb49040 0 126657 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697150.721730] Call Trace: [Wed Feb 6 09:39:36 2019][1697150.724361] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1697150.729536] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.736416] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1697150.742215] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.748691] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.756262] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1697150.761315] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.767581] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1697150.774195] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.780467] mdt_out03_001 S ffffa08b5eb4a080 0 126658 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697150.787824] Call Trace: [Wed Feb 6 09:39:36 2019][1697150.790453] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1697150.795630] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.802507] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1697150.808283] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.814755] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.822329] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1697150.827383] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.833658] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1697150.840277] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.846544] mdt_seqs_0000 S ffffa08b5eb4b0c0 0 126659 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697150.853899] Call Trace: [Wed Feb 6 09:39:36 2019][1697150.856529] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1697150.861704] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.868584] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1697150.874358] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.880833] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.888405] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1697150.893466] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.899732] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1697150.906345] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.912612] mdt_seqs_0001 S ffffa08b5eb4c100 0 126660 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697150.919974] Call Trace: [Wed Feb 6 09:39:36 2019][1697150.922605] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1697150.927780] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.934660] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1697150.940443] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.946918] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697150.954489] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1697150.959551] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.965826] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1697150.972446] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697150.978711] mdt_seqm_0000 S ffffa08b5eb4d140 0 126661 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697150.986068] Call Trace: [Wed Feb 6 09:39:36 2019][1697150.988695] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1697150.993866] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697151.000743] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1697151.006528] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697151.013002] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697151.020572] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1697151.025626] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697151.031902] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1697151.038522] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697151.044797] mdt_seqm_0001 S ffffa08b5eb4e180 0 126662 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697151.052160] Call Trace: [Wed Feb 6 09:39:36 2019][1697151.054788] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1697151.059958] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697151.066836] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1697151.072611] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697151.079085] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697151.086658] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1697151.091719] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697151.097988] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1697151.104605] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697151.110871] mdt_fld_0000 S ffffa08b5d590000 0 126663 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697151.118228] Call Trace: [Wed Feb 6 09:39:36 2019][1697151.120857] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1697151.126034] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697151.132915] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1697151.138695] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697151.145171] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1697151.152742] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1697151.157804] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697151.164078] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1697151.170700] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1697151.176975] mdt_fld_0001 S ffffa08b5d591040 0 126664 2 0x00000080 [Wed Feb 6 09:39:36 2019][1697151.184338] Call Trace: [Wed Feb 6 09:39:37 2019][1697151.186965] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1697151.192133] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.199014] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1697151.204795] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.211271] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.218843] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1697151.223895] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.230162] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1697151.236783] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.243049] mdt_io00_000 S ffffa08b5d592080 0 126665 2 0x00000080 [Wed Feb 6 09:39:37 2019][1697151.250405] Call Trace: [Wed Feb 6 09:39:37 2019][1697151.253034] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1697151.258209] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.265089] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1697151.270872] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.277345] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.284919] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1697151.289979] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.296247] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1697151.302866] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.309135] mdt_io00_001 S ffffa08b5d5930c0 0 126666 2 0x00000080 [Wed Feb 6 09:39:37 2019][1697151.316495] Call Trace: [Wed Feb 6 09:39:37 2019][1697151.319125] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1697151.324293] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.331174] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1697151.336956] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.343432] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.351003] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1697151.356065] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.362339] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1697151.368960] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.375234] mdt_io00_002 S ffffa08b5d594100 0 126667 2 0x00000080 [Wed Feb 6 09:39:37 2019][1697151.382590] Call Trace: [Wed Feb 6 09:39:37 2019][1697151.385225] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1697151.390397] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.397273] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1697151.403048] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.409523] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.417097] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1697151.422157] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.428432] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1697151.435054] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.441329] mdt_io01_000 S ffffa08b5d595140 0 126668 2 0x00000080 [Wed Feb 6 09:39:37 2019][1697151.448690] Call Trace: [Wed Feb 6 09:39:37 2019][1697151.451319] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1697151.456487] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.463367] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1697151.469140] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.475617] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.483187] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1697151.488241] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.494508] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1697151.501129] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.507404] mdt_io01_001 S ffffa08b5d596180 0 126669 2 0x00000080 [Wed Feb 6 09:39:37 2019][1697151.514766] Call Trace: [Wed Feb 6 09:39:37 2019][1697151.517397] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1697151.522572] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.529452] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1697151.535235] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.541707] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.549282] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1697151.554343] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.560617] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1697151.567231] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.573505] mdt_io01_002 S ffffa08b5f2c0000 0 126670 2 0x00000080 [Wed Feb 6 09:39:37 2019][1697151.580861] Call Trace: [Wed Feb 6 09:39:37 2019][1697151.583489] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1697151.588655] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.595534] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1697151.601310] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.607783] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.615355] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1697151.620410] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.626687] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1697151.633307] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.639581] mdt_io02_000 S ffffa08b5f2c1040 0 126671 2 0x00000080 [Wed Feb 6 09:39:37 2019][1697151.646936] Call Trace: [Wed Feb 6 09:39:37 2019][1697151.649563] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1697151.654733] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.661611] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1697151.667386] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.673859] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.681431] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1697151.686486] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.692761] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1697151.699382] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.705655] mdt_io02_001 S ffffa08b5f2c2080 0 126672 2 0x00000080 [Wed Feb 6 09:39:37 2019][1697151.713010] Call Trace: [Wed Feb 6 09:39:37 2019][1697151.715641] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1697151.720818] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.727696] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1697151.733478] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.739953] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.747525] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1697151.752576] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.758846] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1697151.765465] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.771731] mdt_io02_002 S ffffa08b5f2c30c0 0 126673 2 0x00000080 [Wed Feb 6 09:39:37 2019][1697151.779086] Call Trace: [Wed Feb 6 09:39:37 2019][1697151.781715] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1697151.786884] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.793762] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1697151.799536] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.806012] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.813581] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1697151.818635] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.824901] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1697151.831517] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.837790] mdt_io03_000 S ffffa08b5f2c4100 0 126674 2 0x00000080 [Wed Feb 6 09:39:37 2019][1697151.845154] Call Trace: [Wed Feb 6 09:39:37 2019][1697151.847782] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1697151.852949] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.859829] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1697151.865611] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.872086] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.879659] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1697151.884721] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.890995] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1697151.897618] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.903891] mdt_io03_001 S ffffa08b5f2c5140 0 126675 2 0x00000080 [Wed Feb 6 09:39:37 2019][1697151.911255] Call Trace: [Wed Feb 6 09:39:37 2019][1697151.913883] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1697151.919051] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.925932] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1697151.931714] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.938188] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.945761] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1697151.950822] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.957098] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1697151.963718] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697151.969983] mdt_io03_002 S ffffa08b5f2c6180 0 126676 2 0x00000080 [Wed Feb 6 09:39:37 2019][1697151.977340] Call Trace: [Wed Feb 6 09:39:37 2019][1697151.979967] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1697151.985135] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697151.992016] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1697151.997798] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697152.004270] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697152.011844] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1697152.016897] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697152.023172] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1697152.029784] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697152.036049] ll_evictor S ffffa08b5e641040 0 126688 2 0x00000080 [Wed Feb 6 09:39:37 2019][1697152.043406] Call Trace: [Wed Feb 6 09:39:37 2019][1697152.046034] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:37 2019][1697152.052128] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1697152.057304] [] ping_evictor_main+0x4be/0x700 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697152.064185] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1697152.069965] [] ? ptlrpc_del_timeout_client+0x190/0x190 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697152.077712] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1697152.082774] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697152.089048] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1697152.095668] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697152.101936] osp_up2-3 S ffffa08b5e642080 0 126697 2 0x00000080 [Wed Feb 6 09:39:37 2019][1697152.109300] Call Trace: [Wed Feb 6 09:39:37 2019][1697152.111955] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Wed Feb 6 09:39:37 2019][1697152.119032] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Wed Feb 6 09:39:37 2019][1697152.125994] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1697152.131139] [] osp_send_update_thread+0x2f8/0x590 [osp] [Wed Feb 6 09:39:37 2019][1697152.138188] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1697152.143940] [] ? osp_invalidate_request+0x390/0x390 [osp] [Wed Feb 6 09:39:37 2019][1697152.151163] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1697152.156216] [] ? finish_task_switch+0x54/0x1c0 [Wed Feb 6 09:39:37 2019][1697152.162483] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697152.168756] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1697152.175378] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1697152.181644] osp-pre-2-3 S ffffa08b5e6430c0 0 126698 2 0x00000080 [Wed Feb 6 09:39:38 2019][1697152.189000] Call Trace: [Wed Feb 6 09:39:38 2019][1697152.191630] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:38 2019][1697152.197037] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1697152.202189] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:38 2019][1697152.209153] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1697152.214913] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:38 2019][1697152.221614] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1697152.226667] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.232935] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1697152.239556] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.245831] osp_up0-3 S ffffa08b5e644100 0 126699 2 0x00000080 [Wed Feb 6 09:39:38 2019][1697152.253203] Call Trace: [Wed Feb 6 09:39:38 2019][1697152.255862] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Wed Feb 6 09:39:38 2019][1697152.262916] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Wed Feb 6 09:39:38 2019][1697152.269881] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1697152.275048] [] osp_send_update_thread+0x2f8/0x590 [osp] [Wed Feb 6 09:39:38 2019][1697152.282102] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1697152.287862] [] ? osp_invalidate_request+0x390/0x390 [osp] [Wed Feb 6 09:39:38 2019][1697152.295082] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1697152.300137] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.306413] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1697152.313033] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.319307] osp-pre-0-3 S ffffa08b5e645140 0 126700 2 0x00000080 [Wed Feb 6 09:39:38 2019][1697152.326663] Call Trace: [Wed Feb 6 09:39:38 2019][1697152.329292] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:38 2019][1697152.334699] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1697152.339844] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:38 2019][1697152.346806] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1697152.352556] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:38 2019][1697152.359259] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1697152.364315] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.370588] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1697152.377210] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.383476] osp_up1-3 S ffffa08b5e646180 0 126701 2 0x00000080 [Wed Feb 6 09:39:38 2019][1697152.390832] Call Trace: [Wed Feb 6 09:39:38 2019][1697152.393465] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Wed Feb 6 09:39:38 2019][1697152.400426] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1697152.405574] [] osp_send_update_thread+0x2f8/0x590 [osp] [Wed Feb 6 09:39:38 2019][1697152.412623] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1697152.418380] [] ? osp_invalidate_request+0x390/0x390 [osp] [Wed Feb 6 09:39:38 2019][1697152.425606] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1697152.430666] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.436933] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1697152.443553] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.449821] osp-pre-1-3 S ffffa08b5df10000 0 126702 2 0x00000080 [Wed Feb 6 09:39:38 2019][1697152.457182] Call Trace: [Wed Feb 6 09:39:38 2019][1697152.459814] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:38 2019][1697152.465221] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1697152.470365] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:38 2019][1697152.477329] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1697152.483086] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:38 2019][1697152.489791] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1697152.494842] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.501111] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1697152.507732] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.514005] osp-syn-12-3 S ffffa08b5df11040 0 126704 2 0x00000080 [Wed Feb 6 09:39:38 2019][1697152.521362] Call Trace: [Wed Feb 6 09:39:38 2019][1697152.523991] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:38 2019][1697152.530091] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1697152.535236] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:38 2019][1697152.542548] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1697152.548323] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.555638] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:38 2019][1697152.562182] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.569420] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.576658] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.584002] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.591761] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.598999] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.606754] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.614412] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:38 2019][1697152.620860] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:38 2019][1697152.626877] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:38 2019][1697152.633425] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.640216] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:38 2019][1697152.646659] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:38 2019][1697152.652324] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:38 2019][1697152.659893] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1697152.664946] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.671214] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1697152.677835] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.684108] osp-pre-12-3 S ffffa08b5df12080 0 126705 2 0x00000080 [Wed Feb 6 09:39:38 2019][1697152.691464] Call Trace: [Wed Feb 6 09:39:38 2019][1697152.694095] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:38 2019][1697152.699502] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1697152.704652] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:38 2019][1697152.711619] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1697152.717375] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:38 2019][1697152.724081] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1697152.729142] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.735418] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1697152.742037] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.748301] osp-syn-13-3 S ffffa08b5df130c0 0 126706 2 0x00000080 [Wed Feb 6 09:39:38 2019][1697152.755658] Call Trace: [Wed Feb 6 09:39:38 2019][1697152.758289] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:38 2019][1697152.764390] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1697152.769542] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:38 2019][1697152.776852] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1697152.782615] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.789927] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:38 2019][1697152.796470] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.803708] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.810943] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.818268] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.826025] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.833262] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.841019] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.848675] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:38 2019][1697152.855121] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:38 2019][1697152.861141] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:38 2019][1697152.867685] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:38 2019][1697152.874477] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:38 2019][1697152.880923] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:38 2019][1697152.886594] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:38 2019][1697152.894165] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1697152.899218] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.905486] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1697152.912106] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.918380] osp-pre-13-3 S ffffa08b5df14100 0 126707 2 0x00000080 [Wed Feb 6 09:39:38 2019][1697152.925736] Call Trace: [Wed Feb 6 09:39:38 2019][1697152.928364] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:38 2019][1697152.933765] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1697152.938914] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:38 2019][1697152.945881] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1697152.951639] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:38 2019][1697152.958357] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1697152.963412] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.969681] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1697152.976302] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697152.982576] osp-syn-39-3 S ffffa08b5df15140 0 126708 2 0x00000080 [Wed Feb 6 09:39:38 2019][1697152.989940] Call Trace: [Wed Feb 6 09:39:38 2019][1697152.992569] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:38 2019][1697152.998669] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1697153.003813] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:38 2019][1697153.011124] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1697153.016893] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:38 2019][1697153.024206] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:38 2019][1697153.030750] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:38 2019][1697153.037989] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:38 2019][1697153.045226] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:38 2019][1697153.052549] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:38 2019][1697153.060304] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:38 2019][1697153.067542] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:38 2019][1697153.075299] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:38 2019][1697153.082953] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:38 2019][1697153.089393] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:38 2019][1697153.095404] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:38 2019][1697153.101949] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:38 2019][1697153.108743] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:38 2019][1697153.115185] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:38 2019][1697153.120850] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:38 2019][1697153.128420] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1697153.133483] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697153.139757] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1697153.146377] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1697153.152644] osp-pre-39-3 S ffffa08b5df16180 0 126709 2 0x00000080 [Wed Feb 6 09:39:38 2019][1697153.160000] Call Trace: [Wed Feb 6 09:39:38 2019][1697153.162629] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:38 2019][1697153.168036] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1697153.173188] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:38 2019][1697153.180152] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1697153.185910] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:39 2019][1697153.192613] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1697153.197670] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697153.203943] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1697153.210563] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697153.216831] osp-syn-46-3 S ffffa08b5d548000 0 126710 2 0x00000080 [Wed Feb 6 09:39:39 2019][1697153.224194] Call Trace: [Wed Feb 6 09:39:39 2019][1697153.226822] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:39 2019][1697153.232916] [] schedule+0x29/0x70 [Wed Feb 6 09:39:39 2019][1697153.238067] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:39 2019][1697153.245379] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:39 2019][1697153.251147] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.258461] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:39 2019][1697153.265005] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.272245] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.279488] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.286811] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.294567] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.301806] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.309562] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.317215] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:39 2019][1697153.323657] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:39 2019][1697153.329676] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:39 2019][1697153.336219] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.343014] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:39 2019][1697153.349458] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:39 2019][1697153.355130] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:39 2019][1697153.362714] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1697153.367771] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697153.374046] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1697153.380665] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697153.386933] osp-pre-46-3 S ffffa08b5d549040 0 126711 2 0x00000080 [Wed Feb 6 09:39:39 2019][1697153.394311] Call Trace: [Wed Feb 6 09:39:39 2019][1697153.396943] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:39 2019][1697153.402351] [] schedule+0x29/0x70 [Wed Feb 6 09:39:39 2019][1697153.407502] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:39 2019][1697153.414468] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:39 2019][1697153.420225] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:39 2019][1697153.426929] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1697153.431982] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697153.438249] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1697153.444871] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697153.451144] osp-syn-10-3 S ffffa08b5d54a080 0 126712 2 0x00000080 [Wed Feb 6 09:39:39 2019][1697153.458500] Call Trace: [Wed Feb 6 09:39:39 2019][1697153.461128] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:39 2019][1697153.467222] [] schedule+0x29/0x70 [Wed Feb 6 09:39:39 2019][1697153.472373] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:39 2019][1697153.479685] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:39 2019][1697153.485454] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.492767] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:39 2019][1697153.499311] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.506549] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.513785] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.521110] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.528881] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.536119] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.543876] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.551532] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:39 2019][1697153.557979] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:39 2019][1697153.563992] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:39 2019][1697153.570536] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.577328] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:39 2019][1697153.583788] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:39 2019][1697153.589454] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:39 2019][1697153.597023] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1697153.602077] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697153.608351] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1697153.614964] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697153.621230] osp-pre-10-3 S ffffa08b5d54b0c0 0 126713 2 0x00000080 [Wed Feb 6 09:39:39 2019][1697153.628584] Call Trace: [Wed Feb 6 09:39:39 2019][1697153.631217] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:39 2019][1697153.636623] [] schedule+0x29/0x70 [Wed Feb 6 09:39:39 2019][1697153.641773] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:39 2019][1697153.648738] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:39 2019][1697153.654489] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:39 2019][1697153.661192] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1697153.666245] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697153.672513] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1697153.679132] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697153.685396] osp-syn-11-3 S ffffa08b5d54c100 0 126714 2 0x00000080 [Wed Feb 6 09:39:39 2019][1697153.692754] Call Trace: [Wed Feb 6 09:39:39 2019][1697153.695382] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:39 2019][1697153.701476] [] schedule+0x29/0x70 [Wed Feb 6 09:39:39 2019][1697153.706629] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:39 2019][1697153.713940] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:39 2019][1697153.719710] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.727028] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:39 2019][1697153.733573] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.740812] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.748047] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.755365] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:39 2019][1697153.762340] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.770096] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.777333] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.785091] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.792745] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:39 2019][1697153.799185] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:39 2019][1697153.805196] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:39 2019][1697153.811741] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.818554] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:39 2019][1697153.824996] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:39 2019][1697153.830665] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:39 2019][1697153.838240] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1697153.843297] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697153.849565] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1697153.856179] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697153.862453] osp-pre-11-3 S ffffa08b5d54d140 0 126715 2 0x00000080 [Wed Feb 6 09:39:39 2019][1697153.869817] Call Trace: [Wed Feb 6 09:39:39 2019][1697153.872446] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:39 2019][1697153.877852] [] schedule+0x29/0x70 [Wed Feb 6 09:39:39 2019][1697153.882997] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:39 2019][1697153.889961] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:39 2019][1697153.895712] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:39 2019][1697153.902415] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1697153.907469] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697153.913743] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1697153.920377] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697153.926649] osp-syn-14-3 S ffffa08b5d54e180 0 126716 2 0x00000080 [Wed Feb 6 09:39:39 2019][1697153.934010] Call Trace: [Wed Feb 6 09:39:39 2019][1697153.936642] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:39 2019][1697153.942742] [] schedule+0x29/0x70 [Wed Feb 6 09:39:39 2019][1697153.947895] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:39 2019][1697153.955206] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:39 2019][1697153.960976] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.968297] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:39 2019][1697153.974840] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.982079] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.989313] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:39 2019][1697153.996630] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:39 2019][1697154.003604] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:39 2019][1697154.011361] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:39 2019][1697154.018599] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:39 2019][1697154.026354] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:39 2019][1697154.034012] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:39 2019][1697154.040460] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:39 2019][1697154.046477] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:39 2019][1697154.053022] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:39 2019][1697154.059816] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:39 2019][1697154.066261] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:39 2019][1697154.071931] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:39 2019][1697154.079504] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1697154.084564] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697154.090830] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1697154.097443] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697154.103709] osp-pre-14-3 S ffffa08b5e3a8000 0 126717 2 0x00000080 [Wed Feb 6 09:39:39 2019][1697154.111067] Call Trace: [Wed Feb 6 09:39:39 2019][1697154.113694] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:39 2019][1697154.119102] [] schedule+0x29/0x70 [Wed Feb 6 09:39:39 2019][1697154.124255] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:39 2019][1697154.131218] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:39 2019][1697154.136968] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:39 2019][1697154.143673] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1697154.148734] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697154.155000] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1697154.161620] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1697154.167887] osp-syn-26-3 S ffffa08b5e3a9040 0 126718 2 0x00000080 [Wed Feb 6 09:39:39 2019][1697154.175245] Call Trace: [Wed Feb 6 09:39:39 2019][1697154.177870] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:39 2019][1697154.183963] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1697154.189106] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:40 2019][1697154.196420] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1697154.202189] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.209499] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:40 2019][1697154.216046] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.223283] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.230518] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.237837] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:40 2019][1697154.244821] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.252576] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.259812] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.267569] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.275225] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:40 2019][1697154.281675] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:40 2019][1697154.287691] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:40 2019][1697154.294235] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.301029] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:40 2019][1697154.307474] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:40 2019][1697154.313136] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:40 2019][1697154.320708] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1697154.325762] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697154.332037] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1697154.338656] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697154.344925] osp-pre-26-3 S ffffa08b5e3aa080 0 126719 2 0x00000080 [Wed Feb 6 09:39:40 2019][1697154.352286] Call Trace: [Wed Feb 6 09:39:40 2019][1697154.354917] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:40 2019][1697154.360326] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1697154.365477] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:40 2019][1697154.372441] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1697154.378189] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:40 2019][1697154.384895] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1697154.389948] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697154.396221] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1697154.402834] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697154.409101] osp-syn-31-3 S ffffa08b5e3ab0c0 0 126720 2 0x00000080 [Wed Feb 6 09:39:40 2019][1697154.416456] Call Trace: [Wed Feb 6 09:39:40 2019][1697154.419084] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:40 2019][1697154.425179] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1697154.430329] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:40 2019][1697154.437641] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1697154.443409] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.450724] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:40 2019][1697154.457267] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.464505] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.471742] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.479056] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:40 2019][1697154.486033] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.493789] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.501027] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.508784] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.516440] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:40 2019][1697154.522888] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:40 2019][1697154.528905] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:40 2019][1697154.535450] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.542245] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:40 2019][1697154.548687] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:40 2019][1697154.554353] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:40 2019][1697154.561922] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1697154.566974] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697154.573242] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1697154.579861] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697154.586130] osp-pre-31-3 S ffffa08b5e3ac100 0 126721 2 0x00000080 [Wed Feb 6 09:39:40 2019][1697154.593493] Call Trace: [Wed Feb 6 09:39:40 2019][1697154.596123] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:40 2019][1697154.601531] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1697154.606681] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:40 2019][1697154.613644] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1697154.619396] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:40 2019][1697154.626100] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1697154.631163] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697154.637435] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1697154.644050] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697154.650322] osp-syn-42-3 S ffffa08b5e3ad140 0 126722 2 0x00000080 [Wed Feb 6 09:39:40 2019][1697154.657677] Call Trace: [Wed Feb 6 09:39:40 2019][1697154.660307] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:40 2019][1697154.666400] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1697154.671544] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:40 2019][1697154.678856] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1697154.684626] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.691937] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:40 2019][1697154.698481] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.705718] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.712955] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.720279] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.728037] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.735272] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.743029] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.750685] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:40 2019][1697154.757150] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:40 2019][1697154.763161] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:40 2019][1697154.769704] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.776499] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:40 2019][1697154.782942] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:40 2019][1697154.788606] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:40 2019][1697154.796176] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1697154.801230] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697154.807503] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1697154.814115] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697154.820381] osp-pre-42-3 S ffffa08b5e3ae180 0 126723 2 0x00000080 [Wed Feb 6 09:39:40 2019][1697154.827736] Call Trace: [Wed Feb 6 09:39:40 2019][1697154.830367] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:40 2019][1697154.835776] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1697154.840926] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:40 2019][1697154.847890] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1697154.853643] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:40 2019][1697154.860344] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1697154.865397] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697154.871666] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1697154.878284] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697154.884552] osp-syn-9-3 S ffffa08b72f90000 0 126724 2 0x00000080 [Wed Feb 6 09:39:40 2019][1697154.891907] Call Trace: [Wed Feb 6 09:39:40 2019][1697154.894537] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:40 2019][1697154.900635] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1697154.905795] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:40 2019][1697154.913109] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1697154.918880] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.926192] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:40 2019][1697154.932737] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.939974] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.947210] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.954533] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.962290] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.969527] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.977285] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:40 2019][1697154.984939] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:40 2019][1697154.991378] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:40 2019][1697154.997389] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:40 2019][1697155.003933] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:40 2019][1697155.010726] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:40 2019][1697155.017170] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:40 2019][1697155.022834] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:40 2019][1697155.030404] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1697155.035474] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697155.041741] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1697155.048381] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697155.054655] osp-pre-9-3 S ffffa08b72f91040 0 126725 2 0x00000080 [Wed Feb 6 09:39:40 2019][1697155.062019] Call Trace: [Wed Feb 6 09:39:40 2019][1697155.064648] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:40 2019][1697155.070056] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1697155.075198] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:40 2019][1697155.082164] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1697155.087922] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:40 2019][1697155.094626] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1697155.099686] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697155.105955] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1697155.112577] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1697155.118848] osp-syn-2-3 S ffffa08b72f92080 0 126726 2 0x00000080 [Wed Feb 6 09:39:40 2019][1697155.126203] Call Trace: [Wed Feb 6 09:39:40 2019][1697155.128833] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:40 2019][1697155.134927] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1697155.140077] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:40 2019][1697155.147390] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1697155.153160] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:40 2019][1697155.160471] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:40 2019][1697155.167015] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:40 2019][1697155.174257] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:40 2019][1697155.181497] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.188822] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.196577] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.203814] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.211573] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.219228] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:41 2019][1697155.225667] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:41 2019][1697155.231677] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:41 2019][1697155.238222] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.245015] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:41 2019][1697155.251461] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:41 2019][1697155.257132] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:41 2019][1697155.264703] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1697155.269765] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697155.276038] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1697155.282654] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697155.288927] osp-pre-2-3 S ffffa08b72f930c0 0 126727 2 0x00000080 [Wed Feb 6 09:39:41 2019][1697155.296290] Call Trace: [Wed Feb 6 09:39:41 2019][1697155.298921] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:41 2019][1697155.304329] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1697155.309478] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:41 2019][1697155.316443] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1697155.322191] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:41 2019][1697155.328898] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1697155.333960] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697155.340235] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1697155.346855] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697155.353132] osp-syn-43-3 S ffffa08b72f94100 0 126728 2 0x00000080 [Wed Feb 6 09:39:41 2019][1697155.360492] Call Trace: [Wed Feb 6 09:39:41 2019][1697155.363123] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:41 2019][1697155.369223] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1697155.374368] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:41 2019][1697155.381681] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1697155.387447] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.394761] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:41 2019][1697155.401306] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.408541] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.415779] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.423094] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:41 2019][1697155.430071] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.437826] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.445065] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.452821] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.460477] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:41 2019][1697155.466925] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:41 2019][1697155.472943] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:41 2019][1697155.479489] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.486282] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:41 2019][1697155.492727] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:41 2019][1697155.498399] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:41 2019][1697155.505968] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1697155.511020] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697155.517289] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1697155.523908] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697155.530174] osp-pre-43-3 S ffffa08b72f95140 0 126729 2 0x00000080 [Wed Feb 6 09:39:41 2019][1697155.537529] Call Trace: [Wed Feb 6 09:39:41 2019][1697155.540158] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:41 2019][1697155.545558] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1697155.550700] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:41 2019][1697155.557666] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1697155.563426] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:41 2019][1697155.570130] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1697155.575191] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697155.581466] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1697155.588086] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697155.594361] osp-syn-19-3 S ffffa08b72f96180 0 126730 2 0x00000080 [Wed Feb 6 09:39:41 2019][1697155.601716] Call Trace: [Wed Feb 6 09:39:41 2019][1697155.604346] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:41 2019][1697155.610447] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1697155.615597] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:41 2019][1697155.622910] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1697155.628678] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.635992] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:41 2019][1697155.642536] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.649774] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.657010] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.664333] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.672088] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.679328] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.687084] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.694739] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:41 2019][1697155.701180] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:41 2019][1697155.707198] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:41 2019][1697155.713742] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.720535] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:41 2019][1697155.726981] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:41 2019][1697155.732652] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:41 2019][1697155.740223] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1697155.745275] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697155.751544] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1697155.758163] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697155.764430] osp-pre-19-3 S ffffa08b5d440000 0 126731 2 0x00000080 [Wed Feb 6 09:39:41 2019][1697155.771791] Call Trace: [Wed Feb 6 09:39:41 2019][1697155.774424] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:41 2019][1697155.779830] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1697155.784975] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:41 2019][1697155.791938] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1697155.797698] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:41 2019][1697155.804401] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1697155.809463] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697155.815737] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1697155.822357] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697155.828623] osp-syn-18-3 S ffffa08b5d441040 0 126732 2 0x00000080 [Wed Feb 6 09:39:41 2019][1697155.835977] Call Trace: [Wed Feb 6 09:39:41 2019][1697155.838609] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:41 2019][1697155.844710] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1697155.849861] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:41 2019][1697155.857173] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1697155.862941] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.870256] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:41 2019][1697155.876799] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.884035] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.891276] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.898606] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.906360] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.913597] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.921355] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.929011] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:41 2019][1697155.935451] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:41 2019][1697155.941472] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:41 2019][1697155.948014] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:41 2019][1697155.954808] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:41 2019][1697155.961250] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:41 2019][1697155.966915] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:41 2019][1697155.974485] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1697155.979539] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697155.985814] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1697155.992436] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697155.998710] osp-pre-18-3 S ffffa08b5d442080 0 126733 2 0x00000080 [Wed Feb 6 09:39:41 2019][1697156.006065] Call Trace: [Wed Feb 6 09:39:41 2019][1697156.008695] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:41 2019][1697156.014101] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1697156.019245] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:41 2019][1697156.026207] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1697156.031959] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:41 2019][1697156.038662] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1697156.043716] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697156.049983] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1697156.056602] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1697156.062885] osp-syn-25-3 S ffffa08b5d4430c0 0 126734 2 0x00000080 [Wed Feb 6 09:39:41 2019][1697156.070242] Call Trace: [Wed Feb 6 09:39:41 2019][1697156.072916] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:41 2019][1697156.080490] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1697156.085643] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:41 2019][1697156.092953] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1697156.098722] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:41 2019][1697156.106034] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:41 2019][1697156.112578] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:41 2019][1697156.119820] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:41 2019][1697156.127061] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:41 2019][1697156.134386] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:41 2019][1697156.142141] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:41 2019][1697156.149378] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:41 2019][1697156.157136] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:41 2019][1697156.164790] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:41 2019][1697156.171231] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:41 2019][1697156.177248] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:41 2019][1697156.183796] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.190588] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:42 2019][1697156.197033] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:42 2019][1697156.202704] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:42 2019][1697156.210273] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:42 2019][1697156.215328] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.221601] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:42 2019][1697156.228214] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.234495] osp-pre-25-3 S ffffa08b5d444100 0 126735 2 0x00000080 [Wed Feb 6 09:39:42 2019][1697156.241851] Call Trace: [Wed Feb 6 09:39:42 2019][1697156.244482] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:42 2019][1697156.249883] [] schedule+0x29/0x70 [Wed Feb 6 09:39:42 2019][1697156.255033] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:42 2019][1697156.261999] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:42 2019][1697156.267757] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:42 2019][1697156.274461] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:42 2019][1697156.279521] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.285789] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:42 2019][1697156.292409] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.298675] osp-syn-17-3 S ffffa08b5d445140 0 126736 2 0x00000080 [Wed Feb 6 09:39:42 2019][1697156.306031] Call Trace: [Wed Feb 6 09:39:42 2019][1697156.308692] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:42 2019][1697156.316264] [] schedule+0x29/0x70 [Wed Feb 6 09:39:42 2019][1697156.321412] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:42 2019][1697156.328722] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:42 2019][1697156.334486] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.341805] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:42 2019][1697156.348350] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.355586] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.362823] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.370147] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.377903] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.385144] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.392906] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.400562] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:42 2019][1697156.407002] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:42 2019][1697156.413019] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:42 2019][1697156.419565] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.426358] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:42 2019][1697156.432802] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:42 2019][1697156.438464] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:42 2019][1697156.446036] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:42 2019][1697156.451090] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.457364] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:42 2019][1697156.463987] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.470264] osp-pre-17-3 S ffffa08b5d446180 0 126737 2 0x00000080 [Wed Feb 6 09:39:42 2019][1697156.477624] Call Trace: [Wed Feb 6 09:39:42 2019][1697156.480253] [] schedule+0x29/0x70 [Wed Feb 6 09:39:42 2019][1697156.485397] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:42 2019][1697156.492360] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:42 2019][1697156.498117] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:42 2019][1697156.504824] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:42 2019][1697156.509885] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.516151] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:42 2019][1697156.522771] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.529040] osp-syn-0-3 S ffffa08b5f108000 0 126738 2 0x00000080 [Wed Feb 6 09:39:42 2019][1697156.536403] Call Trace: [Wed Feb 6 09:39:42 2019][1697156.539032] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:42 2019][1697156.545133] [] schedule+0x29/0x70 [Wed Feb 6 09:39:42 2019][1697156.550285] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:42 2019][1697156.557597] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:42 2019][1697156.563365] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.570679] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:42 2019][1697156.577221] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.584473] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.591714] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.599035] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.606794] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.614030] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.621787] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.629442] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:42 2019][1697156.635881] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:42 2019][1697156.641893] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:42 2019][1697156.648440] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.655231] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:42 2019][1697156.661674] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:42 2019][1697156.667338] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:42 2019][1697156.674908] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:42 2019][1697156.679963] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.686236] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:42 2019][1697156.692848] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.699116] osp-pre-0-3 S ffffa08b5f109040 0 126739 2 0x00000080 [Wed Feb 6 09:39:42 2019][1697156.706479] Call Trace: [Wed Feb 6 09:39:42 2019][1697156.709108] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:42 2019][1697156.714508] [] schedule+0x29/0x70 [Wed Feb 6 09:39:42 2019][1697156.719660] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:42 2019][1697156.726623] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:42 2019][1697156.732372] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:42 2019][1697156.739076] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:42 2019][1697156.744131] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.750405] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:42 2019][1697156.757017] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.763285] osp-syn-5-3 S ffffa08b5f10a080 0 126740 2 0x00000080 [Wed Feb 6 09:39:42 2019][1697156.770657] Call Trace: [Wed Feb 6 09:39:42 2019][1697156.773284] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:42 2019][1697156.779376] [] schedule+0x29/0x70 [Wed Feb 6 09:39:42 2019][1697156.784522] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:42 2019][1697156.791832] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:42 2019][1697156.797594] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.804907] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:42 2019][1697156.811450] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.818687] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.825922] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.833248] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.841005] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.848242] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.855998] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.863654] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:42 2019][1697156.870102] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:42 2019][1697156.876128] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:42 2019][1697156.882673] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:42 2019][1697156.889469] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:42 2019][1697156.895912] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:42 2019][1697156.901584] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:42 2019][1697156.909153] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:42 2019][1697156.914207] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.920482] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:42 2019][1697156.927103] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.933370] osp-pre-5-3 S ffffa08b5f10b0c0 0 126741 2 0x00000080 [Wed Feb 6 09:39:42 2019][1697156.940733] Call Trace: [Wed Feb 6 09:39:42 2019][1697156.943362] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:42 2019][1697156.948771] [] schedule+0x29/0x70 [Wed Feb 6 09:39:42 2019][1697156.953922] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:42 2019][1697156.960888] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:42 2019][1697156.966646] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:42 2019][1697156.973350] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:42 2019][1697156.978411] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.984684] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:42 2019][1697156.991297] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697156.997564] osp-syn-44-3 S ffffa08b5f10c100 0 126742 2 0x00000080 [Wed Feb 6 09:39:42 2019][1697157.004920] Call Trace: [Wed Feb 6 09:39:42 2019][1697157.007547] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:42 2019][1697157.013640] [] schedule+0x29/0x70 [Wed Feb 6 09:39:42 2019][1697157.018785] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:42 2019][1697157.026096] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:42 2019][1697157.031864] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:42 2019][1697157.039178] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:42 2019][1697157.045723] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:42 2019][1697157.052959] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:42 2019][1697157.060195] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:42 2019][1697157.067520] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:42 2019][1697157.075278] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:42 2019][1697157.082519] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:42 2019][1697157.090281] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:42 2019][1697157.097935] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:42 2019][1697157.104383] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:42 2019][1697157.110404] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:42 2019][1697157.116946] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:42 2019][1697157.123739] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:42 2019][1697157.130184] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:42 2019][1697157.135856] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:42 2019][1697157.143426] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:42 2019][1697157.148479] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697157.154744] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:42 2019][1697157.161373] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1697157.167641] osp-pre-44-3 S ffffa08b5f10d140 0 126743 2 0x00000080 [Wed Feb 6 09:39:42 2019][1697157.174996] Call Trace: [Wed Feb 6 09:39:42 2019][1697157.177623] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:42 2019][1697157.183024] [] schedule+0x29/0x70 [Wed Feb 6 09:39:42 2019][1697157.188169] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:43 2019][1697157.195133] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1697157.200894] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:43 2019][1697157.207595] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1697157.212657] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697157.218931] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1697157.225545] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697157.231819] osp-syn-32-3 S ffffa08b5f10e180 0 126744 2 0x00000080 [Wed Feb 6 09:39:43 2019][1697157.239182] Call Trace: [Wed Feb 6 09:39:43 2019][1697157.241811] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:43 2019][1697157.247905] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1697157.253059] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:43 2019][1697157.260368] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1697157.266135] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.273449] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:43 2019][1697157.279995] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.287233] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.294482] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.301808] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.309566] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.316802] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.324560] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.332215] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:43 2019][1697157.338664] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:43 2019][1697157.344681] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:43 2019][1697157.351226] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.358020] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:43 2019][1697157.364464] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:43 2019][1697157.370137] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:43 2019][1697157.377706] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1697157.382759] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697157.389033] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1697157.395647] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697157.401921] osp-pre-32-3 S ffffa08b5f7a0000 0 126745 2 0x00000080 [Wed Feb 6 09:39:43 2019][1697157.409277] Call Trace: [Wed Feb 6 09:39:43 2019][1697157.411906] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:43 2019][1697157.417314] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1697157.422466] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:43 2019][1697157.429446] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1697157.435198] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:43 2019][1697157.441903] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1697157.446963] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697157.453236] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1697157.459852] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697157.466124] osp-syn-21-3 S ffffa08b5f7a1040 0 126746 2 0x00000080 [Wed Feb 6 09:39:43 2019][1697157.473480] Call Trace: [Wed Feb 6 09:39:43 2019][1697157.476107] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:43 2019][1697157.482202] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1697157.487355] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:43 2019][1697157.494667] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1697157.500434] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.507748] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:43 2019][1697157.514290] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.521531] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.528765] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.536089] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.543844] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.551082] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.558839] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.566496] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:43 2019][1697157.572943] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:43 2019][1697157.578954] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:43 2019][1697157.585511] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.592308] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:43 2019][1697157.598752] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:43 2019][1697157.604416] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:43 2019][1697157.611986] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1697157.617039] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697157.623306] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1697157.629928] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697157.636202] osp-pre-21-3 S ffffa08b5f7a2080 0 126747 2 0x00000080 [Wed Feb 6 09:39:43 2019][1697157.643564] Call Trace: [Wed Feb 6 09:39:43 2019][1697157.646194] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:43 2019][1697157.651593] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1697157.656736] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:43 2019][1697157.663701] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1697157.669462] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:43 2019][1697157.676165] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1697157.681226] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697157.687501] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1697157.694121] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697157.700389] osp-syn-16-3 S ffffa08b5f7a30c0 0 126748 2 0x00000080 [Wed Feb 6 09:39:43 2019][1697157.707752] Call Trace: [Wed Feb 6 09:39:43 2019][1697157.710382] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:43 2019][1697157.716482] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1697157.721625] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:43 2019][1697157.728936] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1697157.734705] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.742020] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:43 2019][1697157.748562] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.755799] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.763038] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.770362] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.778117] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.785354] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.793111] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.800766] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:43 2019][1697157.807205] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:43 2019][1697157.813217] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:43 2019][1697157.819759] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.826554] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:43 2019][1697157.832998] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:43 2019][1697157.838670] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:43 2019][1697157.846242] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1697157.851303] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697157.857577] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1697157.864188] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697157.870455] osp-pre-16-3 S ffffa08b5f7a4100 0 126749 2 0x00000080 [Wed Feb 6 09:39:43 2019][1697157.877811] Call Trace: [Wed Feb 6 09:39:43 2019][1697157.880439] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:43 2019][1697157.885840] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1697157.890993] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:43 2019][1697157.897955] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1697157.903715] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:43 2019][1697157.910419] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1697157.915481] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697157.921753] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1697157.928366] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697157.934634] osp-syn-41-3 S ffffa08b5f7a5140 0 126750 2 0x00000080 [Wed Feb 6 09:39:43 2019][1697157.941998] Call Trace: [Wed Feb 6 09:39:43 2019][1697157.944625] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:43 2019][1697157.950719] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1697157.955863] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:43 2019][1697157.963175] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1697157.968942] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.976258] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:43 2019][1697157.982800] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.990037] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:43 2019][1697157.997277] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:43 2019][1697158.004604] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:43 2019][1697158.012361] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:43 2019][1697158.019603] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:43 2019][1697158.027369] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:43 2019][1697158.035030] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:43 2019][1697158.041480] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:43 2019][1697158.047496] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:43 2019][1697158.054039] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:43 2019][1697158.060836] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:43 2019][1697158.067281] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:43 2019][1697158.072952] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:43 2019][1697158.080522] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1697158.085582] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697158.091850] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1697158.098470] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697158.104737] osp-pre-41-3 S ffffa08b5f7a6180 0 126751 2 0x00000080 [Wed Feb 6 09:39:43 2019][1697158.112099] Call Trace: [Wed Feb 6 09:39:43 2019][1697158.114727] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:43 2019][1697158.120130] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1697158.125286] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:43 2019][1697158.132253] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1697158.138003] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:43 2019][1697158.144706] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1697158.149761] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697158.156037] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1697158.162658] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1697158.168931] osp-syn-4-3 S ffffa08b5d098000 0 126752 2 0x00000080 [Wed Feb 6 09:39:43 2019][1697158.176286] Call Trace: [Wed Feb 6 09:39:43 2019][1697158.178915] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:43 2019][1697158.185017] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1697158.190162] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:44 2019][1697158.197470] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1697158.203231] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.210544] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:44 2019][1697158.217089] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.224326] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.231563] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.238885] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.246643] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.253881] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.261635] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.269292] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:44 2019][1697158.275731] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:44 2019][1697158.281742] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:44 2019][1697158.288287] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.295079] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:44 2019][1697158.301524] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:44 2019][1697158.307196] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:44 2019][1697158.314768] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1697158.319828] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697158.326095] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1697158.332732] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697158.339000] osp-pre-4-3 S ffffa08b5d099040 0 126753 2 0x00000080 [Wed Feb 6 09:39:44 2019][1697158.346364] Call Trace: [Wed Feb 6 09:39:44 2019][1697158.348991] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:44 2019][1697158.354392] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1697158.359544] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:44 2019][1697158.366508] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1697158.372267] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:44 2019][1697158.378971] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1697158.384031] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697158.390299] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1697158.396922] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697158.403195] osp-syn-23-3 S ffffa08b5d09a080 0 126754 2 0x00000080 [Wed Feb 6 09:39:44 2019][1697158.410559] Call Trace: [Wed Feb 6 09:39:44 2019][1697158.413188] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:44 2019][1697158.419287] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1697158.424431] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:44 2019][1697158.431743] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1697158.437512] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.444826] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:44 2019][1697158.451370] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.458606] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.465843] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.473166] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.480921] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.488161] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.495916] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.503574] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:44 2019][1697158.510019] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:44 2019][1697158.516032] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:44 2019][1697158.522575] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.529369] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:44 2019][1697158.535813] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:44 2019][1697158.541477] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:44 2019][1697158.549048] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1697158.554109] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697158.560383] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1697158.566997] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697158.573272] osp-pre-23-3 S ffffa08b5d09b0c0 0 126755 2 0x00000080 [Wed Feb 6 09:39:44 2019][1697158.580635] Call Trace: [Wed Feb 6 09:39:44 2019][1697158.583264] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:44 2019][1697158.588664] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1697158.593816] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:44 2019][1697158.600780] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1697158.606538] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:44 2019][1697158.613243] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1697158.618304] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697158.624579] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1697158.631201] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697158.637475] osp-syn-33-3 S ffffa08b5d09c100 0 126756 2 0x00000080 [Wed Feb 6 09:39:44 2019][1697158.644829] Call Trace: [Wed Feb 6 09:39:44 2019][1697158.647457] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:44 2019][1697158.653551] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1697158.658693] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:44 2019][1697158.666006] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1697158.671777] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.679098] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:44 2019][1643] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.692878] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.700128] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.707455] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.715210] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.722449] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.730205] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.737861] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:44 2019][1697158.744301] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:44 2019][1697158.750321] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:44 2019][1697158.756866] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.763659] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:44 2019][1697158.770103] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:44 2019][1697158.775775] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:44 2019][1697158.783346] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1697158.788407] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697158.794681] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1697158.801294] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697158.807570] osp-pre-33-3 S ffffa08b5d09d140 0 126757 2 0x00000080 [Wed Feb 6 09:39:44 2019][1697158.814932] Call Trace: [Wed Feb 6 09:39:44 2019][1697158.817563] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:44 2019][1697158.822969] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1697158.828133] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:44 2019][1697158.835094] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1697158.840844] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:44 2019][1697158.847562] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1697158.852619] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697158.858885] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1697158.865504] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697158.871772] osp-syn-20-3 S ffffa08b5d09e180 0 126758 2 0x00000080 [Wed Feb 6 09:39:44 2019][1697158.879137] Call Trace: [Wed Feb 6 09:39:44 2019][1697158.881802] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:44 2019][1697158.889375] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1697158.894527] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:44 2019][1697158.901838] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1697158.907608] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.914919] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:44 2019][1697158.921482] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.928720] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.935959] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.943289] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.951050] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.958290] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.966045] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.973702] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:44 2019][1697158.980143] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:44 2019][1697158.986160] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:44 2019][1697158.992704] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:44 2019][1697158.999497] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:44 2019][1697159.005943] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:44 2019][1697159.011612] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:44 2019][1697159.019184] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1697159.024239] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697159.030514] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1697159.037134] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697159.043400] osp-pre-20-3 S ffffa08b5f7e0000 0 126759 2 0x00000080 [Wed Feb 6 09:39:44 2019][1697159.050756] Call Trace: [Wed Feb 6 09:39:44 2019][1697159.053386] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1697159.058537] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:44 2019][1697159.065503] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1697159.071259] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:44 2019][1697159.077963] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1697159.083017] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697159.089290] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1697159.095903] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1697159.102168] osp-syn-7-3 S ffffa08b5f7e1040 0 126760 2 0x00000080 [Wed Feb 6 09:39:44 2019][1697159.109525] Call Trace: [Wed Feb 6 09:39:44 2019][1697159.112154] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:44 2019][1697159.118255] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1697159.123399] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:44 2019][1697159.130709] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1697159.136470] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:44 2019][1697159.143783] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:44 2019][1697159.150326] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:44 2019][1697159.157566] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:44 2019][1697159.164801] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:44 2019][1697159.172125] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:44 2019][1697159.179882] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:44 2019][1697159.187120] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.194877] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.202529] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:45 2019][1697159.208970] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:45 2019][1697159.214983] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:45 2019][1697159.221526] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.228320] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:45 2019][1697159.234763] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:45 2019][1697159.240436] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:45 2019][1697159.248007] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1697159.253068] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697159.259343] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1697159.265964] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697159.272238] osp-pre-7-3 S ffffa08b5f7e2080 0 126761 2 0x00000080 [Wed Feb 6 09:39:45 2019][1697159.279602] Call Trace: [Wed Feb 6 09:39:45 2019][1697159.282230] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:45 2019][1697159.287631] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1697159.292783] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:45 2019][1697159.299747] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:45 2019][1697159.305240] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1697159.310991] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:45 2019][1697159.317696] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1697159.322756] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697159.329023] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1697159.335644] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697159.341912] osp-syn-40-3 S ffffa08b5f7e30c0 0 126762 2 0x00000080 [Wed Feb 6 09:39:45 2019][1697159.349272] Call Trace: [Wed Feb 6 09:39:45 2019][1697159.351902] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:45 2019][1697159.357996] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1697159.363147] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:45 2019][1697159.370458] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1697159.376220] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.383531] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:45 2019][1697159.390077] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.397315] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.404549] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.411874] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.419629] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.426869] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.434625] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.442281] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:45 2019][1697159.448735] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:45 2019][1697159.454750] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:45 2019][1697159.461295] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.468093] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:45 2019][1697159.474539] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:45 2019][1697159.480210] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:45 2019][1697159.487781] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1697159.492832] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697159.499098] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1697159.505711] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697159.511978] osp-pre-40-3 S ffffa08b5f7e4100 0 126763 2 0x00000080 [Wed Feb 6 09:39:45 2019][1697159.519333] Call Trace: [Wed Feb 6 09:39:45 2019][1697159.521963] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:45 2019][1697159.527370] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1697159.532512] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:45 2019][1697159.539477] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1697159.545227] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:45 2019][1697159.551932] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1697159.556986] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697159.563261] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1697159.569879] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697159.576147] osp-syn-45-3 S ffffa08b5f7e5140 0 126764 2 0x00000080 [Wed Feb 6 09:39:45 2019][1697159.583504] Call Trace: [Wed Feb 6 09:39:45 2019][1697159.586129] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:45 2019][1697159.592223] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1697159.597366] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:45 2019][1697159.604679] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1697159.610447] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.617762] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:45 2019][1697159.624306] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.631545] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.638778] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.646103] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.653857] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.661095] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.668852] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.676510] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:45 2019][1697159.682956] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:45 2019][1697159.688964] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:45 2019][1697159.695510] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.702305] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:45 2019][1697159.708749] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:45 2019][1697159.714422] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:45 2019][1697159.721992] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1697159.727045] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697159.733318] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1697159.739933] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697159.746207] osp-pre-45-3 S ffffa08b5f7e6180 0 126765 2 0x00000080 [Wed Feb 6 09:39:45 2019][1697159.753570] Call Trace: [Wed Feb 6 09:39:45 2019][1697159.756199] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1697159.761353] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:45 2019][1697159.768315] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1697159.774066] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:45 2019][1697159.780770] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1697159.785831] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697159.792106] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1697159.798727] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697159.804993] osp-syn-6-3 S ffffa08b60578000 0 126766 2 0x00000080 [Wed Feb 6 09:39:45 2019][1697159.812349] Call Trace: [Wed Feb 6 09:39:45 2019][1697159.814979] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:45 2019][1697159.821080] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1697159.826233] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:45 2019][1697159.833542] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1697159.839306] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.846624] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:45 2019][1697159.853170] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.860405] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.867643] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.874967] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.882723] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.889959] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.897715] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.905371] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:45 2019][1697159.911812] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:45 2019][1697159.917831] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:45 2019][1697159.924376] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:45 2019][1697159.931168] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:45 2019][1697159.937612] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:45 2019][1697159.943274] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:45 2019][1697159.950845] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1697159.955900] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697159.962174] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1697159.968787] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697159.975063] osp-pre-6-3 S ffffa08b60579040 0 126767 2 0x00000080 [Wed Feb 6 09:39:45 2019][1697159.982426] Call Trace: [Wed Feb 6 09:39:45 2019][1697159.985053] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:45 2019][1697159.990454] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1697159.995606] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:45 2019][1697160.002568] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1697160.008322] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:45 2019][1697160.015023] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1697160.020077] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697160.026350] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1697160.032965] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1697160.039236] osp-syn-24-3 S ffffa08b6057a080 0 126768 2 0x00000080 [Wed Feb 6 09:39:45 2019][1697160.046593] Call Trace: [Wed Feb 6 09:39:45 2019][1697160.049223] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:45 2019][1697160.055316] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1697160.060467] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:45 2019][1697160.067780] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1697160.073548] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:45 2019][1697160.080862] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:45 2019][1697160.087405] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:45 2019][1697160.094643] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:45 2019][1697160.101879] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:45 2019][1697160.109204] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:45 2019][1697160.116961] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:45 2019][1697160.124197] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:45 2019][1697160.131953] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:45 2019][1697160.139608] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:45 2019][1697160.146048] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:45 2019][1697160.152060] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:45 2019][1697160.158602] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:45 2019][1697160.165396] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:45 2019][1697160.171842] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:45 2019][1697160.177513] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:45 2019][1697160.185083] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1697160.190136] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.196420] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1697160.203042] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.209315] osp-pre-24-3 S ffffa08b6057b0c0 0 126769 2 0x00000080 [Wed Feb 6 09:39:46 2019][1697160.216680] Call Trace: [Wed Feb 6 09:39:46 2019][1697160.219308] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1697160.224452] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:46 2019][1697160.231417] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1697160.237177] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:46 2019][1697160.243879] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1697160.248942] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.255230] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1697160.261846] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.268119] osp-syn-47-3 S ffffa08b6057c100 0 126770 2 0x00000080 [Wed Feb 6 09:39:46 2019][1697160.275475] Call Trace: [Wed Feb 6 09:39:46 2019][1697160.278102] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:46 2019][1697160.284198] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1697160.289349] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:46 2019][1697160.296659] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1697160.302422] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.309733] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:46 2019][1697160.316282] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.323523] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.330778] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.338102] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.345856] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.353095] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.360850] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.368508] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:46 2019][1697160.374955] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:46 2019][1697160.380965] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:46 2019][1697160.387511] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.394308] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:46 2019][1697160.400747] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:46 2019][1697160.406410] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:46 2019][1697160.413982] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1697160.419042] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.425311] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1697160.431932] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.438206] osp-pre-47-3 S ffffa08b6057d140 0 126771 2 0x00000080 [Wed Feb 6 09:39:46 2019][1697160.445569] Call Trace: [Wed Feb 6 09:39:46 2019][1697160.448198] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:46 2019][1697160.453597] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1697160.458742] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:46 2019][1697160.465706] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1697160.471464] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:46 2019][1697160.478168] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1697160.483221] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.489495] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1697160.496108] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.502375] osp-syn-38-3 S ffffa08b6057e180 0 126772 2 0x00000080 [Wed Feb 6 09:39:46 2019][1697160.509738] Call Trace: [Wed Feb 6 09:39:46 2019][1697160.512368] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:46 2019][1697160.518468] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1697160.523611] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:46 2019][1697160.530923] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1697160.536685] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.543995] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:46 2019][1697160.550541] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.557779] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.565015] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.572338] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.580096] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.587330] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.595089] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.602745] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:46 2019][1697160.609191] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:46 2019][1697160.615202] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:46 2019][1697160.621748] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.628540] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:46 2019][1697160.634986] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:46 2019][1697160.640656] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:46 2019][1697160.648228] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1697160.653289] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.659555] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1697160.666167] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.672434] osp-pre-38-3 S ffffa08b60438000 0 126773 2 0x00000080 [Wed Feb 6 09:39:46 2019][1697160.679789] Call Trace: [Wed Feb 6 09:39:46 2019][1697160.682417] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:46 2019][1697160.687817] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1697160.692961] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:46 2019][1697160.699927] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1697160.705684] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:46 2019][1697160.712387] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1697160.717440] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.723716] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1697160.730336] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.736602] osp-syn-15-3 S ffffa08b60439040 0 126774 2 0x00000080 [Wed Feb 6 09:39:46 2019][1697160.743957] Call Trace: [Wed Feb 6 09:39:46 2019][1697160.746588] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:46 2019][1697160.752690] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1697160.757840] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:46 2019][1697160.765152] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1697160.770921] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.778234] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:46 2019][1697160.784778] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.792015] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.799252] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.806576] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.814332] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.821569] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.829324] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.836981] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:46 2019][1697160.843419] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:46 2019][1697160.849432] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:46 2019][1697160.855975] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:46 2019][1697160.862767] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:46 2019][1697160.869212] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:46 2019][1697160.874878] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:46 2019][1697160.882446] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1697160.887501] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.893775] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1697160.900388] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.906661] osp-pre-15-3 S ffffa08b6043a080 0 126775 2 0x00000080 [Wed Feb 6 09:39:46 2019][1697160.914017] Call Trace: [Wed Feb 6 09:39:46 2019][1697160.916646] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:46 2019][1697160.922053] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1697160.927198] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:46 2019][1697160.934162] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1697160.939920] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:46 2019][1697160.946626] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1697160.951685] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.957952] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1697160.964574] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697160.970848] osp-syn-29-3 S ffffa08b6043b0c0 0 126776 2 0x00000080 [Wed Feb 6 09:39:46 2019][1697160.978204] Call Trace: [Wed Feb 6 09:39:46 2019][1697160.980830] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:46 2019][1697160.986925] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1697160.992069] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:46 2019][1697160.999378] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1697161.005141] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:46 2019][1697161.012456] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:46 2019][1697161.018997] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:46 2019][1697161.026235] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:46 2019][1697161.033471] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:46 2019][1697161.040787] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:46 2019][1697161.047763] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:46 2019][1697161.055520] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:46 2019][1697161.062755] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:46 2019][1697161.070515] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:46 2019][1697161.078171] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:46 2019][1697161.084615] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:46 2019][1697161.090627] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:46 2019][1697161.097172] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:46 2019][1697161.103966] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:46 2019][1697161.110408] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:46 2019][1697161.116072] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:46 2019][1697161.123643] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1697161.128697] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697161.134972] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1697161.141593] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1697161.147858] osp-pre-29-3 S ffffa08b6043c100 0 126777 2 0x00000080 [Wed Feb 6 09:39:46 2019][1697161.155216] Call Trace: [Wed Feb 6 09:39:46 2019][1697161.157843] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1697161.162986] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:46 2019][1697161.169952] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:39:47 2019][1697161.175445] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1697161.181205] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:47 2019][1697161.187908] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1697161.192962] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697161.199235] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1697161.205849] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697161.212124] osp-syn-1-3 S ffffa08b6043d140 0 126778 2 0x00000080 [Wed Feb 6 09:39:47 2019][1697161.219487] Call Trace: [Wed Feb 6 09:39:47 2019][1697161.222116] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:47 2019][1697161.228210] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1697161.233362] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:47 2019][1697161.240672] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1697161.246440] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.253753] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:47 2019][1697161.260299] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.267541] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.274780] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.282103] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.289861] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.297098] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.304853] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.312510] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:47 2019][1697161.318949] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:47 2019][1697161.324961] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:47 2019][1697161.331505] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.338302] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:47 2019][1697161.344750] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:47 2019][1697161.350415] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:47 2019][1697161.357985] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1697161.363046] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697161.369312] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1697161.375934] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697161.382209] osp-pre-1-3 S ffffa08b6043e180 0 126779 2 0x00000080 [Wed Feb 6 09:39:47 2019][1697161.389571] Call Trace: [Wed Feb 6 09:39:47 2019][1697161.392201] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:47 2019][1697161.397608] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1697161.402751] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:47 2019][1697161.409716] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1697161.415466] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:47 2019][1697161.422173] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1697161.427231] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697161.433500] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1697161.440120] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697161.446394] osp-syn-22-3 S ffffa08b786f8000 0 126780 2 0x00000080 [Wed Feb 6 09:39:47 2019][1697161.453748] Call Trace: [Wed Feb 6 09:39:47 2019][1697161.456379] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:47 2019][1697161.462472] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1697161.467623] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:47 2019][1697161.474935] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1697161.480704] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.488016] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:47 2019][1697161.494561] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.501798] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.509034] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.516346] [] ? probe_sched_wakeup+0x2b/0xa0 [Wed Feb 6 09:39:47 2019][1697161.522547] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.530303] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.537538] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.545299] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.552960] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:47 2019][1697161.559402] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:47 2019][1697161.565420] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:47 2019][1697161.571965] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.578758] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:47 2019][1697161.585216] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:47 2019][1697161.590881] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:47 2019][1697161.598454] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1697161.603513] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697161.609782] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1697161.616403] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697161.622678] osp-pre-22-3 S ffffa08b786f9040 0 126781 2 0x00000080 [Wed Feb 6 09:39:47 2019][1697161.630041] Call Trace: [Wed Feb 6 09:39:47 2019][1697161.632686] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:47 2019][1697161.638086] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1697161.643231] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:47 2019][1697161.650193] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1697161.655946] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:47 2019][1697161.662647] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1697161.667702] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697161.673976] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1697161.680600] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697161.686870] osp-syn-3-3 S ffffa08b786fa080 0 126782 2 0x00000080 [Wed Feb 6 09:39:47 2019][1697161.694225] Call Trace: [Wed Feb 6 09:39:47 2019][1697161.696855] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:47 2019][1697161.702959] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1697161.708108] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:47 2019][1697161.715420] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1697161.721189] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.728502] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:47 2019][1697161.735046] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.742285] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.749521] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.756844] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.764602] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.771839] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.779595] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.787250] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:47 2019][1697161.793709] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:47 2019][1697161.799723] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:47 2019][1697161.806270] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.813065] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:47 2019][1697161.819509] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:47 2019][1697161.825180] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:47 2019][1697161.832749] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1697161.837805] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697161.844077] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1697161.850691] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697161.856967] osp-pre-3-3 S ffffa08b786fb0c0 0 126783 2 0x00000080 [Wed Feb 6 09:39:47 2019][1697161.864331] Call Trace: [Wed Feb 6 09:39:47 2019][1697161.866958] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:47 2019][1697161.872359] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1697161.877511] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:47 2019][1697161.884491] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1697161.890243] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:47 2019][1697161.896945] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1697161.902000] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697161.908272] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1697161.914886] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697161.921152] osp-syn-27-3 S ffffa08b786fc100 0 126784 2 0x00000080 [Wed Feb 6 09:39:47 2019][1697161.928516] Call Trace: [Wed Feb 6 09:39:47 2019][1697161.931144] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:47 2019][1697161.937236] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1697161.942381] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:47 2019][1697161.949691] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1697161.955460] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.962776] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:47 2019][1697161.969323] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.976565] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.983800] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.991125] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:47 2019][1697161.998883] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:47 2019][1697162.006119] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:47 2019][1697162.013874] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:47 2019][1697162.021530] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:47 2019][1697162.027969] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:47 2019][1697162.033979] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:47 2019][1697162.040526] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:47 2019][1697162.047319] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:47 2019][1697162.053761] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:47 2019][1697162.059425] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:47 2019][1697162.066994] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1697162.072050] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697162.078322] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1697162.084936] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697162.091204] osp-pre-27-3 S ffffa08b786fd140 0 126785 2 0x00000080 [Wed Feb 6 09:39:47 2019][1697162.098567] Call Trace: [Wed Feb 6 09:39:47 2019][1697162.101197] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:47 2019][1697162.106603] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1697162.111748] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:47 2019][1697162.118711] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1697162.124462] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:47 2019][1697162.131165] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1697162.136219] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697162.142494] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1697162.149114] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1697162.155389] osp-syn-37-3 S ffffa08b786fe180 0 126786 2 0x00000080 [Wed Feb 6 09:39:47 2019][1697162.162744] Call Trace: [Wed Feb 6 09:39:47 2019][1697162.165374] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:48 2019][1697162.171474] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1697162.176619] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:48 2019][1697162.183930] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1697162.189698] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.197011] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:48 2019][1697162.203557] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.210793] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.218031] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.225354] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.233109] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.240347] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.248101] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.255758] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:48 2019][1697162.262200] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:48 2019][1697162.268217] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:48 2019][1697162.274763] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.281556] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:48 2019][1697162.288000] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:48 2019][1697162.293671] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:48 2019][1697162.301243] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1697162.306307] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697162.312586] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1697162.319201] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697162.325473] osp-pre-37-3 S ffffa08b611b8000 0 126787 2 0x00000080 [Wed Feb 6 09:39:48 2019][1697162.332830] Call Trace: [Wed Feb 6 09:39:48 2019][1697162.335458] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:48 2019][1697162.340856] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1697162.346002] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:48 2019][1697162.352967] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1697162.358725] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:48 2019][1697162.365429] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1697162.370490] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697162.376763] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1697162.383378] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697162.389652] osp-syn-36-3 S ffffa08b611b9040 0 126788 2 0x00000080 [Wed Feb 6 09:39:48 2019][1697162.397016] Call Trace: [Wed Feb 6 09:39:48 2019][1697162.399646] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:48 2019][1697162.405746] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1697162.410898] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:48 2019][1697162.418209] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1697162.423977] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.431291] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:48 2019][1697162.437838] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.445074] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.452314] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.459632] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:48 2019][1697162.466612] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.474365] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.481604] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.489358] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.497015] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:48 2019][1697162.503456] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:48 2019][1697162.509472] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:48 2019][1697162.516018] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.522812] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:48 2019][1697162.529257] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:48 2019][1697162.534929] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:48 2019][1697162.542499] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1697162.547552] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697162.553825] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1697162.560438] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697162.566705] osp-pre-36-3 S ffffa08b611ba080 0 126789 2 0x00000080 [Wed Feb 6 09:39:48 2019][1697162.574059] Call Trace: [Wed Feb 6 09:39:48 2019][1697162.576691] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:48 2019][1697162.582112] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1697162.587259] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:48 2019][1697162.594223] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1697162.599981] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:48 2019][1697162.606686] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1697162.611748] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697162.618019] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1697162.624634] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697162.630901] osp-syn-8-3 S ffffa08b611bb0c0 0 126790 2 0x00000080 [Wed Feb 6 09:39:48 2019][1697162.638265] Call Trace: [Wed Feb 6 09:39:48 2019][1697162.640894] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:48 2019][1697162.646995] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1697162.652146] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:48 2019][1697162.659459] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1697162.665225] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.672540] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:48 2019][1697162.679085] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.686326] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.693566] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.700892] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.708645] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.715884] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.723642] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.731298] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:48 2019][1697162.737745] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:48 2019][1697162.743763] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:48 2019][1697162.750309] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.757110] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:48 2019][1697162.763554] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:48 2019][1697162.769225] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:48 2019][1697162.776797] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1697162.781852] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697162.788124] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1697162.794738] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697162.801011] osp-pre-8-3 S ffffa08b611bc100 0 126791 2 0x00000080 [Wed Feb 6 09:39:48 2019][1697162.808366] Call Trace: [Wed Feb 6 09:39:48 2019][1697162.810995] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:48 2019][1697162.816394] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1697162.821539] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:48 2019][1697162.828505] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1697162.834262] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:48 2019][1697162.840964] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1697162.846019] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697162.852294] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1697162.858914] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697162.865181] osp-syn-35-3 S ffffa08b611bd140 0 126792 2 0x00000080 [Wed Feb 6 09:39:48 2019][1697162.872544] Call Trace: [Wed Feb 6 09:39:48 2019][1697162.875182] [] ? osd_ldiskfs_read+0x131/0x2d0 [osd_ldiskfs] [Wed Feb 6 09:39:48 2019][1697162.882574] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:48 2019][1697162.888668] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1697162.893820] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:48 2019][1697162.901129] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1697162.906892] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.914206] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:48 2019][1697162.920747] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.927986] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.935224] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.942544] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.950303] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.957539] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.965296] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.972966] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:48 2019][1697162.979408] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:48 2019][1697162.985418] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:48 2019][1697162.991965] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:48 2019][1697162.998756] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:48 2019][1697163.005200] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:48 2019][1697163.010864] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:48 2019][1697163.018434] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1697163.023490] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697163.029763] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1697163.036384] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697163.042649] osp-pre-35-3 S ffffa08b611be180 0 126793 2 0x00000080 [Wed Feb 6 09:39:48 2019][1697163.050006] Call Trace: [Wed Feb 6 09:39:48 2019][1697163.052635] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:48 2019][1697163.058041] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1697163.063185] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:48 2019][1697163.070149] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1697163.075898] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:48 2019][1697163.082604] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1697163.087655] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697163.093923] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1697163.100545] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1697163.106820] osp-syn-28-3 S ffffa08b5d150000 0 126794 2 0x00000080 [Wed Feb 6 09:39:48 2019][1697163.114182] Call Trace: [Wed Feb 6 09:39:48 2019][1697163.116811] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:48 2019][1697163.122905] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1697163.128056] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:48 2019][1697163.135369] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1697163.141137] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:48 2019][1697163.148449] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:48 2019][1697163.154992] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:48 2019][1697163.162231] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:48 2019][1697163.169468] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.176793] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.184546] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.191784] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.199540] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.207198] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:49 2019][1697163.213646] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:49 2019][1697163.219663] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:49 2019][1697163.226208] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.233004] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:49 2019][1697163.239446] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:49 2019][1697163.245110] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:49 2019][1697163.252682] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1697163.257740] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.264008] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1697163.270621] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.276888] osp-pre-28-3 S ffffa08b5d151040 0 126795 2 0x00000080 [Wed Feb 6 09:39:49 2019][1697163.284251] Call Trace: [Wed Feb 6 09:39:49 2019][1697163.286879] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:49 2019][1697163.292281] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1697163.297430] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:49 2019][1697163.304395] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1697163.310146] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:49 2019][1697163.316850] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1697163.321911] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.328176] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1697163.334791] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.341066] osp-syn-30-3 S ffffa08b5d152080 0 126796 2 0x00000080 [Wed Feb 6 09:39:49 2019][1697163.348429] Call Trace: [Wed Feb 6 09:39:49 2019][1697163.351056] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:49 2019][1697163.357151] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1697163.362301] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:49 2019][1697163.369613] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1697163.375383] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.382696] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:49 2019][1697163.389240] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.396476] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.403713] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.411036] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.418794] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.426031] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.433785] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.441442] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:49 2019][1697163.447882] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:49 2019][1697163.453896] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:49 2019][1697163.460436] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.467232] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:49 2019][1697163.473675] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:49 2019][1697163.479340] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:49 2019][1697163.486910] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1697163.491969] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.498236] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1697163.504850] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.511124] osp-pre-30-3 S ffffa08b5d1530c0 0 126797 2 0x00000080 [Wed Feb 6 09:39:49 2019][1697163.518487] Call Trace: [Wed Feb 6 09:39:49 2019][1697163.521116] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:49 2019][1697163.526516] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1697163.531668] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:49 2019][1697163.538632] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1697163.544383] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:49 2019][1697163.551087] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1697163.556149] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.562424] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1697163.569044] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.575312] osp-syn-34-3 S ffffa08b5d154100 0 126798 2 0x00000080 [Wed Feb 6 09:39:49 2019][1697163.582688] Call Trace: [Wed Feb 6 09:39:49 2019][1697163.585321] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:49 2019][1697163.591423] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1697163.596573] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:49 2019][1697163.603884] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1697163.609647] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.616957] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:49 2019][1697163.623506] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.630749] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.637986] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.645309] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.653065] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.660302] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.668057] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.675715] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:49 2019][1697163.682164] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:49 2019][1697163.688182] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:49 2019][1697163.694727] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:49 2019][1697163.701519] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:49 2019][1697163.707965] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:49 2019][1697163.713636] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:49 2019][1697163.721207] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1697163.726260] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.732535] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1697163.739156] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.745430] osp-pre-34-3 S ffffa08b5d155140 0 126799 2 0x00000080 [Wed Feb 6 09:39:49 2019][1697163.752785] Call Trace: [Wed Feb 6 09:39:49 2019][1697163.755415] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:49 2019][1697163.760821] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1697163.765967] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:49 2019][1697163.772931] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1697163.778689] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:49 2019][1697163.785394] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1697163.790453] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.796721] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1697163.803342] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.809616] lquota_wb_fir-M S ffffa08b7bcbc100 0 126870 2 0x00000080 [Wed Feb 6 09:39:49 2019][1697163.816970] Call Trace: [Wed Feb 6 09:39:49 2019][1697163.819600] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1697163.824742] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:49 2019][1697163.830755] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:49 2019][1697163.837315] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Wed Feb 6 09:39:49 2019][1697163.844106] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Wed Feb 6 09:39:49 2019][1697163.850722] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1697163.856473] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Wed Feb 6 09:39:49 2019][1697163.862831] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1697163.867891] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.874158] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1697163.880771] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.887037] lquota_wb_fir-M S ffffa08b7bcbd140 0 126871 2 0x00000080 [Wed Feb 6 09:39:49 2019][1697163.894399] Call Trace: [Wed Feb 6 09:39:49 2019][1697163.897029] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1697163.902170] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:49 2019][1697163.908185] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:49 2019][1697163.914725] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Wed Feb 6 09:39:49 2019][1697163.921534] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Wed Feb 6 09:39:49 2019][1697163.928152] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1697163.933904] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Wed Feb 6 09:39:49 2019][1697163.940259] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1697163.945312] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.951580] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1697163.958199] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697163.964466] dist_txn-3 S ffffa08b7bcbe180 0 126872 2 0x00000080 [Wed Feb 6 09:39:49 2019][1697163.971828] Call Trace: [Wed Feb 6 09:39:49 2019][1697163.974458] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:49 2019][1697163.980550] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1697163.985745] [] distribute_txn_commit_thread+0x9ac/0xca0 [ptlrpc] [Wed Feb 6 09:39:49 2019][1697163.993579] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1697163.999369] [] ? sub_trans_commit_cb+0x20/0x20 [ptlrpc] [Wed Feb 6 09:39:49 2019][1697164.006421] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1697164.011473] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697164.017739] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1697164.024354] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697164.030630] osp_up2-1 S ffffa06b785b1040 0 126886 2 0x00000080 [Wed Feb 6 09:39:49 2019][1697164.037991] Call Trace: [Wed Feb 6 09:39:49 2019][1697164.040651] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Wed Feb 6 09:39:49 2019][1697164.047708] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Wed Feb 6 09:39:49 2019][1697164.054668] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1697164.059816] [] osp_send_update_thread+0x2f8/0x590 [osp] [Wed Feb 6 09:39:49 2019][1697164.066863] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1697164.072614] [] ? osp_invalidate_request+0x390/0x390 [osp] [Wed Feb 6 09:39:49 2019][1697164.079837] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1697164.084889] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697164.091158] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1697164.097779] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697164.104054] osp-pre-2-1 S ffffa06b785b5140 0 126887 2 0x00000080 [Wed Feb 6 09:39:49 2019][1697164.111417] Call Trace: [Wed Feb 6 09:39:49 2019][1697164.114048] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:49 2019][1697164.119453] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1697164.124598] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:49 2019][1697164.131561] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1697164.137312] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:49 2019][1697164.144015] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1697164.149069] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697164.155343] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1697164.161964] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1697164.168230] osp_up0-1 S ffffa06b785b30c0 0 126888 2 0x00000080 [Wed Feb 6 09:39:50 2019][1697164.175586] Call Trace: [Wed Feb 6 09:39:50 2019][1697164.178244] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Wed Feb 6 09:39:50 2019][1697164.185301] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Wed Feb 6 09:39:50 2019][1697164.192262] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1697164.197408] [] osp_send_update_thread+0x2f8/0x590 [osp] [Wed Feb 6 09:39:50 2019][1697164.204457] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1697164.210208] [] ? osp_invalidate_request+0x390/0x390 [osp] [Wed Feb 6 09:39:50 2019][1697164.217430] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1697164.222486] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.228758] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1697164.235371] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.241637] osp-pre-0-1 S ffffa06b785b2080 0 126889 2 0x00000080 [Wed Feb 6 09:39:50 2019][1697164.248994] Call Trace: [Wed Feb 6 09:39:50 2019][1697164.251622] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:50 2019][1697164.257030] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1697164.262172] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:50 2019][1697164.269137] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1697164.274888] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:50 2019][1697164.281592] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1697164.286653] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.292922] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1697164.299539] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.305806] osp_up3-1 S ffffa08b7bcbb0c0 0 126891 2 0x00000080 [Wed Feb 6 09:39:50 2019][1697164.313162] Call Trace: [Wed Feb 6 09:39:50 2019][1697164.315818] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Wed Feb 6 09:39:50 2019][1697164.322879] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Wed Feb 6 09:39:50 2019][1697164.329839] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1697164.334983] [] osp_send_update_thread+0x2f8/0x590 [osp] [Wed Feb 6 09:39:50 2019][1697164.342036] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1697164.347793] [] ? osp_invalidate_request+0x390/0x390 [osp] [Wed Feb 6 09:39:50 2019][1697164.355016] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1697164.360071] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.366346] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1697164.372965] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.379232] osp-pre-3-1 S ffffa08b7bcba080 0 126892 2 0x00000080 [Wed Feb 6 09:39:50 2019][1697164.386588] Call Trace: [Wed Feb 6 09:39:50 2019][1697164.389216] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:50 2019][1697164.394615] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1697164.399760] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:50 2019][1697164.406724] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1697164.412480] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:50 2019][1697164.419185] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1697164.424238] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.430505] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1697164.437126] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.443400] osp-syn-12-1 S ffffa08b7bcb9040 0 126893 2 0x00000080 [Wed Feb 6 09:39:50 2019][1697164.450754] Call Trace: [Wed Feb 6 09:39:50 2019][1697164.453384] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:50 2019][1697164.459477] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1697164.464622] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:50 2019][1697164.471932] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1697164.477705] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.485022] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:50 2019][1697164.491567] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.498805] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.506040] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.513352] [] ? probe_sched_wakeup+0x2b/0xa0 [Wed Feb 6 09:39:50 2019][1697164.519552] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.527308] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.534546] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.542302] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.549959] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:50 2019][1697164.556398] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:50 2019][1697164.562410] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:50 2019][1697164.568954] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.575747] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:50 2019][1697164.582189] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:50 2019][1697164.587870] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:50 2019][1697164.595441] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1697164.600493] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.606762] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1697164.613382] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.619657] osp-pre-12-1 S ffffa08b7bcb8000 0 126894 2 0x00000080 [Wed Feb 6 09:39:50 2019][1697164.627012] Call Trace: [Wed Feb 6 09:39:50 2019][1697164.629641] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:50 2019][1697164.635040] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1697164.640183] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:50 2019][1697164.647148] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1697164.652905] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:50 2019][1697164.659609] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1697164.664663] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.670931] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1697164.677551] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.683827] osp-syn-13-1 S ffffa06b04210000 0 126895 2 0x00000080 [Wed Feb 6 09:39:50 2019][1697164.691190] Call Trace: [Wed Feb 6 09:39:50 2019][1697164.693819] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:50 2019][1697164.699921] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1697164.705071] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:50 2019][1697164.712383] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1697164.718143] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.725455] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:50 2019][1697164.732004] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.739246] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.746483] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.753808] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.761568] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.768808] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.776566] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.784221] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:50 2019][1697164.790660] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:50 2019][1697164.796671] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:50 2019][1697164.803222] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.810021] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:50 2019][1697164.816466] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:50 2019][1697164.822134] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:50 2019][1697164.829707] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1697164.834768] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.841042] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1697164.847662] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.853929] osp-pre-13-1 S ffffa06b04212080 0 126896 2 0x00000080 [Wed Feb 6 09:39:50 2019][1697164.861294] Call Trace: [Wed Feb 6 09:39:50 2019][1697164.863924] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:50 2019][1697164.869331] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1697164.874486] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:50 2019][1697164.881447] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1697164.887210] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:50 2019][1697164.893920] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1697164.898979] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.905254] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1697164.911874] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697164.918140] osp-syn-39-1 S ffffa06b04215140 0 126897 2 0x00000080 [Wed Feb 6 09:39:50 2019][1697164.925504] Call Trace: [Wed Feb 6 09:39:50 2019][1697164.928136] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:50 2019][1697164.934235] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1697164.939381] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:50 2019][1697164.946692] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1697164.952467] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.959783] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:50 2019][1697164.966332] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.973573] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.980809] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.988133] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:50 2019][1697164.995889] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:50 2019][1697165.003128] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:50 2019][1697165.010893] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:50 2019][1697165.018556] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:50 2019][1697165.025004] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:50 2019][1697165.031021] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:50 2019][1697165.037573] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:50 2019][1697165.044371] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:50 2019][1697165.050816] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:50 2019][1697165.056485] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:50 2019][1697165.064058] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1697165.069117] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697165.075394] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1697165.082014] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697165.088286] osp-pre-39-1 S ffffa06b04211040 0 126898 2 0x00000080 [Wed Feb 6 09:39:50 2019][1697165.095644] Call Trace: [Wed Feb 6 09:39:50 2019][1697165.098273] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1697165.103429] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:50 2019][1697165.110398] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1697165.116155] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:50 2019][1697165.122858] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1697165.127919] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697165.134189] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1697165.140808] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1697165.147081] osp-syn-46-1 S ffffa06b042130c0 0 126899 2 0x00000080 [Wed Feb 6 09:39:50 2019][1697165.154437] Call Trace: [Wed Feb 6 09:39:50 2019][1697165.157067] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:50 2019][1697165.163172] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1697165.168324] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:51 2019][1697165.175641] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1697165.181415] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.188732] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:51 2019][1697165.195282] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.202523] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.209761] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.217089] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.224846] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.232082] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.239840] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.247497] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:51 2019][1697165.253944] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:51 2019][1697165.259964] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:51 2019][1697165.266509] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.273302] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:51 2019][1697165.279744] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:51 2019][1697165.285409] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:51 2019][1697165.292982] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1697165.298044] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697165.304318] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1697165.310936] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697165.317213] osp-pre-46-1 S ffffa06b04214100 0 126900 2 0x00000080 [Wed Feb 6 09:39:51 2019][1697165.324575] Call Trace: [Wed Feb 6 09:39:51 2019][1697165.327205] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1697165.332358] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:51 2019][1697165.339323] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1697165.345079] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:51 2019][1697165.351783] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1697165.356841] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697165.363111] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1697165.369731] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697165.376007] osp-syn-10-1 S ffffa06b04216180 0 126901 2 0x00000080 [Wed Feb 6 09:39:51 2019][1697165.383362] Call Trace: [Wed Feb 6 09:39:51 2019][1697165.385990] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:51 2019][1697165.392087] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1697165.397238] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:51 2019][1697165.404550] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1697165.410319] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.417637] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:51 2019][1697165.424181] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.431421] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.438654] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.445980] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.453735] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.460974] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.468730] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.476387] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:51 2019][1697165.482835] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:51 2019][1697165.488853] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:51 2019][1697165.495397] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.502192] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:51 2019][1697165.508634] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:51 2019][1697165.514297] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:51 2019][1697165.521868] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1697165.526923] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697165.533196] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1697165.539819] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697165.546092] osp-pre-10-1 S ffffa06b776a1040 0 126902 2 0x00000080 [Wed Feb 6 09:39:51 2019][1697165.553458] Call Trace: [Wed Feb 6 09:39:51 2019][1697165.556087] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:51 2019][1697165.561494] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1697165.566645] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:51 2019][1697165.573611] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1697165.579369] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:51 2019][1697165.586073] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1697165.591133] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697165.597402] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1697165.604022] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697165.610295] osp-syn-11-1 S ffffa06b776a0000 0 126903 2 0x00000080 [Wed Feb 6 09:39:51 2019][1697165.617652] Call Trace: [Wed Feb 6 09:39:51 2019][1697165.620280] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:51 2019][1697165.626381] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1697165.631525] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:51 2019][1697165.638836] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1697165.644605] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.651918] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:51 2019][1697165.658462] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.665701] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.672935] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.680259] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.688016] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.695254] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.703012] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.710667] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:51 2019][1697165.717114] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:51 2019][1697165.723133] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:51 2019][1697165.729680] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.736478] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:51 2019][1697165.742924] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:51 2019][1697165.748595] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:51 2019][1697165.756165] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1697165.761218] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697165.767486] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1697165.774107] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697165.780380] osp-pre-11-1 S ffffa06b776a5140 0 126904 2 0x00000080 [Wed Feb 6 09:39:51 2019][1697165.787736] Call Trace: [Wed Feb 6 09:39:51 2019][1697165.790366] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:51 2019][1697165.795773] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1697165.800918] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:51 2019][1697165.807880] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1697165.813632] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:51 2019][1697165.820338] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1697165.823355] Lustre: 126481:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549474209/real 1549474218] req@ffffa05c8d197800 x1623667959668544/t0(0) o400->fir-MDT0000-lwp-MDT0003@10.0.10.51@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1549474966 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [Wed Feb 6 09:39:51 2019][1697165.823358] Lustre: 126481:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 626 previous similar messages [Wed Feb 6 09:39:51 2019][1697165.823365] Lustre: fir-MDT0000-lwp-MDT0003: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 09:39:51 2019][1697165.823366] Lustre: Skipped 90 previous similar messages [Wed Feb 6 09:39:51 2019][1697165.885820] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697165.892087] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1697165.898700] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697165.904973] osp-syn-14-1 S ffffa06b776a30c0 0 126905 2 0x00000080 [Wed Feb 6 09:39:51 2019][1697165.912328] Call Trace: [Wed Feb 6 09:39:51 2019][1697165.914959] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:51 2019][1697165.921060] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1697165.926205] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:51 2019][1697165.933516] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1697165.939285] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.946598] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:51 2019][1697165.953142] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.960378] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.967615] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.974939] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.982695] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.989931] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:51 2019][1697165.997692] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:51 2019][1697166.005354] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:51 2019][1697166.011803] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:51 2019][1697166.017820] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:51 2019][1697166.024367] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:51 2019][1697166.031159] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:51 2019][1697166.037604] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:51 2019][1697166.043273] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:51 2019][1697166.050843] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1697166.055898] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697166.062174] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1697166.068794] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697166.075061] osp-pre-14-1 S ffffa06b776a4100 0 126906 2 0x00000080 [Wed Feb 6 09:39:51 2019][1697166.082425] Call Trace: [Wed Feb 6 09:39:51 2019][1697166.085052] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:51 2019][1697166.090451] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1697166.095595] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:51 2019][1697166.102562] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1697166.108319] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:51 2019][1697166.115023] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1697166.120076] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697166.126350] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1697166.132963] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1697166.139230] osp-syn-26-1 S ffffa06b776a2080 0 126907 2 0x00000080 [Wed Feb 6 09:39:51 2019][1697166.146592] Call Trace: [Wed Feb 6 09:39:51 2019][1697166.149222] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:51 2019][1697166.155323] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1697166.160466] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:51 2019][1697166.167778] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1697166.173548] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.180859] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:52 2019][1697166.187403] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.194640] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.201879] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.209201] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.216959] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.224194] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.231954] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.239611] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:52 2019][1697166.246056] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:52 2019][1697166.252077] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:52 2019][1697166.258621] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.265416] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:52 2019][1697166.271856] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:52 2019][1697166.277519] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:52 2019][1697166.285092] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1697166.290153] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697166.296427] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1697166.303041] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697166.309316] osp-pre-26-1 S ffffa08b5d0d8000 0 126908 2 0x00000080 [Wed Feb 6 09:39:52 2019][1697166.316678] Call Trace: [Wed Feb 6 09:39:52 2019][1697166.319309] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1697166.324460] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:52 2019][1697166.331424] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1697166.337173] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:52 2019][1697166.343876] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1697166.348929] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697166.355196] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1697166.361811] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697166.368084] osp-syn-31-1 S ffffa08b5d0d9040 0 126909 2 0x00000080 [Wed Feb 6 09:39:52 2019][1697166.375438] Call Trace: [Wed Feb 6 09:39:52 2019][1697166.378070] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:52 2019][1697166.384168] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1697166.389315] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:52 2019][1697166.396625] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1697166.402393] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.409706] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:52 2019][1697166.416250] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.423486] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.430726] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.438040] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:52 2019][1697166.445016] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.452771] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.460009] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.467765] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.475421] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:52 2019][1697166.481862] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:52 2019][1697166.487878] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:52 2019][1697166.494425] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.501221] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:52 2019][1697166.507664] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:52 2019][1697166.513336] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:52 2019][1697166.520907] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1697166.525966] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697166.532235] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1697166.538854] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697166.545120] osp-pre-31-1 S ffffa08b5d0da080 0 126910 2 0x00000080 [Wed Feb 6 09:39:52 2019][1697166.552474] Call Trace: [Wed Feb 6 09:39:52 2019][1697166.555105] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:52 2019][1697166.560513] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1697166.565664] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:52 2019][1697166.572629] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1697166.578385] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:52 2019][1697166.585092] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1697166.590152] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697166.596419] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1697166.603043] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697166.609316] osp-syn-42-1 S ffffa08b5d0db0c0 0 126911 2 0x00000080 [Wed Feb 6 09:39:52 2019][1697166.616677] Call Trace: [Wed Feb 6 09:39:52 2019][1697166.619309] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:52 2019][1697166.625410] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1697166.630562] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:52 2019][1697166.637873] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1697166.643643] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.650954] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:52 2019][1697166.657500] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.664736] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.671972] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.679298] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.687053] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.694290] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.702047] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.709704] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:52 2019][1697166.716150] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:52 2019][1697166.722162] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:52 2019][1697166.728706] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.735499] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:52 2019][1697166.741944] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:52 2019][1697166.747615] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:52 2019][1697166.755187] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1697166.760246] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697166.766515] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1697166.773136] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697166.779412] osp-pre-42-1 S ffffa08b5d0dc100 0 126912 2 0x00000080 [Wed Feb 6 09:39:52 2019][1697166.786773] Call Trace: [Wed Feb 6 09:39:52 2019][1697166.789402] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1697166.794555] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:52 2019][1697166.801518] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1697166.807269] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:52 2019][1697166.813974] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1697166.819034] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697166.825300] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1697166.831912] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697166.838178] osp-syn-9-1 S ffffa08b5d0dd140 0 126913 2 0x00000080 [Wed Feb 6 09:39:52 2019][1697166.845535] Call Trace: [Wed Feb 6 09:39:52 2019][1697166.848164] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:52 2019][1697166.854263] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1697166.859407] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:52 2019][1697166.866718] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1697166.872480] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.879792] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:52 2019][1697166.886336] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.893573] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.900809] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.908134] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.915889] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.923128] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.930882] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.938541] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:52 2019][1697166.944988] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:52 2019][1697166.950997] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:52 2019][1697166.957542] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:52 2019][1697166.964337] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:52 2019][1697166.970782] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:52 2019][1697166.976451] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:52 2019][1697166.984023] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1697166.989078] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697166.995352] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1697167.001973] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697167.008246] osp-pre-9-1 S ffffa08b5d0de180 0 126914 2 0x00000080 [Wed Feb 6 09:39:52 2019][1697167.015602] Call Trace: [Wed Feb 6 09:39:52 2019][1697167.018232] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:52 2019][1697167.023639] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1697167.028782] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:52 2019][1697167.035747] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1697167.041504] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:52 2019][1697167.048210] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1697167.053270] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697167.059536] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1697167.066150] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1697167.072426] osp-syn-2-1 S ffffa08b5d7e8000 0 126915 2 0x00000080 [Wed Feb 6 09:39:52 2019][1697167.079789] Call Trace: [Wed Feb 6 09:39:52 2019][1697167.082461] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:52 2019][1697167.090036] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1697167.095186] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:52 2019][1697167.102498] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1697167.108259] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:52 2019][1697167.115571] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:52 2019][1697167.122117] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:52 2019][1697167.129356] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:52 2019][1697167.136597] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:52 2019][1697167.143923] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:52 2019][1697167.151678] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:52 2019][1697167.158917] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:52 2019][1697167.166675] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.174328] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:53 2019][1697167.180768] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:53 2019][1697167.186788] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:53 2019][1697167.193330] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.200125] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:53 2019][1697167.206569] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:53 2019][1697167.212241] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:53 2019][1697167.219811] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1697167.224863] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.231132] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1697167.237751] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.244018] osp-pre-2-1 S ffffa08b5d7e9040 0 126916 2 0x00000080 [Wed Feb 6 09:39:53 2019][1697167.251374] Call Trace: [Wed Feb 6 09:39:53 2019][1697167.254001] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1697167.259145] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:53 2019][1697167.266110] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1697167.271866] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:53 2019][1697167.278574] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1697167.283632] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.289901] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1697167.296522] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.302795] osp-syn-43-1 S ffffa08b5d7ea080 0 126917 2 0x00000080 [Wed Feb 6 09:39:53 2019][1697167.310152] Call Trace: [Wed Feb 6 09:39:53 2019][1697167.312780] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:53 2019][1697167.318874] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1697167.324026] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:53 2019][1697167.331336] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1697167.337106] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.344417] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:53 2019][1697167.350961] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.358201] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.365436] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.372760] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.380518] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.387754] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.395510] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.403165] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:53 2019][1697167.409606] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:53 2019][1697167.415624] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:53 2019][1697167.422170] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.428963] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:53 2019][1697167.435405] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:53 2019][1697167.441070] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:53 2019][1697167.448639] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1697167.453694] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.459971] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1697167.466590] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.472865] osp-pre-43-1 S ffffa08b5d7eb0c0 0 126918 2 0x00000080 [Wed Feb 6 09:39:53 2019][1697167.480228] Call Trace: [Wed Feb 6 09:39:53 2019][1697167.482856] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1697167.488001] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:53 2019][1697167.494965] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1697167.500725] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:53 2019][1697167.507427] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1697167.512481] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.518755] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1697167.525377] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.531652] osp-syn-19-1 S ffffa08b5d7ec100 0 126919 2 0x00000080 [Wed Feb 6 09:39:53 2019][1697167.539013] Call Trace: [Wed Feb 6 09:39:53 2019][1697167.541644] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:53 2019][1697167.547746] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1697167.552898] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:53 2019][1697167.560210] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1697167.565977] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.573291] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:53 2019][1697167.579835] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.587071] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.594313] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.601641] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.609398] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.616635] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.624392] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.632047] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:53 2019][1697167.638487] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:53 2019][1697167.644507] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:53 2019][1697167.651050] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.657842] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:53 2019][1697167.664287] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:53 2019][1697167.669950] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:53 2019][1697167.677522] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1697167.682582] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.688851] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1697167.695470] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.701745] osp-pre-19-1 S ffffa08b5d7ed140 0 126920 2 0x00000080 [Wed Feb 6 09:39:53 2019][1697167.709101] Call Trace: [Wed Feb 6 09:39:53 2019][1697167.711730] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1697167.716882] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:53 2019][1697167.723846] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1697167.729605] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:53 2019][1697167.736308] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1697167.741360] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.747627] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1697167.754241] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.760514] osp-syn-18-1 S ffffa08b5d7ee180 0 126921 2 0x00000080 [Wed Feb 6 09:39:53 2019][1697167.767868] Call Trace: [Wed Feb 6 09:39:53 2019][1697167.770499] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:53 2019][1697167.776601] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1697167.781754] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:53 2019][1697167.789063] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1697167.794833] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.802145] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:53 2019][1697167.808690] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.815925] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.823166] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.830487] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.838243] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.845482] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.853236] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.860892] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:53 2019][1697167.867333] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:53 2019][1697167.873356] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:53 2019][1697167.879904] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:53 2019][1697167.886699] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:53 2019][1697167.893143] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:53 2019][1697167.898815] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:53 2019][1697167.906385] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1697167.911446] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.917712] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1697167.924326] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.930601] osp-pre-18-1 S ffffa08b5d250000 0 126922 2 0x00000080 [Wed Feb 6 09:39:53 2019][1697167.937963] Call Trace: [Wed Feb 6 09:39:53 2019][1697167.940594] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1697167.945744] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:53 2019][1697167.952712] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1697167.958467] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:53 2019][1697167.965173] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1697167.970233] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.976501] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1697167.983122] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697167.989397] osp-syn-25-1 S ffffa08b5d251040 0 126923 2 0x00000080 [Wed Feb 6 09:39:53 2019][1697167.996761] Call Trace: [Wed Feb 6 09:39:53 2019][1697167.999399] [] ? osd_ldiskfs_read+0x131/0x2d0 [osd_ldiskfs] [Wed Feb 6 09:39:53 2019][1697168.006798] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:53 2019][1697168.012891] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1697168.018046] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:53 2019][1697168.025358] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1697168.031123] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:53 2019][1697168.038435] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:53 2019][1697168.044982] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:53 2019][1697168.052217] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:53 2019][1697168.059453] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:53 2019][1697168.066777] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:53 2019][1697168.074533] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:53 2019][1697168.081773] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:53 2019][1697168.089528] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:53 2019][1697168.097184] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:53 2019][1697168.103624] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:53 2019][1697168.109643] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:53 2019][1697168.116188] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:53 2019][1697168.122981] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:53 2019][1697168.129423] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:53 2019][1697168.135089] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:53 2019][1697168.142658] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1697168.147710] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697168.153979] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1697168.160600] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1697168.166874] osp-pre-25-1 S ffffa08b5d252080 0 126924 2 0x00000080 [Wed Feb 6 09:39:54 2019][1697168.174237] Call Trace: [Wed Feb 6 09:39:54 2019][1697168.176867] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:54 2019][1697168.182275] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1697168.187427] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:54 2019][1697168.194391] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1697168.200149] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:54 2019][1697168.206853] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1697168.211907] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697168.218180] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1697168.224794] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697168.231059] osp-syn-17-1 S ffffa08b5d2530c0 0 126925 2 0x00000080 [Wed Feb 6 09:39:54 2019][1697168.238414] Call Trace: [Wed Feb 6 09:39:54 2019][1697168.241043] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:54 2019][1697168.247135] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1697168.252283] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:54 2019][1697168.259591] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1697168.265362] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.272680] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:54 2019][1697168.279225] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.286466] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.293699] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.301015] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:54 2019][1697168.307991] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.315749] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.322985] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.330744] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.338397] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:54 2019][1697168.344838] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:54 2019][1697168.350855] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:54 2019][1697168.357401] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.364192] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:54 2019][1697168.370638] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:54 2019][1697168.376299] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:54 2019][1697168.383874] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1697168.388934] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697168.395200] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1697168.401821] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697168.408089] osp-pre-17-1 S ffffa08b5d254100 0 126926 2 0x00000080 [Wed Feb 6 09:39:54 2019][1697168.415452] Call Trace: [Wed Feb 6 09:39:54 2019][1697168.418080] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1697168.423234] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:54 2019][1697168.430196] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1697168.435947] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:54 2019][1697168.442651] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1697168.447712] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697168.453978] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1697168.460589] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697168.466858] osp-syn-0-1 S ffffa08b5d255140 0 126927 2 0x00000080 [Wed Feb 6 09:39:54 2019][1697168.474223] Call Trace: [Wed Feb 6 09:39:54 2019][1697168.476850] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:54 2019][1697168.482950] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1697168.488095] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:54 2019][1697168.495407] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1697168.501174] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.508488] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:54 2019][1697168.515031] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.522272] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.529506] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.536835] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.544595] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.551833] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.559587] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.567243] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:54 2019][1697168.573684] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:54 2019][1697168.579700] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:54 2019][1697168.586265] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.593057] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:54 2019][1697168.599502] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:54 2019][1697168.605175] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:54 2019][1697168.612745] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1697168.617807] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697168.624083] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1697168.630703] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697168.636975] osp-pre-0-1 S ffffa08b5d256180 0 126928 2 0x00000080 [Wed Feb 6 09:39:54 2019][1697168.644332] Call Trace: [Wed Feb 6 09:39:54 2019][1697168.646961] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1697168.652106] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:54 2019][1697168.659069] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1697168.664818] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:54 2019][1697168.671522] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1697168.676577] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697168.682853] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1697168.689473] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697168.695746] osp-syn-5-1 S ffffa08b5db70000 0 126929 2 0x00000080 [Wed Feb 6 09:39:54 2019][1697168.703103] Call Trace: [Wed Feb 6 09:39:54 2019][1697168.705737] [] ? osd_ldiskfs_read+0x131/0x2d0 [osd_ldiskfs] [Wed Feb 6 09:39:54 2019][1697168.713132] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:54 2019][1697168.719226] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1697168.724378] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:54 2019][1697168.731689] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1697168.737459] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.744780] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:54 2019][1697168.751323] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.758561] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.765796] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.773121] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.780879] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.788114] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.795869] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.803529] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:54 2019][1697168.809975] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:54 2019][1697168.815995] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:54 2019][1697168.822538] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.829331] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:54 2019][1697168.835778] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:54 2019][1697168.841448] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:54 2019][1697168.849018] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1697168.854069] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697168.860337] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1697168.866951] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697168.873223] osp-pre-5-1 S ffffa08b5db71040 0 126930 2 0x00000080 [Wed Feb 6 09:39:54 2019][1697168.880586] Call Trace: [Wed Feb 6 09:39:54 2019][1697168.883218] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1697168.888370] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:54 2019][1697168.895335] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1697168.901094] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:54 2019][1697168.907796] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1697168.912850] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697168.919124] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1697168.925737] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697168.932012] osp-syn-44-1 S ffffa08b5db72080 0 126931 2 0x00000080 [Wed Feb 6 09:39:54 2019][1697168.939376] Call Trace: [Wed Feb 6 09:39:54 2019][1697168.942044] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:54 2019][1697168.949615] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1697168.954765] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:54 2019][1697168.962077] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1697168.967847] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.975159] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:54 2019][1697168.981702] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.988943] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:54 2019][1697168.996179] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:54 2019][1697169.003500] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:54 2019][1697169.011257] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:54 2019][1697169.018494] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:54 2019][1697169.026251] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:54 2019][1697169.033908] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:54 2019][1697169.040357] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:54 2019][1697169.046374] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:54 2019][1697169.052922] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:54 2019][1697169.059722] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:54 2019][1697169.066164] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:54 2019][1697169.071837] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:54 2019][1697169.079406] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1697169.084461] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697169.090734] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1697169.097348] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697169.103621] osp-pre-44-1 S ffffa08b5db730c0 0 126932 2 0x00000080 [Wed Feb 6 09:39:54 2019][1697169.110976] Call Trace: [Wed Feb 6 09:39:54 2019][1697169.113606] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1697169.118752] [] osp_precreate380 [osp] [Wed Feb 6 09:39:54 2019][1697169.125715] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1697169.131475] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:54 2019][1697169.138177] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1697169.143239] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697169.149513] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1697169.156127] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1697169.162401] osp-syn-32-1 S ffffa08b5db74100 0 126933 2 0x00000080 [Wed Feb 6 09:39:54 2019][1697169.169765] Call Trace: [Wed Feb 6 09:39:54 2019][1697169.172431] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:55 2019][1697169.180004] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1697169.185157] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:55 2019][1697169.192465] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1697169.198228] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.205541] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:55 2019][1697169.212084] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.219323] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.226568] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.233892] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.241656] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.248895] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.256651] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.264306] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:55 2019][1697169.270753] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:55 2019][1697169.276771] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:55 2019][1697169.283316] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.290110] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:55 2019][1697169.296553] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:55 2019][1697169.302216] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:55 2019][1697169.309787] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1697169.314842] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697169.321116] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1697169.327736] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697169.334002] osp-pre-32-1 S ffffa08b5db75140 0 126934 2 0x00000080 [Wed Feb 6 09:39:55 2019][1697169.341358] Call Trace: [Wed Feb 6 09:39:55 2019][1697169.343988] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:55 2019][1697169.349396] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1697169.354548] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:55 2019][1697169.361512] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1697169.367271] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:55 2019][1697169.373974] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1697169.379035] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697169.385302] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1697169.391923] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697169.398190] osp-syn-21-1 S ffffa08b5db76180 0 126935 2 0x00000080 [Wed Feb 6 09:39:55 2019][1697169.405553] Call Trace: [Wed Feb 6 09:39:55 2019][1697169.408180] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:55 2019][1697169.414276] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1697169.419425] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:55 2019][1697169.426738] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1697169.432508] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.439819] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:55 2019][1697169.446368] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.453608] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.460847] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.468170] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.475926] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.483164] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.490920] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.498575] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:55 2019][1697169.505016] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:55 2019][1697169.511035] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:55 2019][1697169.517581] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.524374] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:55 2019][1697169.530817] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:55 2019][1697169.536488] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:55 2019][1697169.544058] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1697169.549111] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697169.555380] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1697169.561999] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697169.568266] osp-pre-21-1 S ffffa08b5d170000 0 126936 2 0x00000080 [Wed Feb 6 09:39:55 2019][1697169.575630] Call Trace: [Wed Feb 6 09:39:55 2019][1697169.578259] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1697169.583410] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:55 2019][1697169.590376] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1697169.596133] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:55 2019][1697169.602837] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1697169.607890] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697169.614158] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1697169.620777] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697169.627043] osp-syn-16-1 S ffffa08b5d171040 0 126937 2 0x00000080 [Wed Feb 6 09:39:55 2019][1697169.634399] Call Trace: [Wed Feb 6 09:39:55 2019][1697169.637029] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:55 2019][1697169.643129] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1697169.648276] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:55 2019][1697169.655585] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1697169.661354] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.668664] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:55 2019][1697169.675212] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.682447] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.689684] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.697008] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.704766] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.712000] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.719759] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.727415] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:55 2019][1697169.733862] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:55 2019][1697169.739881] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:55 2019][1697169.746424] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.753219] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:55 2019][1697169.759662] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:55 2019][1697169.765327] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:55 2019][1697169.772897] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1697169.777960] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697169.784232] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1697169.790845] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697169.797112] osp-pre-16-1 S ffffa08b5d172080 0 126938 2 0x00000080 [Wed Feb 6 09:39:55 2019][1697169.804484] Call Trace: [Wed Feb 6 09:39:55 2019][1697169.807113] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1697169.812258] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:55 2019][1697169.819223] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1697169.824979] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:55 2019][1697169.831684] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1697169.836745] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697169.843012] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1697169.849633] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697169.855908] osp-syn-41-1 S ffffa08b5d1730c0 0 126939 2 0x00000080 [Wed Feb 6 09:39:55 2019][1697169.863270] Call Trace: [Wed Feb 6 09:39:55 2019][1697169.865935] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:55 2019][1697169.873510] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1697169.878659] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:55 2019][1697169.885971] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1697169.891733] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.899048] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:55 2019][1697169.905592] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.912827] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.920063] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.927392] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.935153] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.942391] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.950145] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.957802] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:55 2019][1697169.964241] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:55 2019][1697169.970251] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:55 2019][1697169.976797] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:55 2019][1697169.983589] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:55 2019][1697169.990034] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:55 2019][1697169.995706] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:55 2019][1697170.003277] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1697170.008338] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697170.014605] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1697170.021218] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697170.027492] osp-pre-41-1 S ffffa08b5d174100 0 126940 2 0x00000080 [Wed Feb 6 09:39:55 2019][1697170.034848] Call Trace: [Wed Feb 6 09:39:55 2019][1697170.037475] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:55 2019][1697170.042875] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1697170.048020] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:55 2019][1697170.054983] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1697170.060731] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:55 2019][1697170.067436] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1697170.072490] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697170.078765] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1697170.085380] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1697170.091654] osp-syn-4-1 S ffffa08b5d175140 0 126941 2 0x00000080 [Wed Feb 6 09:39:55 2019][1697170.099015] Call Trace: [Wed Feb 6 09:39:55 2019][1697170.101646] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:55 2019][1697170.107746] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1697170.112890] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:55 2019][1697170.120202] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1697170.125971] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:55 2019][1697170.133285] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:55 2019][1697170.139826] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:55 2019][1697170.147065] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:55 2019][1697170.154301] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:55 2019][1697170.161618] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:55 2019][1697170.168594] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.176352] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.183586] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.191343] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.199001] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:56 2019][1697170.205460] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:56 2019][1697170.211476] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:56 2019][1697170.218018] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.224813] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:56 2019][1697170.231256] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:56 2019][1697170.236920] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:56 2019][1697170.244490] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1697170.249545] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697170.255819] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1697170.262438] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697170.268707] osp-pre-4-1 S ffffa08b5d176180 0 126942 2 0x00000080 [Wed Feb 6 09:39:56 2019][1697170.276068] Call Trace: [Wed Feb 6 09:39:56 2019][1697170.278698] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1697170.283841] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:56 2019][1697170.290807] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:56 2019][1697170.296565] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:56 2019][1697170.303270] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1697170.308332] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697170.314604] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1697170.321218] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697170.327484] osp-syn-23-1 S ffffa07c29570000 0 126943 2 0x00000080 [Wed Feb 6 09:39:56 2019][1697170.334839] Call Trace: [Wed Feb 6 09:39:56 2019][1697170.337469] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:56 2019][1697170.343568] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1697170.348714] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:56 2019][1697170.356024] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:56 2019][1697170.361794] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.369106] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:56 2019][1697170.375651] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.382887] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.390124] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.397449] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.405206] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.412441] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.420199] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.427855] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:56 2019][1697170.434302] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:56 2019][1697170.440311] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:56 2019][1697170.446858] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.453648] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:56 2019][1697170.460094] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:56 2019][1697170.465757] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:56 2019][1697170.473329] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1697170.478391] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697170.484664] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1697170.491276] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697170.497545] osp-pre-23-1 S ffffa07c29571040 0 126944 2 0x00000080 [Wed Feb 6 09:39:56 2019][1697170.504908] Call Trace: [Wed Feb 6 09:39:56 2019][1697170.507535] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1697170.512681] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:56 2019][1697170.519643] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:56 2019][1697170.525393] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:56 2019][1697170.532098] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1697170.537159] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697170.543427] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1697170.550045] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697170.556314] osp-syn-33-1 S ffffa07c29572080 0 126945 2 0x00000080 [Wed Feb 6 09:39:56 2019][1697170.563676] Call Trace: [Wed Feb 6 09:39:56 2019][1697170.566336] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:56 2019][1697170.573906] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1697170.579051] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:56 2019][1697170.586383] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:56 2019][1697170.592146] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.599459] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:56 2019][1697170.606006] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.613242] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.620479] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.627803] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.635559] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.642795] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.650567] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.658226] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:56 2019][1697170.664672] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:56 2019][1697170.670684] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:56 2019][1697170.677231] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.684024] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:56 2019][1697170.690467] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:56 2019][1697170.696140] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:56 2019][1697170.703709] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1697170.708768] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697170.715038] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1697170.721657] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697170.727924] osp-pre-33-1 S ffffa07c295730c0 0 126946 2 0x00000080 [Wed Feb 6 09:39:56 2019][1697170.735280] Call Trace: [Wed Feb 6 09:39:56 2019][1697170.737908] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1697170.743059] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:56 2019][1697170.750025] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:56 2019][1697170.755783] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:56 2019][1697170.762486] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1697170.767539] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697170.773808] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1697170.780428] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697170.786703] osp-syn-20-1 S ffffa07c29574100 0 126947 2 0x00000080 [Wed Feb 6 09:39:56 2019][1697170.794065] Call Trace: [Wed Feb 6 09:39:56 2019][1697170.796695] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:56 2019][1697170.802796] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1697170.807948] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:56 2019][1697170.815260] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:56 2019][1697170.821029] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.828340] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:56 2019][1697170.834884] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.842121] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.849361] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.856675] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:56 2019][1697170.863652] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.871408] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.878645] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.886401] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.894058] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:56 2019][1697170.900505] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:56 2019][1697170.906515] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:56 2019][1697170.913061] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:56 2019][1697170.919854] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:56 2019][1697170.926299] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:56 2019][1697170.931969] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:56 2019][1697170.939540] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1697170.944595] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697170.950868] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1697170.957481] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697170.963754] osp-pre-20-1 S ffffa07c29575140 0 126948 2 0x00000080 [Wed Feb 6 09:39:56 2019][1697170.971111] Call Trace: [Wed Feb 6 09:39:56 2019][1697170.973739] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1697170.978881] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:56 2019][1697170.985846] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:56 2019][1697170.991597] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:56 2019][1697170.998300] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1697171.003355] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697171.009630] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1697171.016250] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1697171.022525] osp-syn-7-1 S ffffa07c29576180 0 126949 2 0x00000080 [Wed Feb 6 09:39:56 2019][1697171.029889] Call Trace: [Wed Feb 6 09:39:56 2019][1697171.032518] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:56 2019][1697171.038619] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1697171.043771] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:56 2019][1697171.051081] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:56 2019][1697171.056844] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:56 2019][1697171.064157] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:56 2019][1697171.070700] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:56 2019][1697171.077941] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:56 2019][1697171.085182] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:56 2019][1697171.092506] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:56 2019][1697171.100261] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:56 2019][1697171.107499] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:56 2019][1697171.115256] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:56 2019][1697171.122911] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:56 2019][1697171.129351] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:56 2019][1697171.135360] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:56 2019][1697171.141907] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:56 2019][1697171.148698] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:56 2019][1697171.155144] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:56 2019][1697171.160816] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:56 2019][1697171.168387] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1697171.173448] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.179715] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1697171.186337] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.192609] osp-pre-7-1 S ffffa08b5d188000 0 126950 2 0x00000080 [Wed Feb 6 09:39:57 2019][1697171.199965] Call Trace: [Wed Feb 6 09:39:57 2019][1697171.202594] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:57 2019][1697171.207995] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1697171.213145] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:57 2019][1697171.220110] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1697171.225861] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:57 2019][1697171.232562] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1697171.237617] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.243884] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1697171.250505] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.256779] osp-syn-40-1 S ffffa08b5d189040 0 126951 2 0x00000080 [Wed Feb 6 09:39:57 2019][1697171.264135] Call Trace: [Wed Feb 6 09:39:57 2019][1697171.266764] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:57 2019][1697171.272866] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1697171.278018] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:57 2019][1697171.285328] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1697171.291098] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.298410] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:57 2019][1697171.304953] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.312192] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.319427] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.326751] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.334508] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.341744] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.349502] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.357159] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:57 2019][1697171.363607] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:57 2019][1697171.369624] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:57 2019][1697171.376172] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.382963] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:57 2019][1697171.389407] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:57 2019][1697171.395081] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:57 2019][1697171.402651] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1697171.407711] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.413978] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1697171.420598] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.426866] osp-pre-40-1 S ffffa08b5d18a080 0 126952 2 0x00000080 [Wed Feb 6 09:39:57 2019][1697171.434229] Call Trace: [Wed Feb 6 09:39:57 2019][1697171.436857] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1697171.442011] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:57 2019][1697171.448974] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1697171.454733] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:57 2019][1697171.461435] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1697171.466489] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.472757] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1697171.479375] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.485642] osp-syn-45-1 S ffffa08b5d18b0c0 0 126953 2 0x00000080 [Wed Feb 6 09:39:57 2019][1697171.492998] Call Trace: [Wed Feb 6 09:39:57 2019][1697171.495628] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:57 2019][1697171.501728] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1697171.506871] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:57 2019][1697171.514184] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1697171.519951] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.527263] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:57 2019][1697171.533811] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.541047] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.548282] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.555607] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.563365] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.570598] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.578356] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.586013] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:57 2019][1697171.592460] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:57 2019][1697171.598471] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:57 2019][1697171.605017] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.611810] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:57 2019][1697171.618254] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:57 2019][1697171.623925] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:57 2019][1697171.631496] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1697171.636549] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.642816] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1697171.649436] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.655703] osp-pre-45-1 S ffffa08b5d18c100 0 126954 2 0x00000080 [Wed Feb 6 09:39:57 2019][1697171.663066] Call Trace: [Wed Feb 6 09:39:57 2019][1697171.665697] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1697171.670847] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:57 2019][1697171.677812] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1697171.683571] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:57 2019][1697171.690273] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1697171.695328] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.701603] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1697171.708223] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.714498] osp-syn-6-1 S ffffa08b5d18d140 0 126955 2 0x00000080 [Wed Feb 6 09:39:57 2019][1697171.721862] Call Trace: [Wed Feb 6 09:39:57 2019][1697171.724497] [] ? osd_ldiskfs_read+0x131/0x2d0 [osd_ldiskfs] [Wed Feb 6 09:39:57 2019][1697171.731893] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:57 2019][1697171.737994] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1697171.743145] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:57 2019][1697171.750457] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1697171.756226] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.763539] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:57 2019][1697171.770083] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.777321] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.784555] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.791874] [] ? mdd_index_delete.isra.33+0x40/0x4e0 [mdd] [Wed Feb 6 09:39:57 2019][1697171.799196] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.806952] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.814188] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.821946] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.829600] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:57 2019][1697171.836040] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:57 2019][1697171.842050] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:57 2019][1697171.848595] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:57 2019][1697171.855393] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:57 2019][1697171.861834] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:57 2019][1697171.867506] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:57 2019][1697171.875075] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1697171.880134] [] ? mdd_dir_declare_destroy_stripe+0x2c0/0x410 [mdd] [Wed Feb 6 09:39:57 2019][1697171.888051] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.894326] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1697171.900946] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.907212] osp-pre-6-1 S ffffa08b5d18e180 0 126956 2 0x00000080 [Wed Feb 6 09:39:57 2019][1697171.914568] Call Trace: [Wed Feb 6 09:39:57 2019][1697171.917197] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1697171.922349] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:57 2019][1697171.929312] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1697171.935063] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:57 2019][1697171.941767] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1697171.946827] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.953095] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1697171.959717] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697171.965992] osp-syn-24-1 S ffffa08b603a0000 0 126957 2 0x00000080 [Wed Feb 6 09:39:57 2019][1697171.973353] Call Trace: [Wed Feb 6 09:39:57 2019][1697171.975983] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:57 2019][1697171.982074] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1697171.987219] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:57 2019][1697171.994530] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1697172.000301] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:57 2019][1697172.007612] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:57 2019][1697172.014155] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:57 2019][1697172.021396] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:57 2019][1697172.028637] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:57 2019][1697172.035963] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:57 2019][1697172.043719] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:57 2019][1697172.050959] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:57 2019][1697172.058721] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:57 2019][1697172.066379] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:57 2019][1697172.072824] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:57 2019][1697172.078836] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:57 2019][1697172.085384] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:57 2019][1697172.092184] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:57 2019][1697172.098627] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:57 2019][1697172.104298] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:57 2019][1697172.111871] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1697172.116931] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697172.123196] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1697172.129810] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1697172.136084] osp-pre-24-1 S ffffa08b603a1040 0 126958 2 0x00000080 [Wed Feb 6 09:39:57 2019][1697172.143440] Call Trace: [Wed Feb 6 09:39:57 2019][1697172.146069] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:57 2019][1697172.151476] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1697172.156620] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:57 2019][1697172.163585] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1697172.169344] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:57 2019][1697172.176047] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1697172.181101] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697172.187374] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1697172.193986] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697172.200254] osp-syn-47-1 S ffffa08b603a2080 0 126959 2 0x00000080 [Wed Feb 6 09:39:58 2019][1697172.207616] Call Trace: [Wed Feb 6 09:39:58 2019][1697172.210282] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:58 2019][1697172.217856] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1697172.223009] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:58 2019][1697172.230320] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:58 2019][1697172.236087] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.243401] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:58 2019][1697172.249944] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.257183] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.264421] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.271745] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.279499] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.286737] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.294493] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.302150] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:58 2019][1697172.308597] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:58 2019][1697172.314614] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:58 2019][1697172.321163] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.327954] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:58 2019][1697172.334400] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:58 2019][1697172.340069] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:58 2019][1697172.347641] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1697172.352701] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697172.358969] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1697172.365589] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697172.371864] osp-pre-47-1 S ffffa08b603a30c0 0 126960 2 0x00000080 [Wed Feb 6 09:39:58 2019][1697172.379229] Call Trace: [Wed Feb 6 09:39:58 2019][1697172.381858] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1697172.387010] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:58 2019][1697172.393974] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:58 2019][1697172.399733] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:58 2019][1697172.406437] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1697172.411497] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697172.417762] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1697172.424378] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697172.430650] osp-syn-38-1 S ffffa08b603a4100 0 126961 2 0x00000080 [Wed Feb 6 09:39:58 2019][1697172.438006] Call Trace: [Wed Feb 6 09:39:58 2019][1697172.440634] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:58 2019][1697172.446729] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1697172.451878] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:58 2019][1697172.459191] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:58 2019][1697172.464955] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.472273] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:58 2019][1697172.478818] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.486055] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.493292] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.500614] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.508374] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.515608] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.523365] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.531020] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:58 2019][1697172.537461] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:58 2019][1697172.543478] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:58 2019][1697172.550024] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.556816] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:58 2019][1697172.563262] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:58 2019][1697172.568933] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:58 2019][1697172.576504] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1697172.581566] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697172.587856] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1697172.594470] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697172.600735] osp-pre-38-1 S ffffa08b603a5140 0 126962 2 0x00000080 [Wed Feb 6 09:39:58 2019][1697172.608090] Call Trace: [Wed Feb 6 09:39:58 2019][1697172.610722] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:58 2019][1697172.616127] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1697172.621274] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:58 2019][1697172.628236] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:58 2019][1697172.633994] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:58 2019][1697172.640698] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1697172.645751] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697172.652019] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1697172.658639] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697172.664905] osp-syn-15-1 S ffffa08b603a6180 0 126963 2 0x00000080 [Wed Feb 6 09:39:58 2019][1697172.672260] Call Trace: [Wed Feb 6 09:39:58 2019][1697172.674896] [] ? osd_ldiskfs_read+0x131/0x2d0 [osd_ldiskfs] [Wed Feb 6 09:39:58 2019][1697172.682291] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:58 2019][1697172.688396] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1697172.693544] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:58 2019][1697172.700857] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:58 2019][1697172.706624] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.713938] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:58 2019][1697172.720481] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.727720] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.734955] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.742271] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:58 2019][1697172.749246] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.757005] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.764239] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.771997] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.779655] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:58 2019][1697172.786102] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:58 2019][1697172.792120] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:58 2019][1697172.798666] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.805461] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:58 2019][1697172.811902] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:58 2019][1697172.817567] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:58 2019][1697172.825138] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1697172.830197] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697172.836465] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1697172.843085] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697172.849351] osp-pre-15-1 S ffffa08b764e0000 0 126964 2 0x00000080 [Wed Feb 6 09:39:58 2019][1697172.856706] Call Trace: [Wed Feb 6 09:39:58 2019][1697172.859336] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1697172.864487] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:58 2019][1697172.871453] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:58 2019][1697172.877213] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:58 2019][1697172.883914] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1697172.888968] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697172.895243] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1697172.901863] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697172.908130] osp-syn-29-1 S ffffa08b764e1040 0 126965 2 0x00000080 [Wed Feb 6 09:39:58 2019][1697172.915492] Call Trace: [Wed Feb 6 09:39:58 2019][1697172.918130] [] ? osd_ldiskfs_read+0x131/0x2d0 [osd_ldiskfs] [Wed Feb 6 09:39:58 2019][1697172.925523] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:58 2019][1697172.931618] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1697172.936770] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:58 2019][1697172.944080] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:58 2019][1697172.949850] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.957161] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:58 2019][1697172.963709] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.970942] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.978181] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.985503] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:58 2019][1697172.993258] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:58 2019][1697173.000496] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:58 2019][1697173.008268] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:58 2019][1697173.015926] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:58 2019][1697173.022367] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:58 2019][1697173.028384] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:58 2019][1697173.034932] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:58 2019][1697173.041725] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:58 2019][1697173.048183] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:58 2019][1697173.053848] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:58 2019][1697173.061420] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1697173.066479] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697173.072746] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1697173.079374] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697173.085642] osp-pre-29-1 S ffffa08b764e2080 0 126966 2 0x00000080 [Wed Feb 6 09:39:58 2019][1697173.092997] Call Trace: [Wed Feb 6 09:39:58 2019][1697173.095628] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1697173.100778] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:58 2019][1697173.107743] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:58 2019][1697173.113493] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:58 2019][1697173.120196] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1697173.125251] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697173.131525] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1697173.138146] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1697173.144412] osp-syn-1-1 S ffffa08b764e30c0 0 126967 2 0x00000080 [Wed Feb 6 09:39:58 2019][1697173.151766] Call Trace: [Wed Feb 6 09:39:58 2019][1697173.154398] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:58 2019][1697173.160496] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1697173.165640] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:58 2019][1697173.172953] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1697173.178723] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.186035] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:59 2019][1697173.192577] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.199817] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.207052] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.214376] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.222132] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.229369] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.237127] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.244783] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:59 2019][1697173.251232] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:59 2019][1697173.257247] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:59 2019][1697173.263808] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.270605] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:59 2019][1697173.277049] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:59 2019][1697173.282737] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:59 2019][1697173.290308] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1697173.295360] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697173.301627] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1697173.308240] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697173.314515] osp-pre-1-1 S ffffa08b764e4100 0 126968 2 0x00000080 [Wed Feb 6 09:39:59 2019][1697173.321871] Call Trace: [Wed Feb 6 09:39:59 2019][1697173.324501] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:59 2019][1697173.329908] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1697173.335059] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:59 2019][1697173.342024] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1697173.347782] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:59 2019][1697173.354486] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1697173.359549] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697173.365822] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1697173.372436] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697173.378710] osp-syn-22-1 S ffffa08b764e5140 0 126969 2 0x00000080 [Wed Feb 6 09:39:59 2019][1697173.386072] Call Trace: [Wed Feb 6 09:39:59 2019][1697173.388709] [] ? osd_ldiskfs_read+0x131/0x2d0 [osd_ldiskfs] [Wed Feb 6 09:39:59 2019][1697173.396103] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:59 2019][1697173.402196] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1697173.407341] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:59 2019][1697173.414650] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1697173.420417] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.427749] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:59 2019][1697173.434295] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.441533] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.448767] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.456092] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.463864] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.471103] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.478862] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.486517] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:59 2019][1697173.492963] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:59 2019][1697173.498976] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:59 2019][1697173.505518] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.512313] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:59 2019][1697173.518772] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:59 2019][1697173.524437] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:59 2019][1697173.532007] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1697173.537061] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697173.543337] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1697173.549955] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697173.556222] osp-pre-22-1 S ffffa08b764e6180 0 126970 2 0x00000080 [Wed Feb 6 09:39:59 2019][1697173.563579] Call Trace: [Wed Feb 6 09:39:59 2019][1697173.566207] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1697173.571358] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:59 2019][1697173.578323] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1697173.584088] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:59 2019][1697173.590795] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1697173.595855] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697173.602122] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1697173.608735] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697173.615023] osp-syn-3-1 S ffffa08b793a0000 0 126971 2 0x00000080 [Wed Feb 6 09:39:59 2019][1697173.622382] Call Trace: [Wed Feb 6 09:39:59 2019][1697173.625012] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:59 2019][1697173.631114] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1697173.636267] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:59 2019][1697173.643574] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1697173.649335] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.656649] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:59 2019][1697173.663196] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.670447] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.677683] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.685007] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.692765] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.700002] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.707758] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.715414] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:59 2019][1697173.721861] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:59 2019][1697173.727872] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:59 2019][1697173.734421] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.741219] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:59 2019][1697173.747664] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:59 2019][1697173.753335] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:59 2019][1697173.760906] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1697173.765967] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697173.772242] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1697173.778864] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697173.785139] osp-pre-3-1 S ffffa08b793a1040 0 126972 2 0x00000080 [Wed Feb 6 09:39:59 2019][1697173.792503] Call Trace: [Wed Feb 6 09:39:59 2019][1697173.795132] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1697173.800284] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:59 2019][1697173.807262] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1697173.813015] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:59 2019][1697173.819719] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1697173.824782] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697173.831056] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1697173.837678] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697173.843951] osp-syn-27-1 S ffffa08b793a2080 0 126973 2 0x00000080 [Wed Feb 6 09:39:59 2019][1697173.851324] Call Trace: [Wed Feb 6 09:39:59 2019][1697173.853954] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:59 2019][1697173.860054] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1697173.865207] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:59 2019][1697173.872517] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1697173.878287] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.885600] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:59 2019][1697173.892144] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.899382] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.906618] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.913941] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.921716] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.928951] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.936708] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.944365] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:59 2019][1697173.950814] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:59 2019][1697173.956832] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:59 2019][1697173.963376] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:59 2019][1697173.970172] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:59 2019][1697173.976614] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:59 2019][1697173.982285] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:59 2019][1697173.989855] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1697173.994909] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697174.001174] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1697174.007787] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697174.014052] osp-pre-27-1 S ffffa08b793a30c0 0 126974 2 0x00000080 [Wed Feb 6 09:39:59 2019][1697174.021410] Call Trace: [Wed Feb 6 09:39:59 2019][1697174.024039] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:59 2019][1697174.029446] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1697174.034589] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:59 2019][1697174.041554] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1697174.047312] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:59 2019][1697174.054016] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1697174.059078] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697174.065345] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1697174.071967] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1697174.078241] osp-syn-37-1 S ffffa08b793a4100 0 126975 2 0x00000080 [Wed Feb 6 09:39:59 2019][1697174.085606] Call Trace: [Wed Feb 6 09:39:59 2019][1697174.088234] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:59 2019][1697174.094335] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1697174.099487] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:59 2019][1697174.106796] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1697174.112558] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:59 2019][1697174.119871] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:59 2019][1697174.126436] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:59 2019][1697174.133671] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:59 2019][1697174.140909] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:59 2019][1697174.148239] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:59 2019][1697174.156000] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:59 2019][1697174.163240] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:59 2019][1697174.170997] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.178655] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:00 2019][1697174.185102] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:00 2019][1697174.191137] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:40:00 2019][1697174.197682] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.204479] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:40:00 2019][1697174.210921] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:40:00 2019][1697174.216593] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:40:00 2019][1697174.224161] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1697174.229215] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697174.235489] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1697174.242103] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697174.248376] osp-pre-37-1 S ffffa08b793a5140 0 126976 2 0x00000080 [Wed Feb 6 09:40:00 2019][1697174.255732] Call Trace: [Wed Feb 6 09:40:00 2019][1697174.258363] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1697174.263527] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:40:00 2019][1697174.270495] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1697174.276255] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:40:00 2019][1697174.282958] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1697174.288019] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697174.294294] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1697174.300906] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697174.307187] osp-syn-36-1 S ffffa08b793a6180 0 126977 2 0x00000080 [Wed Feb 6 09:40:00 2019][1697174.314554] Call Trace: [Wed Feb 6 09:40:00 2019][1697174.317181] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:40:00 2019][1697174.323276] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1697174.328419] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:40:00 2019][1697174.335730] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1697174.341512] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.348833] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:40:00 2019][1697174.355373] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.362611] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.369850] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.377171] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.384926] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.392166] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.399923] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.407578] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:00 2019][1697174.414026] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:00 2019][1697174.420044] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:40:00 2019][1697174.426589] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.433385] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:40:00 2019][1697174.439827] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:40:00 2019][1697174.445499] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:40:00 2019][1697174.453071] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1697174.458133] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697174.464426] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1697174.471045] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697174.477321] osp-pre-36-1 S ffffa08b61ca0000 0 126978 2 0x00000080 [Wed Feb 6 09:40:00 2019][1697174.484684] Call Trace: [Wed Feb 6 09:40:00 2019][1697174.487312] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:40:00 2019][1697174.492720] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1697174.497862] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:40:00 2019][1697174.504829] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1697174.510586] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:40:00 2019][1697174.517292] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1697174.522353] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697174.528626] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1697174.535239] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697174.541506] osp-syn-8-1 S ffffa08b61ca1040 0 126979 2 0x00000080 [Wed Feb 6 09:40:00 2019][1697174.548868] Call Trace: [Wed Feb 6 09:40:00 2019][1697174.551533] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:00 2019][1697174.559122] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1697174.564268] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:40:00 2019][1697174.571579] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1697174.577348] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.584676] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:40:00 2019][1697174.591224] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.598461] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.605696] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.613011] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Wed Feb 6 09:40:00 2019][1697174.619988] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.627745] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.634981] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.642740] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.650395] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:00 2019][1697174.656856] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:00 2019][1697174.662869] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:40:00 2019][1697174.669421] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.676215] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:40:00 2019][1697174.682676] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:40:00 2019][1697174.688343] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:40:00 2019][1697174.695913] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1697174.700974] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697174.707240] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1697174.713861] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697174.720129] osp-pre-8-1 S ffffa08b61ca2080 0 126980 2 0x00000080 [Wed Feb 6 09:40:00 2019][1697174.727492] Call Trace: [Wed Feb 6 09:40:00 2019][1697174.730134] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:40:00 2019][1697174.735538] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1697174.740691] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:40:00 2019][1697174.747655] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1697174.753412] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:40:00 2019][1697174.760116] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1697174.765179] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697174.771451] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1697174.778066] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697174.784339] osp-syn-35-1 S ffffa08b61ca30c0 0 126981 2 0x00000080 [Wed Feb 6 09:40:00 2019][1697174.791695] Call Trace: [Wed Feb 6 09:40:00 2019][1697174.794356] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:00 2019][1697174.801924] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1697174.807069] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:40:00 2019][1697174.814395] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1697174.820157] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.827469] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:40:00 2019][1697174.834030] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.841269] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.848510] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.855837] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.863611] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.870848] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.878604] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.886261] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:00 2019][1697174.892699] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:00 2019][1697174.898710] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:40:00 2019][1697174.905256] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:40:00 2019][1697174.912048] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:40:00 2019][1697174.918495] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:40:00 2019][1697174.924166] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:40:00 2019][1697174.931737] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1697174.936799] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697174.943072] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1697174.949692] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697174.955973] osp-pre-35-1 S ffffa08b61ca4100 0 126982 2 0x00000080 [Wed Feb 6 09:40:00 2019][1697174.963330] Call Trace: [Wed Feb 6 09:40:00 2019][1697174.965961] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:40:00 2019][1697174.971370] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1697174.976521] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:40:00 2019][1697174.983500] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1697174.989251] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:40:00 2019][1697174.995958] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1697175.001018] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697175.007283] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1697175.013896] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697175.020163] osp-syn-28-1 S ffffa08b61ca5140 0 126983 2 0x00000080 [Wed Feb 6 09:40:00 2019][1697175.027518] Call Trace: [Wed Feb 6 09:40:00 2019][1697175.030147] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:40:00 2019][1697175.036239] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1697175.041383] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:40:00 2019][1697175.048695] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1697175.054462] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:40:00 2019][1697175.061777] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:40:00 2019][1697175.068323] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:40:00 2019][1697175.075558] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:40:00 2019][1697175.082794] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:40:00 2019][1697175.090117] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:40:00 2019][1697175.097873] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:40:00 2019][1697175.105112] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:40:00 2019][1697175.112867] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:40:00 2019][1697175.120525] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:00 2019][1697175.126973] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:00 2019][1697175.133005] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:40:00 2019][1697175.139552] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:40:00 2019][1697175.146347] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:40:00 2019][1697175.152792] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:40:00 2019][1697175.158465] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:40:00 2019][1697175.166034] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1697175.171095] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1697175.177371] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1697175.183990] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.190257] osp-pre-28-1 S ffffa08b61ca6180 0 126984 2 0x00000080 [Wed Feb 6 09:40:01 2019][1697175.197621] Call Trace: [Wed Feb 6 09:40:01 2019][1697175.200251] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:40:01 2019][1697175.205672] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1697175.210819] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:40:01 2019][1697175.217782] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1697175.223530] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:40:01 2019][1697175.230251] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:01 2019][1697175.235308] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.241583] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1697175.248205] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.254481] osp-syn-30-1 S ffffa08b61cb8000 0 126985 2 0x00000080 [Wed Feb 6 09:40:01 2019][1697175.261843] Call Trace: [Wed Feb 6 09:40:01 2019][1697175.264470] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:40:01 2019][1697175.270564] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1697175.275731] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:40:01 2019][1697175.283044] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1697175.288815] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.296141] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:40:01 2019][1697175.302687] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.309923] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.317160] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.324485] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.332241] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.339479] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.347234] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.354891] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:01 2019][1697175.361331] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:01 2019][1697175.367350] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:40:01 2019][1697175.373893] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.380703] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:40:01 2019][1697175.387151] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:40:01 2019][1697175.392820] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:40:01 2019][1697175.400391] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:01 2019][1697175.405444] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.411711] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1697175.418331] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.424613] osp-pre-30-1 S ffffa08b61cb9040 0 126986 2 0x00000080 [Wed Feb 6 09:40:01 2019][1697175.431970] Call Trace: [Wed Feb 6 09:40:01 2019][1697175.434600] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:40:01 2019][1697175.440008] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1697175.445160] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:40:01 2019][1697175.452123] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1697175.457890] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:40:01 2019][1697175.464596] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:01 2019][1697175.469657] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.475948] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1697175.482560] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.488843] osp-syn-34-1 S ffffa08b61cba080 0 126987 2 0x00000080 [Wed Feb 6 09:40:01 2019][1697175.496199] Call Trace: [Wed Feb 6 09:40:01 2019][1697175.498862] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:01 2019][1697175.506429] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1697175.511572] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:40:01 2019][1697175.518884] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1697175.524645] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.531959] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:40:01 2019][1697175.538501] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.545738] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.552979] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.560299] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.568061] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.575300] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.583058] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.590715] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:01 2019][1697175.597177] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:01 2019][1697175.603188] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:40:01 2019][1697175.609734] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:40:01 2019][1697175.616530] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:40:01 2019][1697175.622972] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:40:01 2019][1697175.628645] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:40:01 2019][1697175.636215] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:01 2019][1697175.641290] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.647562] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1697175.654182] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.660456] osp-pre-34-1 S ffffa08b61cbb0c0 0 126988 2 0x00000080 [Wed Feb 6 09:40:01 2019][1697175.667819] Call Trace: [Wed Feb 6 09:40:01 2019][1697175.670451] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:40:01 2019][1697175.675858] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1697175.681009] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:40:01 2019][1697175.687988] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1697175.693739] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:40:01 2019][1697175.700457] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:01 2019][1697175.705513] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.711781] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1697175.718402] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.724668] lquota_wb_fir-M S ffffa07b4eb4b0c0 0 127047 2 0x00000080 [Wed Feb 6 09:40:01 2019][1697175.732023] Call Trace: [Wed Feb 6 09:40:01 2019][1697175.734651] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1697175.739792] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:40:01 2019][1697175.745808] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:40:01 2019][1697175.752346] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Wed Feb 6 09:40:01 2019][1697175.759144] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Wed Feb 6 09:40:01 2019][1697175.765757] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1697175.771517] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Wed Feb 6 09:40:01 2019][1697175.777874] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:01 2019][1697175.782934] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.789200] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1697175.795813] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.802088] lquota_wb_fir-M S ffffa07b4eb4d140 0 127048 2 0x00000080 [Wed Feb 6 09:40:01 2019][1697175.809442] Call Trace: [Wed Feb 6 09:40:01 2019][1697175.812072] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1697175.817211] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:40:01 2019][1697175.823218] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:40:01 2019][1697175.829758] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Wed Feb 6 09:40:01 2019][1697175.836552] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Wed Feb 6 09:40:01 2019][1697175.843170] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1697175.848927] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Wed Feb 6 09:40:01 2019][1697175.855285] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:01 2019][1697175.860337] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.866602] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1697175.873218] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.879492] dist_txn-1 S ffffa07b4eb4c100 0 127049 2 0x00000080 [Wed Feb 6 09:40:01 2019][1697175.886855] Call Trace: [Wed Feb 6 09:40:01 2019][1697175.889485] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:40:01 2019][1697175.895600] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1697175.900789] [] distribute_txn_commit_thread+0x9ac/0xca0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1697175.908623] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1697175.914409] [] ? sub_trans_commit_cb+0x20/0x20 [ptlrpc] [Wed Feb 6 09:40:01 2019][1697175.921455] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:01 2019][1697175.926510] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.932784] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1697175.939404] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.945670] mdt00_003 S ffffa07b47b9e180 0 127063 2 0x00000080 [Wed Feb 6 09:40:01 2019][1697175.953027] Call Trace: [Wed Feb 6 09:40:01 2019][1697175.955657] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1697175.960834] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:01 2019][1697175.967711] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1697175.973495] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1697175.979983] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:01 2019][1697175.987556] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:01 2019][1697175.992609] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697175.998877] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1697176.005496] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697176.011763] mdt01_003 S ffffa07b47b9c100 0 127066 2 0x00000080 [Wed Feb 6 09:40:01 2019][1697176.019117] Call Trace: [Wed Feb 6 09:40:01 2019][1697176.021747] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1697176.026917] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:01 2019][1697176.033797] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1697176.039593] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1697176.046070] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:01 2019][1697176.053641] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:01 2019][1697176.058703] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697176.064981] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1697176.071599] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697176.077864] mdt03_003 S ffffa07b47b98000 0 127130 2 0x00000080 [Wed Feb 6 09:40:01 2019][1697176.085237] Call Trace: [Wed Feb 6 09:40:01 2019][1697176.087869] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1697176.093047] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:01 2019][1697176.099924] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1697176.105706] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1697176.112182] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:01 2019][1697176.119751] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:01 2019][1697176.124814] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697176.131088] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1697176.137710] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1697176.143983] mdt01_004 S ffffa07b47b99040 0 127131 2 0x00000080 [Wed Feb 6 09:40:01 2019][1697176.151339] Call Trace: [Wed Feb 6 09:40:01 2019][1697176.153966] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1697176.159155] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:01 2019][1697176.166032] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1697176.171833] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1697176.178309] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.185880] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1697176.190956] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.197225] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1697176.203845] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.210113] mdt03_004 S ffffa07b47b9a080 0 127132 2 0x00000080 [Wed Feb 6 09:40:02 2019][1697176.217474] Call Trace: [Wed Feb 6 09:40:02 2019][1697176.220104] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1697176.225296] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.232178] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1697176.237960] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.244436] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.252005] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1697176.257059] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.263334] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1697176.269955] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.276222] mdt02_003 S ffffa06b76380000 0 127133 2 0x00000080 [Wed Feb 6 09:40:02 2019][1697176.283586] Call Trace: [Wed Feb 6 09:40:02 2019][1697176.286214] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1697176.291381] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.298262] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1697176.304046] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.310519] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.318093] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1697176.323152] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.329428] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1697176.336049] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.342322] mdt00_004 S ffffa06b76386180 0 127134 2 0x00000080 [Wed Feb 6 09:40:02 2019][1697176.349694] Call Trace: [Wed Feb 6 09:40:02 2019][1697176.352324] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1697176.357492] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.364370] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1697176.370144] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.376621] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.384193] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1697176.389254] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.395520] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1697176.402140] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.408425] mdt02_004 S ffffa06b763830c0 0 127135 2 0x00000080 [Wed Feb 6 09:40:02 2019][1697176.415788] Call Trace: [Wed Feb 6 09:40:02 2019][1697176.418420] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1697176.423593] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.430472] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1697176.436249] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.442722] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.450295] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1697176.455356] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.461628] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1697176.468243] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.474509] mdt00_005 S ffffa08b764e8000 0 127136 2 0x00000080 [Wed Feb 6 09:40:02 2019][1697176.481865] Call Trace: [Wed Feb 6 09:40:02 2019][1697176.484495] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1697176.489671] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.496549] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1697176.502331] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.508805] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.516377] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1697176.521431] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.527698] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1697176.534334] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.540600] mdt_out02_002 S ffffa08b764e9040 0 127137 2 0x00000080 [Wed Feb 6 09:40:02 2019][1697176.547955] Call Trace: [Wed Feb 6 09:40:02 2019][1697176.550586] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1697176.555762] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.562642] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1697176.568426] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.574898] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.582470] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1697176.587538] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.593808] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1697176.600431] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.606703] mdt03_005 S ffffa08b764ea080 0 127195 2 0x00000080 [Wed Feb 6 09:40:02 2019][1697176.614057] Call Trace: [Wed Feb 6 09:40:02 2019][1697176.616687] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1697176.621864] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.628741] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1697176.634532] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.641001] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.648573] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1697176.653632] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.659900] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1697176.666522] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.672796] mdt03_006 S ffffa08b764eb0c0 0 127196 2 0x00000080 [Wed Feb 6 09:40:02 2019][1697176.680151] Call Trace: [Wed Feb 6 09:40:02 2019][1697176.682780] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1697176.687958] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.694835] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1697176.700613] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.707084] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.714655] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1697176.719710] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.725983] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1697176.732598] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.738872] mdt01_005 S ffffa08b764ec100 0 127197 2 0x00000080 [Wed Feb 6 09:40:02 2019][1697176.746235] Call Trace: [Wed Feb 6 09:40:02 2019][1697176.748865] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1697176.754056] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.760937] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1697176.766711] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.773186] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.780759] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1697176.785819] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.792085] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1697176.798698] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.804972] mdt02_005 S ffffa08b764ed140 0 127198 2 0x00000080 [Wed Feb 6 09:40:02 2019][1697176.812329] Call Trace: [Wed Feb 6 09:40:02 2019][1697176.814956] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1697176.820125] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.827006] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1697176.832787] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.839260] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.846835] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1697176.851896] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.858184] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1697176.864800] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.871076] mdt02_006 S ffffa08b764ee180 0 127199 2 0x00000080 [Wed Feb 6 09:40:02 2019][1697176.878439] Call Trace: [Wed Feb 6 09:40:02 2019][1697176.881068] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1697176.886244] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.893123] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1697176.898904] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.905395] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.912969] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1697176.918022] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.924288] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1697176.930915] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.937184] mdt_out00_002 S ffffa08b5d628000 0 127200 2 0x00000080 [Wed Feb 6 09:40:02 2019][1697176.944541] Call Trace: [Wed Feb 6 09:40:02 2019][1697176.947167] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1697176.952339] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.959217] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1697176.964998] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.971474] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697176.979046] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1697176.984105] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697176.990389] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1697176.997003] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697177.003278] ldlm_cn00_003 S ffffa08b5d62a080 0 127202 2 0x00000080 [Wed Feb 6 09:40:02 2019][1697177.010642] Call Trace: [Wed Feb 6 09:40:02 2019][1697177.013270] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1697177.018460] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697177.025336] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1697177.031133] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697177.037599] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697177.045174] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1697177.050234] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697177.056508] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1697177.063121] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697177.069387] ldlm_cn01_003 S ffffa08b5d62b0c0 0 127203 2 0x00000080 [Wed Feb 6 09:40:02 2019][1697177.076750] Call Trace: [Wed Feb 6 09:40:02 2019][1697177.079381] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1697177.084557] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697177.091436] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1697177.097221] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697177.103691] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697177.111265] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1697177.116328] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697177.122600] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1697177.129213] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1697177.135481] ldlm_cn02_003 S ffffa08b5d62c100 0 127204 2 0x00000080 [Wed Feb 6 09:40:02 2019][1697177.142835] Call Trace: [Wed Feb 6 09:40:02 2019][1697177.145464] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1697177.150633] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697177.157511] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1697177.163284] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697177.169759] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1697177.177333] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697177.182396] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.188669] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1697177.195304] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.201574] mdt01_006 S ffffa08b5d62d140 0 127205 2 0x00000080 [Wed Feb 6 09:40:03 2019][1697177.208938] Call Trace: [Wed Feb 6 09:40:03 2019][1697177.211566] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1697177.216734] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.223615] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1697177.229396] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.235869] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.243442] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697177.248497] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.254770] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1697177.261383] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.267659] mdt01_007 S ffffa08b5d62e180 0 127206 2 0x00000080 [Wed Feb 6 09:40:03 2019][1697177.275021] Call Trace: [Wed Feb 6 09:40:03 2019][1697177.277650] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1697177.282827] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.289706] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1697177.295490] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.301962] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.309537] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697177.314598] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.320871] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1697177.327484] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.333759] mdt01_008 S ffffa08b5f731040 0 127208 2 0x00000080 [Wed Feb 6 09:40:03 2019][1697177.341113] Call Trace: [Wed Feb 6 09:40:03 2019][1697177.343743] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1697177.348920] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.355799] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1697177.361574] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.368062] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.375636] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697177.380691] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.386963] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1697177.393576] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.399844] mdt00_006 S ffffa08b5f732080 0 127209 2 0x00000080 [Wed Feb 6 09:40:03 2019][1697177.407206] Call Trace: [Wed Feb 6 09:40:03 2019][1697177.409836] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1697177.415012] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.421893] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1697177.427674] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.434148] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.441734] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697177.446790] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.453057] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1697177.459669] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.465937] mdt01_009 S ffffa08b5f7330c0 0 127210 2 0x00000080 [Wed Feb 6 09:40:03 2019][1697177.473316] Call Trace: [Wed Feb 6 09:40:03 2019][1697177.475946] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1697177.481115] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.487994] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1697177.493777] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.500248] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.507822] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697177.512876] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.519149] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1697177.525763] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.532038] mdt_rdpg01_002 S ffffa08b5f734100 0 127211 2 0x00000080 [Wed Feb 6 09:40:03 2019][1697177.539393] Call Trace: [Wed Feb 6 09:40:03 2019][1697177.542020] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1697177.547190] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.554069] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1697177.559844] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.566315] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.573905] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697177.578960] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.585249] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1697177.591863] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.598131] mdt01_010 S ffffa08b5f735140 0 127212 2 0x00000080 [Wed Feb 6 09:40:03 2019][1697177.605494] Call Trace: [Wed Feb 6 09:40:03 2019][1697177.608125] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1697177.613301] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.620180] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1697177.625964] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.632440] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.640006] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697177.645060] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.651327] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1697177.657949] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.664224] mdt_rdpg00_002 S ffffa08b5f736180 0 127213 2 0x00000080 [Wed Feb 6 09:40:03 2019][1697177.671587] Call Trace: [Wed Feb 6 09:40:03 2019][1697177.674217] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1697177.679392] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.686271] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1697177.692055] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.698528] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.706100] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697177.711155] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.717429] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1697177.724047] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.730317] mdt00_007 S ffffa08b5f730000 0 127214 2 0x00000080 [Wed Feb 6 09:40:03 2019][1697177.737681] Call Trace: [Wed Feb 6 09:40:03 2019][1697177.740309] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1697177.745486] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.752366] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1697177.758148] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.764623] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.772195] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697177.777257] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.783531] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1697177.790153] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.796427] mdt_rdpg02_002 S ffffa06b7897c100 0 127215 2 0x00000080 [Wed Feb 6 09:40:03 2019][1697177.803789] Call Trace: [Wed Feb 6 09:40:03 2019][1697177.806419] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1697177.811590] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.818466] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1697177.824249] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.830722] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.838297] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697177.843356] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.849622] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1697177.856235] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.862502] mdt02_007 S ffffa06b78979040 0 127216 2 0x00000080 [Wed Feb 6 09:40:03 2019][1697177.869856] Call Trace: [Wed Feb 6 09:40:03 2019][1697177.872487] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1697177.877679] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.884560] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1697177.890343] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.896818] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.904390] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697177.909449] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.915716] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1697177.922336] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.928617] ldlm_cb02_002 S ffffa06b7897a080 0 127217 2 0x00000080 [Wed Feb 6 09:40:03 2019][1697177.935977] Call Trace: [Wed Feb 6 09:40:03 2019][1697177.938604] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1697177.943773] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.950666] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1697177.956445] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.962916] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697177.970490] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697177.975544] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.981818] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1697177.988456] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697177.994722] mdt_io02_003 S ffffa06b7897d140 0 127218 2 0x00000080 [Wed Feb 6 09:40:03 2019][1697178.002078] Call Trace: [Wed Feb 6 09:40:03 2019][1697178.004706] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1697178.009876] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697178.016754] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1697178.022552] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697178.029028] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697178.036601] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697178.041662] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697178.047937] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1697178.054558] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697178.060847] mdt_io01_003 S ffffa06b78978000 0 127219 2 0x00000080 [Wed Feb 6 09:40:03 2019][1697178.068204] Call Trace: [Wed Feb 6 09:40:03 2019][1697178.070833] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1697178.076000] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697178.082881] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1697178.088666] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697178.095137] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697178.102709] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697178.107763] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697178.114037] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1697178.120652] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697178.126926] mdt_io00_003 S ffffa06b7897e180 0 127220 2 0x00000080 [Wed Feb 6 09:40:03 2019][1697178.134289] Call Trace: [Wed Feb 6 09:40:03 2019][1697178.136918] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1697178.142095] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697178.148975] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1697178.154758] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697178.161230] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1697178.168804] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1697178.173866] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1697178.180140] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697178.186759] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.193026] mdt_io01_004 S ffffa07b74621040 0 127221 2 0x00000080 [Wed Feb 6 09:40:04 2019][1697178.200380] Call Trace: [Wed Feb 6 09:40:04 2019][1697178.203012] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1697178.208189] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.215083] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1697178.220859] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.227333] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.234919] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1697178.239976] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.246249] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697178.252862] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.259128] mdt_io01_005 S ffffa07b74624100 0 127222 2 0x00000080 [Wed Feb 6 09:40:04 2019][1697178.266484] Call Trace: [Wed Feb 6 09:40:04 2019][1697178.269113] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1697178.274290] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.281167] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1697178.286942] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.293417] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.300989] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1697178.306040] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.312307] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697178.318921] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.325195] ldlm_bl_03 S ffffa07b746230c0 0 127223 2 0x00000080 [Wed Feb 6 09:40:04 2019][1697178.332560] Call Trace: [Wed Feb 6 09:40:04 2019][1697178.335190] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:04 2019][1697178.341057] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:04 2019][1697178.348457] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1697178.353627] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.360677] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1697178.366461] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.374044] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1697178.379102] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.385369] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697178.391991] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.398263] mdt_io02_004 S ffffa07b74620000 0 127224 2 0x00000080 [Wed Feb 6 09:40:04 2019][1697178.405621] Call Trace: [Wed Feb 6 09:40:04 2019][1697178.408250] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1697178.413427] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.420305] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1697178.426089] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.432562] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.440133] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1697178.445186] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.451454] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697178.458072] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.464341] mdt_io00_004 S ffffa07b74625140 0 127225 2 0x00000080 [Wed Feb 6 09:40:04 2019][1697178.471704] Call Trace: [Wed Feb 6 09:40:04 2019][1697178.474334] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1697178.479510] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.486389] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1697178.492172] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.498644] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.506218] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1697178.511280] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.517546] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697178.524167] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.530446] mdt_io02_005 S ffffa07b74622080 0 127226 2 0x00000080 [Wed Feb 6 09:40:04 2019][1697178.537806] Call Trace: [Wed Feb 6 09:40:04 2019][1697178.540437] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1697178.545610] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.552493] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1697178.558272] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.564747] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.572319] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1697178.577372] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.583649] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697178.590270] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.596543] mdt_io01_006 S ffffa08b76cc0000 0 127227 2 0x00000080 [Wed Feb 6 09:40:04 2019][1697178.603898] Call Trace: [Wed Feb 6 09:40:04 2019][1697178.606528] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1697178.611704] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.618582] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1697178.624358] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.630832] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.638404] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1697178.643466] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.649739] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697178.656353] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.662628] mdt_io01_007 S ffffa08b76cc1040 0 127228 2 0x00000080 [Wed Feb 6 09:40:04 2019][1697178.669983] Call Trace: [Wed Feb 6 09:40:04 2019][1697178.672611] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1697178.677778] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.684660] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1697178.690447] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.696925] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.704497] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1697178.709559] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.715832] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697178.722448] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.728720] mdt_rdpg01_003 S ffffa08b76cc2080 0 127229 2 0x00000080 [Wed Feb 6 09:40:04 2019][1697178.736075] Call Trace: [Wed Feb 6 09:40:04 2019][1697178.738705] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1697178.743880] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.750760] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1697178.756536] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.763009] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.770580] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1697178.775633] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.781899] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697178.788512] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.794786] mdt_io00_005 S ffffa08b76cc30c0 0 127230 2 0x00000080 [Wed Feb 6 09:40:04 2019][1697178.802141] Call Trace: [Wed Feb 6 09:40:04 2019][1697178.804770] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1697178.809940] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.816821] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1697178.822601] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.829076] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.836648] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1697178.841707] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.847977] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697178.854596] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.860883] mdt_io00_006 S ffffa08b76cc4100 0 127231 2 0x00000080 [Wed Feb 6 09:40:04 2019][1697178.868244] Call Trace: [Wed Feb 6 09:40:04 2019][1697178.870873] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1697178.876042] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.882920] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1697178.888703] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.895178] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.902749] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1697178.907802] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.914070] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697178.920703] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.926973] ldlm_cn00_004 S ffffa08b76cc5140 0 127232 2 0x00000080 [Wed Feb 6 09:40:04 2019][1697178.934328] Call Trace: [Wed Feb 6 09:40:04 2019][1697178.936956] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1697178.942127] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.949004] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1697178.954787] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.961261] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697178.968834] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1697178.973896] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.980169] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697178.986791] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697178.993065] mdt_io02_006 S ffffa08b76cc6180 0 127233 2 0x00000080 [Wed Feb 6 09:40:04 2019][1697179.000421] Call Trace: [Wed Feb 6 09:40:04 2019][1697179.003050] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1697179.008218] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697179.015096] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1697179.020879] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697179.027355] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697179.034927] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1697179.039988] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697179.046262] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697179.052876] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697179.059151] ldlm_cb02_003 S ffffa08b78068000 0 127234 2 0x00000080 [Wed Feb 6 09:40:04 2019][1697179.066514] Call Trace: [Wed Feb 6 09:40:04 2019][1697179.069143] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1697179.074321] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697179.081199] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1697179.086987] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697179.093460] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697179.101034] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1697179.106089] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697179.112355] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697179.118978] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697179.125253] mdt_io01_008 S ffffa08b78069040 0 127235 2 0x00000080 [Wed Feb 6 09:40:04 2019][1697179.132614] Call Trace: [Wed Feb 6 09:40:04 2019][1697179.135248] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1697179.140424] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697179.147300] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1697179.153084] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697179.159558] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1697179.167129] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1697179.172190] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1697179.178468] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1697179.185089] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.191363] ldlm_cn01_004 S ffffa08b7806a080 0 127236 2 0x00000080 [Wed Feb 6 09:40:05 2019][1697179.198724] Call Trace: [Wed Feb 6 09:40:05 2019][1697179.201354] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1697179.206524] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.213420] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1697179.219201] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.225679] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.233250] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1697179.238311] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.244585] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1697179.251206] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.257481] mdt_out02_003 S ffffa08b7806b0c0 0 127237 2 0x00000080 [Wed Feb 6 09:40:05 2019][1697179.264836] Call Trace: [Wed Feb 6 09:40:05 2019][1697179.267465] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1697179.272643] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.279521] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1697179.285303] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.291779] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.299351] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1697179.304413] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.310688] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1697179.317307] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.323583] ldlm_bl_04 S ffffa08b7806c100 0 127238 2 0x00000080 [Wed Feb 6 09:40:05 2019][1697179.330944] Call Trace: [Wed Feb 6 09:40:05 2019][1697179.333577] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:05 2019][1697179.339444] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:05 2019][1697179.346844] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1697179.352021] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.359074] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1697179.364855] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.372428] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1697179.377490] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.383766] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1697179.390386] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.396660] ldlm_cn02_004 S ffffa08b7806d140 0 127239 2 0x00000080 [Wed Feb 6 09:40:05 2019][1697179.404014] Call Trace: [Wed Feb 6 09:40:05 2019][1697179.406644] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1697179.411824] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.418701] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1697179.424487] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.430973] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.438546] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1697179.443599] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.449875] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1697179.456496] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.462772] mdt00_008 S ffffa08b7806e180 0 127247 2 0x00000080 [Wed Feb 6 09:40:05 2019][1697179.470143] Call Trace: [Wed Feb 6 09:40:05 2019][1697179.47277279.477950] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.484831] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1697179.490616] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.497106] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.504685] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1697179.509746] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.516023] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1697179.522641] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.528918] mdt02_008 S ffffa08b70618000 0 127248 2 0x00000080 [Wed Feb 6 09:40:05 2019][1697179.536279] Call Trace: [Wed Feb 6 09:40:05 2019][1697179.538911] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1697179.544106] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.550993] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1697179.556774] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.563250] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.570821] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1697179.575881] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.582157] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1697179.588777] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.595052] mdt_rdpg02_003 S ffffa08b70619040 0 127249 2 0x00000080 [Wed Feb 6 09:40:05 2019][1697179.602415] Call Trace: [Wed Feb 6 09:40:05 2019][1697179.605046] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1697179.610224] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.617102] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1697179.622883] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.629360] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.636931] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1697179.641992] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.648266] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1697179.654889] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.661161] mdt00_009 S ffffa08b7061a080 0 127250 2 0x00000080 [Wed Feb 6 09:40:05 2019][1697179.668526] Call Trace: [Wed Feb 6 09:40:05 2019][1697179.671156] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1697179.676330] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.683212] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1697179.688993] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.695469] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.703038] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1697179.708092] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.714369] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1697179.720991] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.727264] mdt_io00_007 S ffffa08b7061b0c0 0 127251 2 0x00000080 [Wed Feb 6 09:40:05 2019][1697179.734628] Call Trace: [Wed Feb 6 09:40:05 2019][1697179.737255] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1697179.742428] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.749304] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1697179.755085] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.761559] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.769134] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1697179.774194] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.780469] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1697179.787092] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.793365] ldlm_cn01_005 S ffffa08b7061c100 0 127252 2 0x00000080 [Wed Feb 6 09:40:05 2019][1697179.800728] Call Trace: [Wed Feb 6 09:40:05 2019][1697179.803357] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1697179.808526] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.815406] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1697179.821189] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.827661] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.835236] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1697179.840297] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.846572] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1697179.853192] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.859467] ldlm_cn00_005 S ffffa08b7061d140 0 127253 2 0x00000080 [Wed Feb 6 09:40:05 2019][1697179.866856] Call Trace: [Wed Feb 6 09:40:05 2019][1697179.869487] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1697179.874663] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.881542] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1697179.887324] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.893799] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.901371] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1697179.906435] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.912708] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1697179.919329] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.925620] mdt03_007 S ffffa08b7061e180 0 127254 2 0x00000080 [Wed Feb 6 09:40:05 2019][1697179.932992] Call Trace: [Wed Feb 6 09:40:05 2019][1697179.935624] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1697179.940814] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.947697] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1697179.953483] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.959976] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697179.967552] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1697179.972612] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.978886] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1697179.985522] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697179.991792] mdt_io01_009 S ffffa08b6ce20000 0 127261 2 0x00000080 [Wed Feb 6 09:40:05 2019][1697179.999155] Call Trace: [Wed Feb 6 09:40:05 2019][1697180.001784] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1697180.006976] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697180.013858] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1697180.019639] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697180.026114] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697180.033688] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1697180.038749] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697180.045022] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1697180.051643] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697180.057919] mdt_io01_010 S ffffa08b6ce21040 0 127262 2 0x00000080 [Wed Feb 6 09:40:05 2019][1697180.065281] Call Trace: [Wed Feb 6 09:40:05 2019][1697180.067912] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1697180.073088] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697180.079970] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1697180.085753] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697180.092225] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697180.099812] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1697180.104868] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697180.111142] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1697180.117778] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697180.124047] mdt_io00_008 S ffffa08b6ce22080 0 127263 2 0x00000080 [Wed Feb 6 09:40:05 2019][1697180.131417] Call Trace: [Wed Feb 6 09:40:05 2019][1697180.134064] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1697180.139241] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697180.146122] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1697180.151904] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697180.158379] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1697180.165950] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1697180.171028] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697180.177295] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1697180.183931] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1697180.190201] mdt_io02_007 S ffffa08b6ce230c0 0 127264 2 0x00000080 [Wed Feb 6 09:40:06 2019][1697180.197563] Call Trace: [Wed Feb 6 09:40:06 2019][1697180.200195] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1697180.205370] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.212265] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1697180.218049] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.224524] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.232095] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1697180.237172] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.243440] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1697180.250063] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.256337] mdt_io02_008 S ffffa08b6ce24100 0 127265 2 0x00000080 [Wed Feb 6 09:40:06 2019][1697180.263698] Call Trace: [Wed Feb 6 09:40:06 2019][1697180.266328] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1697180.271506] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.278387] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1697180.284166] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.290642] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.298215] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1697180.303276] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.309550] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1697180.316173] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.322446] mdt_io00_009 S ffffa08b6ce25140 0 127266 2 0x00000080 [Wed Feb 6 09:40:06 2019][1697180.329810] Call Trace: [Wed Feb 6 09:40:06 2019][1697180.332440] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1697180.337616] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.344494] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1697180.350277] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.356751] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.364324] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1697180.369386] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.375661] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1697180.382281] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.388557] mdt02_009 S ffffa08b6ce26180 0 127273 2 0x00000080 [Wed Feb 6 09:40:06 2019][1697180.395920] Call Trace: [Wed Feb 6 09:40:06 2019][1697180.398550] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1697180.403727] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.410605] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1697180.416389] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.422862] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.430437] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1697180.435495] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.441771] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1697180.448393] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.454665] mdt02_010 S ffffa08b786f0000 0 127297 2 0x00000080 [Wed Feb 6 09:40:06 2019][1697180.462022] Call Trace: [Wed Feb 6 09:40:06 2019][1697180.464652] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1697180.469827] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.476707] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1697180.482495] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.488972] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.496544] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1697180.501606] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.507883] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1697180.514516] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.520787] mdt03_008 S ffffa08b786f2080 0 127317 2 0x00000080 [Wed Feb 6 09:40:06 2019][1697180.528156] Call Trace: [Wed Feb 6 09:40:06 2019][1697180.530788] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1697180.535966] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.542842] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1697180.548626] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.555101] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.562673] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1697180.567733] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.574008] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1697180.580631] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.586905] mdt_out02_004 S ffffa08b786f30c0 0 127318 2 0x00000080 [Wed Feb 6 09:40:06 2019][1697180.594267] Call Trace: [Wed Feb 6 09:40:06 2019][1697180.596896] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1697180.602073] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.608955] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1697180.614735] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.621210] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.628783] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1697180.633845] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.640121] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1697180.646755] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.653024] mdt_io02_009 S ffffa08b786f4100 0 127319 2 0x00000080 [Wed Feb 6 09:40:06 2019][1697180.660386] Call Trace: [Wed Feb 6 09:40:06 2019][1697180.663016] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1697180.668192] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.675071] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1697180.680853] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.687328] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.694902] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1697180.699964] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.706238] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1697180.712859] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.719134] mdt03_009 S ffffa08b786f6180 0 127327 2 0x00000080 [Wed Feb 6 09:40:06 2019][1697180.726496] Call Trace: [Wed Feb 6 09:40:06 2019][1697180.729128] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1697180.734304] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.741183] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1697180.746964] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.753439] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.761013] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1697180.766074] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.772350] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1697180.778969] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.785245] mdt_rdpg03_002 S ffffa08b786f1040 0 127329 2 0x00000080 [Wed Feb 6 09:40:06 2019][1697180.792607] Call Trace: [Wed Feb 6 09:40:06 2019][1697180.795237] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1697180.800413] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.807293] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1697180.813075] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.819550] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.827122] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1697180.832183] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.838462] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1697180.845078] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.851354] mdt_io01_011 S ffffa08b61120000 0 127336 2 0x00000080 [Wed Feb 6 09:40:06 2019][1697180.858717] Call Trace: [Wed Feb 6 09:40:06 2019][1697180.861345] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1697180.866516] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.873394] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1697180.879177] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.885652] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.893242] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1697180.898302] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.904578] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1697180.911198] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.917475] mdt_io02_010 S ffffa08b61122080 0 127350 2 0x00000080 [Wed Feb 6 09:40:06 2019][1697180.924837] Call Trace: [Wed Feb 6 09:40:06 2019][1697180.927467] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1697180.932644] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.939521] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1697180.945304] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.951777] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697180.959353] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1697180.964412] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.970689] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1697180.977313] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697180.983584] mdt_io01_012 S ffffa08b611230c0 0 127363 2 0x00000080 [Wed Feb 6 09:40:06 2019][1697180.990955] Call Trace: [Wed Feb 6 09:40:06 2019][1697180.993588] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1697180.998781] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697181.005661] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1697181.011443] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697181.017916] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697181.025488] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1697181.030549] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697181.036825] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1697181.043445] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697181.049720] mdt_io00_010 S ffffa08b61124100 0 127364 2 0x00000080 [Wed Feb 6 09:40:06 2019][1697181.057082] Call Trace: [Wed Feb 6 09:40:06 2019][1697181.059712] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1697181.064891] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697181.071768] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1697181.077549] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697181.084026] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697181.091597] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1697181.096659] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697181.102934] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1697181.109554] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697181.115831] mdt_io00_011 S ffffa08b61125140 0 127365 2 0x00000080 [Wed Feb 6 09:40:06 2019][1697181.123191] Call Trace: [Wed Feb 6 09:40:06 2019][1697181.125821] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1697181.130991] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697181.137870] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1697181.143651] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697181.150128] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1697181.157699] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1697181.162761] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1697181.169035] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697181.175655] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.181931] mdt03_010 S ffffa08b61126180 0 127390 2 0x00000080 [Wed Feb 6 09:40:07 2019][1697181.189294] Call Trace: [Wed Feb 6 09:40:07 2019][1697181.191922] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1697181.197100] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.203978] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1697181.209761] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.216237] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.223809] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1697181.228870] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.235146] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697181.241767] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.248041] mdt03_011 S ffffa08b61cbc100 0 127411 2 0x00000080 [Wed Feb 6 09:40:07 2019][1697181.255405] Call Trace: [Wed Feb 6 09:40:07 2019][1697181.258034] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1697181.263211] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.270090] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1697181.275871] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.282346] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.289919] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1697181.294981] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.301256] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697181.307876] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.314152] ldlm_cn03_003 S ffffa08b61cbd140 0 127455 2 0x00000080 [Wed Feb 6 09:40:07 2019][1697181.321514] Call Trace: [Wed Feb 6 09:40:07 2019][1697181.324142] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1697181.329321] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.336200] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1697181.341985] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.348459] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.356036] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1697181.361089] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.367367] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697181.373987] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.380261] mdt03_012 S ffffa08b61cbe180 0 127456 2 0x00000080 [Wed Feb 6 09:40:07 2019][1697181.387623] Call Trace: [Wed Feb 6 09:40:07 2019][1697181.390253] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1697181.395430] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.402310] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1697181.408093] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.414568] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.422139] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1697181.427201] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.433475] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697181.440096] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.446373] mdt02_011 S ffffa07b3ca68000 0 127501 2 0x00000080 [Wed Feb 6 09:40:07 2019][1697181.453734] Call Trace: [Wed Feb 6 09:40:07 2019][1697181.456362] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1697181.461532] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.468411] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1697181.474194] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.480669] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.488240] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1697181.493301] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.499577] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697181.506198] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.512473] mdt_io02_011 S ffffa07b3ca6b0c0 0 127515 2 0x00000080 [Wed Feb 6 09:40:07 2019][1697181.519836] Call Trace: [Wed Feb 6 09:40:07 2019][1697181.522469] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1697181.527646] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.534523] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1697181.540304] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.546779] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.554350] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1697181.559413] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.565686] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697181.572309] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.578583] mdt_io02_012 S ffffa07b3ca6e180 0 127535 2 0x00000080 [Wed Feb 6 09:40:07 2019][1697181.585960] Call Trace: [Wed Feb 6 09:40:07 2019][1697181.588592] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1697181.593771] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.600648] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1697181.606433] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.612906] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.620478] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1697181.625539] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.631815] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697181.638437] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.644711] mdt_io00_012 S ffffa05c2993a080 0 127656 2 0x00000080 [Wed Feb 6 09:40:07 2019][1697181.652073] Call Trace: [Wed Feb 6 09:40:07 2019][1697181.654702] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1697181.659871] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.666750] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1697181.672532] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.679007] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.686580] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1697181.691641] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.697915] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697181.704529] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.710806] mdt_io01_013 S ffffa08b76cca080 0 127657 2 0x00000080 [Wed Feb 6 09:40:07 2019][1697181.718167] Call Trace: [Wed Feb 6 09:40:07 2019][1697181.720797] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1697181.725973] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.732854] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1697181.738635] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.745108] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.752682] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1697181.757743] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.764017] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697181.770639] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.776913] mdt_io02_013 S ffffa08b76cce180 0 127669 2 0x00000080 [Wed Feb 6 09:40:07 2019][1697181.784275] Call Trace: [Wed Feb 6 09:40:07 2019][1697181.786906] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1697181.792084] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.798963] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1697181.804746] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.811220] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.818793] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1697181.823853] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.830128] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697181.836748] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.843015] mdt_io02_014 S ffffa06b78806180 0 127690 2 0x00000080 [Wed Feb 6 09:40:07 2019][1697181.850394] Call Trace: [Wed Feb 6 09:40:07 2019][1697181.853024] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1697181.858202] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.865081] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1697181.870864] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.877336] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.884910] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1697181.889973] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.896247] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697181.902868] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.909143] mdt_io01_014 S ffffa06b78800000 0 127691 2 0x00000080 [Wed Feb 6 09:40:07 2019][1697181.916505] Call Trace: [Wed Feb 6 09:40:07 2019][1697181.919136] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1697181.924313] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.931190] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1697181.936976] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.943449] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.951021] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1697181.956082] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.962357] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697181.968978] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697181.975254] mdt_io02_015 S ffffa06b788030c0 0 127692 2 0x00000080 [Wed Feb 6 09:40:07 2019][1697181.982630] Call Trace: [Wed Feb 6 09:40:07 2019][1697181.985263] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1697181.990441] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697181.997319] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1697182.003100] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697182.009575] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697182.017148] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1697182.022209] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697182.028484] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697182.035106] [] ? ret_from_fork+0x48/0x56 [Wed Feb 6 09:40:07 2019][1697182.040859] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697182.047126] mdt_io02_016 S ffffa06b78802080 0 127693 2 0x00000080 [Wed Feb 6 09:40:07 2019][1697182.054488] Call Trace: [Wed Feb 6 09:40:07 2019][1697182.057118] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1697182.062290] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697182.069168] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1697182.074949] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697182.081422] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697182.088995] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1697182.094057] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697182.100332] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697182.106955] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697182.113229] mdt_io01_015 S ffffa06b78805140 0 127694 2 0x00000080 [Wed Feb 6 09:40:07 2019][1697182.120591] Call Trace: [Wed Feb 6 09:40:07 2019][1697182.123221] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1697182.128397] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697182.135276] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1697182.141059] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697182.147535] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1697182.155106] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1697182.160166] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1697182.166432] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1697182.173047] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.179322] mdt_io01_016 S ffffa06b78801040 0 127695 2 0x00000080 [Wed Feb 6 09:40:08 2019][1697182.186682] Call Trace: [Wed Feb 6 09:40:08 2019][1697182.189312] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1697182.194482] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.201359] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1697182.207143] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.213618] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.221190] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1697182.226250] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.232525] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1697182.239147] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.245421] mdt_io01_017 S ffffa07b3babe180 0 127696 2 0x00000080 [Wed Feb 6 09:40:08 2019][1697182.252775] Call Trace: [Wed Feb 6 09:40:08 2019][1697182.255406] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1697182.260582] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.267482] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1697182.273263] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.279736] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.287309] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1697182.292370] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.298645] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1697182.305267] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.311540] mdt_io01_018 S ffffa07b3babd140 0 127697 2 0x00000080 [Wed Feb 6 09:40:08 2019][1697182.318895] Call Trace: [Wed Feb 6 09:40:08 2019][1697182.321524] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1697182.326694] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.333573] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1697182.339356] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.345830] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.353402] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1697182.358464] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.364739] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1697182.371361] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.377633] mdt_io01_019 S ffffa07b3babc100 0 127698 2 0x00000080 [Wed Feb 6 09:40:08 2019][1697182.384987] Call Trace: [Wed Feb 6 09:40:08 2019][1697182.387617] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1697182.392786] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.399665] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1697182.405446] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.411922] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.419497] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1697182.424555] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.430823] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1697182.437444] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.443717] mdt_io02_017 S ffffa07b3bab9040 0 127699 2 0x00000080 [Wed Feb 6 09:40:08 2019][1697182.451074] Call Trace: [Wed Feb 6 09:40:08 2019][1697182.453701] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1697182.458870] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.465748] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1697182.471524] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.478000] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.485569] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1697182.490624] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.496898] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1697182.503521] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.509793] mdt_io00_013 S ffffa07b3babb0c0 0 127700 2 0x00000080 [Wed Feb 6 09:40:08 2019][1697182.517156] Call Trace: [Wed Feb 6 09:40:08 2019][1697182.519786] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1697182.524967] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.531850] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1697182.537636] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.544107] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.551680] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1697182.556742] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.563015] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1697182.569637] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.575912] mdt_io00_014 S ffffa07b3bab8000 0 127701 2 0x00000080 [Wed Feb 6 09:40:08 2019][1697182.583276] Call Trace: [Wed Feb 6 09:40:08 2019][1697182.585921] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1697182.591099] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.597979] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1697182.603761] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.610234] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.617808] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1697182.622869] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.629143] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1697182.635766] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.642041] mdt_io00_015 S ffffa07b3baba080 0 127702 2 0x00000080 [Wed Feb 6 09:40:08 2019][1697182.649401] Call Trace: [Wed Feb 6 09:40:08 2019][1697182.652031] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1697182.657202] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.664080] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1697182.669863] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.676337] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.683908] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1697182.688970] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.695246] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1697182.701868] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.708142] mdt_io00_016 S ffffa06c299530c0 0 127709 2 0x00000080 [Wed Feb 6 09:40:08 2019][1697182.715504] Call Trace: [Wed Feb 6 09:40:08 2019][1697182.718134] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1697182.723312] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.730188] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1697182.735964] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.742439] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.750011] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1697182.755073] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.761348] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1697182.767969] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.774245] mdt_io01_020 S ffffa08b76510000 0 127710 2 0x00000080 [Wed Feb 6 09:40:08 2019][1697182.781607] Call Trace: [Wed Feb 6 09:40:08 2019][1697182.784237] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1697182.789413] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.796291] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1697182.802075] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.808549] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.816123] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1697182.821184] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.827458] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1697182.834077] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.840345] ldlm_cn01_006 S ffffa07b775c9040 0 127800 2 0x00000080 [Wed Feb 6 09:40:08 2019][1697182.847707] Call Trace: [Wed Feb 6 09:40:08 2019][1697182.850339] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1697182.855517] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.862408] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1697182.868184] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.874658] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.882233] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1697182.887292] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.893568] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1697182.900189] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.906462] ldlm_cn02_005 S ffffa07b775ce180 0 127801 2 0x00000080 [Wed Feb 6 09:40:08 2019][1697182.913819] Call Trace: [Wed Feb 6 09:40:08 2019][1697182.916448] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1697182.921625] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.928504] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1697182.934286] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.940760] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.948332] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1697182.953394] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.959670] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1697182.966292] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697182.972565] ldlm_cn02_006 S ffffa07b775ca080 0 127802 2 0x00000080 [Wed Feb 6 09:40:08 2019][1697182.979927] Call Trace: [Wed Feb 6 09:40:08 2019][1697182.982557] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1697182.987734] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697182.994628] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1697183.000406] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697183.006881] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697183.014453] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1697183.019512] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697183.025789] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1697183.032411] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697183.038684] ldlm_cn01_007 S ffffa07b775cc100 0 127803 2 0x00000080 [Wed Feb 6 09:40:08 2019][1697183.046047] Call Trace: [Wed Feb 6 09:40:08 2019][1697183.048676] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1697183.053855] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697183.060733] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1697183.066517] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697183.072988] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697183.080562] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1697183.085624] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697183.091899] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1697183.098518] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697183.104784] ldlm_cn00_006 S ffffa07b775cd140 0 127804 2 0x00000080 [Wed Feb 6 09:40:08 2019][1697183.112140] Call Trace: [Wed Feb 6 09:40:08 2019][1697183.114770] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1697183.119949] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697183.126826] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1697183.132608] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697183.139081] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1697183.146656] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1697183.151716] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697183.157990] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1697183.164612] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1697183.170887] mdt00_010 S ffffa08b765130c0 0 127970 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697183.178250] Call Trace: [Wed Feb 6 09:40:09 2019][1697183.180881] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1697183.186066] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.192944] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1697183.198739] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.205218] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.212793] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1697183.217854] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.224130] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1697183.230750] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.237024] ldlm_cn00_007 S ffffa08b76516180 0 128033 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697183.244385] Call Trace: [Wed Feb 6 09:40:09 2019][1697183.247016] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1697183.252202] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.259080] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:40:09 2019][1697183.266049] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1697183.271842] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.278319] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.285897] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1697183.290957] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.297236] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1697183.303856] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.310128] ldlm_cn00_008 S ffffa08b76511040 0 128040 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697183.317499] Call Trace: [Wed Feb 6 09:40:09 2019][1697183.320130] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1697183.325313] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.332194] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1697183.337981] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.344458] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.352033] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1697183.357092] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.363370] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1697183.369989] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.376265] mdt00_011 S ffffa06b7ebe9040 0 128084 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697183.383628] Call Trace: [Wed Feb 6 09:40:09 2019][1697183.386256] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1697183.391441] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.398320] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1697183.404110] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.410596] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.418169] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1697183.423230] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.429508] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1697183.436127] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.442400] mdt00_012 S ffffa06b7ebec100 0 128085 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697183.449763] Call Trace: [Wed Feb 6 09:40:09 2019][1697183.452393] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1697183.457575] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.464457] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1697183.470244] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.476727] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.484305] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1697183.489367] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.495644] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1697183.502261] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.508536] mdt00_013 S ffffa08b76512080 0 128092 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697183.515907] Call Trace: [Wed Feb 6 09:40:09 2019][1697183.518538] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1697183.523721] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.530628] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1697183.536418] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.542902] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.550478] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1697183.555543] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.561813] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1697183.568435] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.574732] mdt_rdpg00_003 S ffffa07b7a275140 0 128100 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697183.582104] Call Trace: [Wed Feb 6 09:40:09 2019][1697183.584734] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1697183.589938] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.596840] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1697183.602621] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.609097] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.616673] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1697183.621733] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.628010] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1697183.634629] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.640904] mdt02_012 S ffffa07b75ef8000 0 128267 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697183.648268] Call Trace: [Wed Feb 6 09:40:09 2019][1697183.650898] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1697183.656079] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.662963] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1697183.668748] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.675225] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.682800] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1697183.687860] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.694139] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1697183.700758] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.707031] mdt01_011 S ffffa07b74acc100 0 128319 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697183.714412] Call Trace: [Wed Feb 6 09:40:09 2019][1697183.717042] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1697183.722222] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.729097] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1697183.734885] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.741374] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.748945] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1697183.754005] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.760281] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1697183.766902] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.773178] mdt_rdpg01_004 S ffffa06b7b9630c0 0 128484 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697183.780538] Call Trace: [Wed Feb 6 09:40:09 2019][1697183.783170] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1697183.788352] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.795236] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1697183.801024] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.807513] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.815088] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1697183.820150] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.826429] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1697183.833072] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.839346] mdt_rdpg01_005 S ffffa06b7b965140 0 128517 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697183.846718] Call Trace: [Wed Feb 6 09:40:09 2019][1697183.849349] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1697183.854528] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.861405] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1697183.867193] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.873676] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.881252] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1697183.886314] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.892589] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1697183.899210] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.905487] mdt_rdpg02_004 S ffffa07c29684100 0 128524 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697183.912847] Call Trace: [Wed Feb 6 09:40:09 2019][1697183.915478] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1697183.920666] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.927550] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1697183.933336] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.939815] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.947410] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1697183.952493] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.958772] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1697183.965389] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697183.971685] mdt01_012 S ffffa07c296830c0 0 128538 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697183.979053] Call Trace: [Wed Feb 6 09:40:09 2019][1697183.981684] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1697183.986864] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697183.993771] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1697183.999560] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697184.006043] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697184.013619] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1697184.018681] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697184.024957] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1697184.031576] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697184.037850] ldlm_cb00_002 S ffffa07c29685140 0 128539 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697184.045214] Call Trace: [Wed Feb 6 09:40:09 2019][1697184.047843] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1697184.053027] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697184.059908] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1697184.065695] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697184.072197] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697184.079774] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1697184.084834] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697184.091114] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1697184.097761] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697184.104032] mdt_rdpg00_004 S ffffa07c29686180 0 128546 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697184.111409] Call Trace: [Wed Feb 6 09:40:09 2019][1697184.114041] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1697184.119221] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697184.126096] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1697184.131881] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697184.138386] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1697184.145962] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1697184.151023] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697184.157298] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1697184.163919] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1697184.170192] ldlm_cn00_009 S ffffa07b2cf8d140 0 128662 2 0x00000080 [Wed Feb 6 09:40:09 2019][1697184.177555] Call Trace: [Wed Feb 6 09:40:10 2019][1697184.180185] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1697184.185367] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.192273] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1697184.198055] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.204535] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.212107] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1697184.217167] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.223447] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1697184.230065] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.236339] mdt_rdpg01_006 S ffffa08b76cc8000 0 128924 2 0x00000080 [Wed Feb 6 09:40:10 2019][1697184.243727] Call Trace: [Wed Feb 6 09:40:10 2019][1697184.246358] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1697184.251540] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.258421] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1697184.264226] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.270712] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.278286] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1697184.283345] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.289623] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1697184.296244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.302517] ldlm_cn02_007 S ffffa08b76ccc100 0 129080 2 0x00000080 [Wed Feb 6 09:40:10 2019][1697184.309881] Call Trace: [Wed Feb 6 09:40:10 2019][1697184.312508] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1697184.317683] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.324567] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1697184.330353] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.336833] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.344404] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1697184.349467] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.355745] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1697184.362361] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.368636] ldlm_cn01_008 S ffffa04ca9b64100 0 129107 2 0x00000080 [Wed Feb 6 09:40:10 2019][1697184.376017] Call Trace: [Wed Feb 6 09:40:10 2019][1697184.378647] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1697184.383828] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.390734] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1697184.396517] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.402990] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.410566] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1697184.415628] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.421904] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1697184.428524] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.434821] mdt01_013 S ffffa08b7a2b0000 0 129227 2 0x00000080 [Wed Feb 6 09:40:10 2019][1697184.442177] Call Trace: [Wed Feb 6 09:40:10 2019][1697184.444812] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1697184.449988] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.456863] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1697184.462651] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.469126] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.476702] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1697184.481764] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.488041] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1697184.494683] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.500953] mdt01_014 S ffffa08b7a2b1040 0 129228 2 0x00000080 [Wed Feb 6 09:40:10 2019][1697184.508321] Call Trace: [Wed Feb 6 09:40:10 2019][1697184.510954] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1697184.516135] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.523017] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1697184.528804] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.535288] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.542865] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1697184.547925] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.554203] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1697184.560824] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.567097] mdt01_015 S ffffa08b7a2b2080 0 129229 2 0x00000080 [Wed Feb 6 09:40:10 2019][1697184.574466] Call Trace: [Wed Feb 6 09:40:10 2019][1697184.577100] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1697184.582260] [] ? mdt_pack_acl2body+0x1c2/0x9f0 [mdt] [Wed Feb 6 09:40:10 2019][1697184.589077] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:40:10 2019][1697184.594493] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Wed Feb 6 09:40:10 2019][1697184.602417] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Wed Feb 6 09:40:10 2019][1697184.609299] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Wed Feb 6 09:40:10 2019][1697184.616359] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 09:40:10 2019][1697184.623082] [] ? lnet_ni_send+0x44/0xd0 [lnet] [Wed Feb 6 09:40:10 2019][1697184.629362] [] ? lnet_send+0x82/0x1c0 [lnet] [Wed Feb 6 09:40:10 2019][1697184.635470] [] ? LNetPut+0x2cc/0xb50 [lnet] [Wed Feb 6 09:40:10 2019][1697184.641518] [] ? ptl_send_buf+0x146/0x530 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.648146] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:10 2019][1697184.655371] [] ? update_curr+0x14c/0x1e0 [Wed Feb 6 09:40:10 2019][1697184.661127] [] ? account_entity_dequeue+0xae/0xd0 [Wed Feb 6 09:40:10 2019][1697184.667659] [] ? dequeue_entity+0x11c/0x5e0 [Wed Feb 6 09:40:10 2019][1697184.673676] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:40:10 2019][1697184.679972] [] ? sched_clock_cpu+0x85/0xc0 [Wed Feb 6 09:40:10 2019][1697184.685905] [] ? tracing_is_on+0x15/0x30 [Wed Feb 6 09:40:10 2019][1697184.691658] [] ? tracing_record_cmdline+0x1d/0x120 [Wed Feb 6 09:40:10 2019][1697184.698281] [] ? probe_sched_switch+0x2d/0xa0 [Wed Feb 6 09:40:10 2019][1697184.704466] [] ? schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1697184.709822] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.716876] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1697184.722662] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.729319] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.736888] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1697184.742145] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.748419] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1697184.755233] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.761511] mdt01_016 S ffffa08b7a2b30c0 0 129230 2 0x00000080 [Wed Feb 6 09:40:10 2019][1697184.768879] Call Trace: [Wed Feb 6 09:40:10 2019][1697184.771513] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1697184.776692] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.783567] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1697184.789354] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.795832] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.803406] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1697184.808467] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.814741] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1697184.821364] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.827639] mdt01_017 S ffffa08b7a2b4100 0 129231 2 0x00000080 [Wed Feb 6 09:40:10 2019][1697184.834999] Call Trace: [Wed Feb 6 09:40:10 2019][1697184.837631] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1697184.842812] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.849696] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1697184.855481] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.861983] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.869559] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1697184.874622] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.880898] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1697184.887539] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.893809] mdt_io00_017 S ffffa08b7a2b5140 0 129238 2 0x00000080 [Wed Feb 6 09:40:10 2019][1697184.901172] Call Trace: [Wed Feb 6 09:40:10 2019][1697184.903800] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1697184.908983] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.915866] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1697184.921670] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.928156] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.935732] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1697184.940792] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.947068] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1697184.953710] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697184.959979] mdt_io02_018 S ffffa08b7a2b6180 0 129239 2 0x00000080 [Wed Feb 6 09:40:10 2019][1697184.967350] Call Trace: [Wed Feb 6 09:40:10 2019][1697184.969980] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1697184.975162] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.982068] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1697184.987847] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697184.994325] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697185.001903] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1697185.006963] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697185.013240] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1697185.019857] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697185.026132] mdt_io01_021 S ffffa06b77cde180 0 129240 2 0x00000080 [Wed Feb 6 09:40:10 2019][1697185.033502] Call Trace: [Wed Feb 6 09:40:10 2019][1697185.036134] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1697185.041316] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697185.048198] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1697185.053985] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697185.060461] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697185.068038] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1697185.073101] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697185.079375] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1697185.085995] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697185.092269] mdt_io02_019 S ffffa06b77cd9040 0 129241 2 0x00000080 [Wed Feb 6 09:40:10 2019][1697185.099631] Call Trace: [Wed Feb 6 09:40:10 2019][1697185.102262] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1697185.107443] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697185.114351] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1697185.120138] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697185.126623] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1697185.134219] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1697185.139276] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697185.145556] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1697185.152175] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1697185.158447] mdt01_018 S ffffa06b77cd8000 0 129242 2 0x00000080 [Wed Feb 6 09:40:10 2019][1697185.165819] Call Trace: [Wed Feb 6 09:40:10 2019][1697185.168458] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:40:10 2019][1697185.175342] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:40:11 2019][1697185.181482] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.189245] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1697185.194422] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.201510] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.208594] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:11 2019][1697185.215735] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1697185.221520] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.228886] [] ? top_trans_stop+0xed/0xa70 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.235627] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.243647] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:11 2019][1697185.250361] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:40:11 2019][1697185.257339] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:11 2019][1697185.264077] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.272100] [] ? mdd_xattr_set+0x371/0x17d0 [mdd] [Wed Feb 6 09:40:11 2019][1697185.278668] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:11 2019][1697185.285823] [] ? mdt_set_som+0xf4/0x2f0 [mdt] [Wed Feb 6 09:40:11 2019][1697185.292050] [] mdt_reint_object_lock+0x2c/0x60 [mdt] [Wed Feb 6 09:40:11 2019][1697185.298855] [] mdt_reint_striped_lock+0x8c/0x510 [mdt] [Wed Feb 6 09:40:11 2019][1697185.305831] [] ? mdt_lsom_update+0x85/0x440 [mdt] [Wed Feb 6 09:40:11 2019][1697185.312375] [] mdt_reint_setattr+0x6c8/0x12e0 [mdt] [Wed Feb 6 09:40:11 2019][1697185.319090] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 09:40:11 2019][1697185.325285] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 09:40:11 2019][1697185.332005] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Wed Feb 6 09:40:11 2019][1697185.338982] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 09:40:11 2019][1697185.344863] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.351972] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.359723] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:11 2019][1697185.366986] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.374850] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.381818] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:11 2019][1697185.388267] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:11 2019][1697185.394312] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.400815] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.408394] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1697185.413475] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.419746] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1697185.426365] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.432662] mdt01_019 S ffffa06b77cda080 0 129243 2 0x00000080 [Wed Feb 6 09:40:11 2019][1697185.440026] Call Trace: [Wed Feb 6 09:40:11 2019][1697185.442656] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1697185.447838] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.454714] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1697185.460504] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.466994] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.474570] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1697185.479632] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.485905] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1697185.492527] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.498823] mdt01_020 S ffffa06b77cdb0c0 0 129244 2 0x00000080 [Wed Feb 6 09:40:11 2019][1697185.506181] Call Trace: [Wed Feb 6 09:40:11 2019][1697185.508811] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1697185.513990] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.520867] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:40:11 2019][1697185.527836] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1697185.533650] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.540134] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.547709] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1697185.552773] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.559043] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1697185.565665] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.571938] mdt00_014 S ffffa06b77cdd140 0 129245 2 0x00000080 [Wed Feb 6 09:40:11 2019][1697185.579301] Call Trace: [Wed Feb 6 09:40:11 2019][1697185.581934] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1697185.587113] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.593987] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1697185.599774] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.606253] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.613828] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1697185.618888] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.625163] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1697185.631785] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.638058] mdt02_013 S ffffa06b77cdc100 0 129246 2 0x00000080 [Wed Feb 6 09:40:11 2019][1697185.645421] Call Trace: [Wed Feb 6 09:40:11 2019][1697185.648054] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1697185.653234] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.660116] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1697185.665920] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.672423] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.679998] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1697185.685082] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.691352] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1697185.697972] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.704248] mdt02_014 S ffffa05c29a4b0c0 0 129247 2 0x00000080 [Wed Feb 6 09:40:11 2019][1697185.711611] Call Trace: [Wed Feb 6 09:40:11 2019][1697185.714242] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1697185.719423] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.726305] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1697185.732091] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.738573] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.746143] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1697185.751203] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.757481] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1697185.764102] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.770375] mdt02_015 S ffffa06b79fc2080 0 129248 2 0x00000080 [Wed Feb 6 09:40:11 2019][1697185.777736] Call Trace: [Wed Feb 6 09:40:11 2019][1697185.780368] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1697185.785549] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.792482] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1697185.798266] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.804747] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.812323] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1697185.817384] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.823660] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1697185.830302] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.836571] mdt01_021 S ffffa06b79fc6180 0 129249 2 0x00000080 [Wed Feb 6 09:40:11 2019][1697185.843934] Call Trace: [Wed Feb 6 09:40:11 2019][1697185.846563] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1697185.851745] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.858619] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1697185.864407] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.870891] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.878468] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1697185.883526] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.889796] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1697185.896416] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.902692] mdt02_016 S ffffa06b79fc0000 0 129250 2 0x00000080 [Wed Feb 6 09:40:11 2019][1697185.910062] Call Trace: [Wed Feb 6 09:40:11 2019][1697185.912693] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1697185.917874] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.924757] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1697185.930544] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.937036] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.944612] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1697185.949672] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.955950] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1697185.962569] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697185.968844] mdt00_015 S ffffa06b79fc30c0 0 129251 2 0x00000080 [Wed Feb 6 09:40:11 2019][1697185.976225] Call Trace: [Wed Feb 6 09:40:11 2019][1697185.978855] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1697185.984037] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697185.990919] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1697185.996705] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697186.003181] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697186.010757] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1697186.015817] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697186.022095] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1697186.028738] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697186.035007] mdt01_022 S ffffa06b79fc4100 0 129252 2 0x00000080 [Wed Feb 6 09:40:11 2019][1697186.042367] Call Trace: [Wed Feb 6 09:40:11 2019][1697186.044998] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1697186.050180] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697186.057065] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1697186.062850] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697186.069354] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697186.076930] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1697186.081989] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697186.088267] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1697186.094933] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697186.101204] mdt01_023 S ffffa07b41e5e180 0 129253 2 0x00000080 [Wed Feb 6 09:40:11 2019][1697186.108578] Call Trace: [Wed Feb 6 09:40:11 2019][1697186.111214] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1697186.116392] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697186.123269] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1697186.129056] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697186.135536] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1697186.143120] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1697186.148176] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697186.154453] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1697186.161097] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1697186.167365] mdt01_024 S ffffa07b41e5d140 0 129254 2 0x00000080 [Wed Feb 6 09:40:11 2019][1697186.174727] Call Trace: [Wed Feb 6 09:40:11 2019][1697186.177358] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1697186.182540] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.189421] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1697186.195226] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.201710] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.209278] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1697186.214361] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.220633] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1697186.227254] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.233527] mdt01_025 S ffffa07b41e5c100 0 129255 2 0x00000080 [Wed Feb 6 09:40:12 2019][1697186.240897] Call Trace: [Wed Feb 6 09:40:12 2019][1697186.243529] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1697186.248708] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.255584] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1697186.261371] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.267850] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.275424] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1697186.280486] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.286761] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1697186.293379] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.299654] mdt01_026 S ffffa07b41e5a080 0 129256 2 0x00000080 [Wed Feb 6 09:40:12 2019][1697186.307018] Call Trace: [Wed Feb 6 09:40:12 2019][1697186.309647] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1697186.314827] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.321703] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1697186.327490] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.333993] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.341568] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1697186.346627] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.352906] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1697186.359527] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.365800] mdt00_016 S ffffa07b41e5b0c0 0 129257 2 0x00000080 [Wed Feb 6 09:40:12 2019][1697186.373161] Call Trace: [Wed Feb 6 09:40:12 2019][1697186.375794] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1697186.380973] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.387875] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1697186.393662] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.400147] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.407723] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1697186.412782] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.419059] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1697186.425681] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.431952] mdt00_017 S ffffa07b41e58000 0 129258 2 0x00000080 [Wed Feb 6 09:40:12 2019][1697186.439316] Call Trace: [Wed Feb 6 09:40:12 2019][1697186.441944] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1697186.447125] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.454026] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1697186.459805] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.466291] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.473866] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1697186.478927] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.485205] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1697186.491823] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.498099] mdt00_018 S ffffa07b41e59040 0 129259 2 0x00000080 [Wed Feb 6 09:40:12 2019][1697186.505469] Call Trace: [Wed Feb 6 09:40:12 2019][1697186.508102] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1697186.513279] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.520157] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1697186.525941] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.532422] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.540018] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1697186.545083] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.551359] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1697186.557977] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.564251] mdt02_017 S ffffa07b7a270000 0 129260 2 0x00000080 [Wed Feb 6 09:40:12 2019][1697186.571614] Call Trace: [Wed Feb 6 09:40:12 2019][1697186.574244] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1697186.579424] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.586300] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:40:12 2019][1697186.593268] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1697186.599066] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.605548] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.613123] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1697186.618185] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.624460] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1697186.631084] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.637380] mdt00_019 S ffffa07b7a274100 0 129262 2 0x00000080 [Wed Feb 6 09:40:12 2019][1697186.644744] Call Trace: [Wed Feb 6 09:40:12 2019][1697186.647374] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1697186.652556] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.659439] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1697186.665226] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.671730] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.679304] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1697186.684364] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.690643] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1697186.697284] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.703552] mdt_io00_018 S ffffa07b7a276180 0 129263 2 0x00000080 [Wed Feb 6 09:40:12 2019][1697186.710916] Call Trace: [Wed Feb 6 09:40:12 2019][1697186.713546] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1697186.718726] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.725602] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1697186.731390] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.737879] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.745459] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1697186.750518] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.756795] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1697186.763415] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.769689] mdt02_018 S ffffa07b3dee0000 0 129264 2 0x00000080 [Wed Feb 6 09:40:12 2019][1697186.777051] Call Trace: [Wed Feb 6 09:40:12 2019][1697186.779683] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1697186.784865] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.791746] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1697186.797534] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.804011] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.811584] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1697186.816648] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.822919] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1697186.829544] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.835817] mdt01_027 S ffffa07b3dee6180 0 129265 2 0x00000080 [Wed Feb 6 09:40:12 2019][1697186.843179] Call Trace: [Wed Feb 6 09:40:12 2019][1697186.845810] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1697186.850990] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.857866] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1697186.863652] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.870137] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.877713] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1697186.882774] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.889049] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1697186.895669] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.901942] mdt02_019 S ffffa07b3dee4100 0 129266 2 0x00000080 [Wed Feb 6 09:40:12 2019][1697186.909315] Call Trace: [Wed Feb 6 09:40:12 2019][1697186.911947] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1697186.917126] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.924002] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1697186.929787] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.936290] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.943867] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1697186.948927] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.955203] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1697186.961847] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697186.968114] mdt00_020 S ffffa07b3dee5140 0 129267 2 0x00000080 [Wed Feb 6 09:40:12 2019][1697186.975478] Call Trace: [Wed Feb 6 09:40:12 2019][1697186.978109] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1697186.983288] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697186.990164] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1697186.995951] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697187.002435] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697187.010010] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1697187.015072] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697187.021349] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1697187.027967] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697187.034244] mdt_io00_019 S ffffa07b3dee2080 0 129268 2 0x00000080 [Wed Feb 6 09:40:12 2019][1697187.041605] Call Trace: [Wed Feb 6 09:40:12 2019][1697187.044236] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1697187.049415] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697187.056314] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1697187.062096] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697187.068582] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697187.076156] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1697187.081217] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697187.087494] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1697187.094112] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697187.100407] mdt01_028 S ffffa07b3dee1040 0 129271 2 0x00000080 [Wed Feb 6 09:40:12 2019][1697187.107767] Call Trace: [Wed Feb 6 09:40:12 2019][1697187.110399] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1697187.115576] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697187.122453] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1697187.128242] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697187.134726] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:12 2019][1697187.142324] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1697187.147378] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697187.153656] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1697187.160275] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1697187.166550] mdt00_021 S ffffa07b3dee30c0 0 129276 2 0x00000080 [Wed Feb 6 09:40:12 2019][1697187.173920] Call Trace: [Wed Feb 6 09:40:12 2019][1697187.176550] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1697187.181732] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.188608] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697187.194392] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.200870] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.208445] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1697187.213505] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.219778] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1697187.226420] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.232695] mdt_io00_020 S ffffa08b62e90000 0 129277 2 0x00000080 [Wed Feb 6 09:40:13 2019][1697187.240073] Call Trace: [Wed Feb 6 09:40:13 2019][1697187.242706] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1697187.247885] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.254769] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697187.260556] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.267043] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.274617] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1697187.279702] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.285978] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1697187.292599] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.298874] mdt02_020 S ffffa08b62e91040 0 129278 2 0x00000080 [Wed Feb 6 09:40:13 2019][1697187.306237] Call Trace: [Wed Feb 6 09:40:13 2019][1697187.308867] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1697187.314046] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.320923] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697187.326709] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.333193] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.340772] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1697187.345831] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.352107] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1697187.358751] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.365029] mdt02_021 S ffffa08b62e92080 0 129279 2 0x00000080 [Wed Feb 6 09:40:13 2019][1697187.372390] Call Trace: [Wed Feb 6 09:40:13 2019][1697187.375020] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1697187.380202] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.387085] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697187.392891] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.399375] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.406950] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1697187.412009] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.418286] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1697187.424907] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.431181] mdt00_022 S ffffa08b62e930c0 0 129280 2 0x00000080 [Wed Feb 6 09:40:13 2019][1697187.438550] Call Trace: [Wed Feb 6 09:40:13 2019][1697187.441183] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1697187.446363] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.453240] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697187.459025] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.465527] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.473098] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1697187.478179] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.484458] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1697187.491078] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.497353] mdt_io01_022 S ffffa08b62e94100 0 129281 2 0x00000080 [Wed Feb 6 09:40:13 2019][1697187.504714] Call Trace: [Wed Feb 6 09:40:13 2019][1697187.507343] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1697187.512528] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.519407] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697187.525207] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.531688] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.539289] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1697187.544354] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.550626] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1697187.557249] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.563523] mdt_io01_023 S ffffa08b62e95140 0 129282 2 0x00000080 [Wed Feb 6 09:40:13 2019][1697187.570893] Call Trace: [Wed Feb 6 09:40:13 2019][1697187.573525] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1697187.578707] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.585589] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697187.591375] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.597876] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.605454] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1697187.610519] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.616790] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1697187.623409] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.629685] mdt_io01_024 S ffffa08b62e96180 0 129283 2 0x00000080 [Wed Feb 6 09:40:13 2019][1697187.637045] Call Trace: [Wed Feb 6 09:40:13 2019][1697187.639677] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1697187.644859] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.651742] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697187.657532] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.664006] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.671581] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1697187.676645] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.682916] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1697187.689557] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.695831] mdt_io01_025 S ffffa08b76720000 0 129284 2 0x00000080 [Wed Feb 6 09:40:13 2019][1697187.703192] Call Trace: [Wed Feb 6 09:40:13 2019][1697187.705821] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1697187.711002] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.717902] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697187.723691] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.730194] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.737769] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1697187.742831] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.749105] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1697187.755727] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.762000] mdt_io01_026 S ffffa08b76721040 0 129285 2 0x00000080 [Wed Feb 6 09:40:13 2019][1697187.769364] Call Trace: [Wed Feb 6 09:40:13 2019][1697187.771995] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1697187.777173] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.784050] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697187.789837] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.796320] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.803899] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1697187.808955] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.815257] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1697187.821879] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.828155] mdt_io02_020 S ffffa08b76722080 0 129292 2 0x00000080 [Wed Feb 6 09:40:13 2019][1697187.835517] Call Trace: [Wed Feb 6 09:40:13 2019][1697187.838147] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1697187.843327] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.850204] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697187.855988] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.862467] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.870041] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1697187.875101] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.881379] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1697187.888020] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.894290] mdt_io02_021 S ffffa08b767230c0 0 129293 2 0x00000080 [Wed Feb 6 09:40:13 2019][1697187.901670] Call Trace: [Wed Feb 6 09:40:13 2019][1697187.904301] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1697187.909502] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.916383] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697187.922168] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.928645] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.936220] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1697187.941281] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.947557] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1697187.954177] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697187.960452] mdt_io01_027 S ffffa08b76724100 0 129300 2 0x00000080 [Wed Feb 6 09:40:13 2019][1697187.967827] Call Trace: [Wed Feb 6 09:40:13 2019][1697187.970462] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1697187.975642] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.982545] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697187.988329] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697187.994813] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697188.002382] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1697188.007442] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697188.013720] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1697188.020364] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697188.026641] mdt_io01_028 S ffffa08b76725140 0 129301 2 0x00000080 [Wed Feb 6 09:40:13 2019][1697188.034001] Call Trace: [Wed Feb 6 09:40:13 2019][1697188.036633] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1697188.041811] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697188.048690] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697188.054473] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697188.060954] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697188.068527] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1697188.073589] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697188.079863] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1697188.086484] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697188.092761] mdt_io00_021 S ffffa08b76726180 0 129302 2 0x00000080 [Wed Feb 6 09:40:13 2019][1697188.100129] Call Trace: [Wed Feb 6 09:40:13 2019][1697188.102759] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1697188.107938] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697188.114816] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697188.120600] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697188.127121] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697188.134698] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1697188.139804] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697188.146078] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1697188.152725] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1697188.158999] mdt_io02_022 S ffffa08b6dbb0000 0 129309 2 0x00000080 [Wed Feb 6 09:40:13 2019][1697188.166362] Call Trace: [Wed Feb 6 09:40:13 2019][1697188.168991] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1697188.174174] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1697188.181056] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1697188.186846] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.193332] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.200926] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1697188.205980] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.212261] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1697188.218878] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.225174] mdt01_029 S ffffa08b6dbb2080 0 129371 2 0x00000080 [Wed Feb 6 09:40:14 2019][1697188.232531] Call Trace: [Wed Feb 6 09:40:14 2019][1697188.235164] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1697188.240362] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.247246] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1697188.253030] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.259535] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.267105] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1697188.272161] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.278438] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1697188.285057] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.291332] mdt01_030 S ffffa08b6dbb30c0 0 129372 2 0x00000080 [Wed Feb 6 09:40:14 2019][1697188.298702] Call Trace: [Wed Feb 6 09:40:14 2019][1697188.301335] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1697188.306516] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.313411] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1697188.319221] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.325703] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.333272] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1697188.338332] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.344631] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1697188.351247] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.357521] mdt01_031 S ffffa08b6dbb4100 0 129373 2 0x00000080 [Wed Feb 6 09:40:14 2019][1697188.364883] Call Trace: [Wed Feb 6 09:40:14 2019][1697188.367514] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1697188.372689] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.379593] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1697188.385374] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.391863] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.399436] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1697188.404496] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.410791] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1697188.417408] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.423684] mdt01_032 S ffffa08b6dbb5140 0 129374 2 0x00000080 [Wed Feb 6 09:40:14 2019][1697188.431043] Call Trace: [Wed Feb 6 09:40:14 2019][1697188.433675] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1697188.438855] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.445756] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1697188.451546] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.458021] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.465618] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1697188.470676] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.476951] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1697188.483571] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.489844] mdt00_023 S ffffa06b7f4ab0c0 0 129587 2 0x00000080 [Wed Feb 6 09:40:14 2019][1697188.497208] Call Trace: [Wed Feb 6 09:40:14 2019][1697188.499838] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1697188.505018] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.511895] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1697188.517680] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.524162] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.531732] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1697188.536793] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.543068] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1697188.549690] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.555963] mdt02_022 S ffffa06b7f4aa080 0 129594 2 0x00000080 [Wed Feb 6 09:40:14 2019][1697188.563351] Call Trace: [Wed Feb 6 09:40:14 2019][1697188.565984] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1697188.571163] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.578039] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1697188.583824] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.590325] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.597896] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1697188.602957] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.609231] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1697188.615853] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.622125] mdt02_023 S ffffa08b6dbb1040 0 129595 2 0x00000080 [Wed Feb 6 09:40:14 2019][1697188.629488] Call Trace: [Wed Feb 6 09:40:14 2019][1697188.632117] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1697188.637301] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.644183] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1697188.649969] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.656446] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.664023] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1697188.669084] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.675360] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1697188.682004] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.688280] mdt_io00_022 S ffffa07b75ef9040 0 129737 2 0x00000080 [Wed Feb 6 09:40:14 2019][1697188.695641] Call Trace: [Wed Feb 6 09:40:14 2019][1697188.698271] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1697188.703454] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.710339] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1697188.716125] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.722615] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.730186] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1697188.735245] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.741545] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1697188.748166] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.754442] mdt02_024 S ffffa07b75efa080 0 129793 2 0x00000080 [Wed Feb 6 09:40:14 2019][1697188.761803] Call Trace: [Wed Feb 6 09:40:14 2019][1697188.764435] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1697188.769615] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.776489] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1697188.782277] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.788753] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.796330] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1697188.801413] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.807682] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1697188.814304] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.820579] mdt00_024 S ffffa08b786f5140 0 129794 2 0x00000080 [Wed Feb 6 09:40:14 2019][1697188.827940] Call Trace: [Wed Feb 6 09:40:14 2019][1697188.830570] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1697188.835750] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.842628] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1697188.848414] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.854898] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.862476] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1697188.867535] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.873811] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1697188.880431] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.886705] mdt01_033 S ffffa07b78140000 0 129795 2 0x00000080 [Wed Feb 6 09:40:14 2019][1697188.894076] Call Trace: [Wed Feb 6 09:40:14 2019][1697188.896708] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1697188.901889] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.908773] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1697188.914566] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.921059] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.928635] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1697188.933698] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.939971] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1697188.946595] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697188.952869] mdt01_034 S ffffa07b781430c0 0 129796 2 0x00000080 [Wed Feb 6 09:40:14 2019][1697188.960228] Call Trace: [Wed Feb 6 09:40:14 2019][1697188.962860] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1697188.968043] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.974924] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1697188.980712] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.987189] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697188.994784] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1697188.999842] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697189.006116] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1697189.012737] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697189.019013] mdt01_035 S ffffa07b78146180 0 129797 2 0x00000080 [Wed Feb 6 09:40:14 2019][1697189.026373] Call Trace: [Wed Feb 6 09:40:14 2019][1697189.029005] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1697189.034186] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697189.041092] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1697189.046873] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697189.053351] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697189.060926] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1697189.065987] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697189.072265] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1697189.078881] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697189.085157] mdt02_025 S ffffa07b78142080 0 129805 2 0x00000080 [Wed Feb 6 09:40:14 2019][1697189.092520] Call Trace: [Wed Feb 6 09:40:14 2019][1697189.095155] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1697189.100334] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697189.107237] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1697189.113029] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697189.119510] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1697189.127076] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1697189.132142] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697189.138416] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1697189.145038] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1697189.151308] mdt02_026 S ffffa07b78145140 0 129806 2 0x00000080 [Wed Feb 6 09:40:14 2019][1697189.158672] Call Trace: [Wed Feb 6 09:40:14 2019][1697189.161304] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1697189.166484] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.173359] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1697189.179147] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.185631] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.193208] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1697189.198267] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.204543] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1697189.211193] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.217464] mdt_io01_029 S ffffa06b76152080 0 129974 2 0x00000080 [Wed Feb 6 09:40:15 2019][1697189.224844] Call Trace: [Wed Feb 6 09:40:15 2019][1697189.227476] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1697189.232653] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.239532] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1697189.245315] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.251794] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.259369] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1697189.264428] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.270705] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1697189.277327] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.283600] mdt_io01_030 S ffffa06b76151040 0 130108 2 0x00000080 [Wed Feb 6 09:40:15 2019][1697189.290963] Call Trace: [Wed Feb 6 09:40:15 2019][1697189.293618] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1697189.298802] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.305683] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1697189.311471] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.317949] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.325521] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1697189.330589] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.336857] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1697189.343481] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.349753] mdt_io01_031 S ffffa06b76154100 0 130109 2 0x00000080 [Wed Feb 6 09:40:15 2019][1697189.357116] Call Trace: [Wed Feb 6 09:40:15 2019][1697189.359746] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1697189.364927] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.371805] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1697189.377591] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.384078] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.391650] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1697189.396710] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.402986] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1697189.409607] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.415882] mdt_io01_032 S ffffa06b76155140 0 130112 2 0x00000080 [Wed Feb 6 09:40:15 2019][1697189.423244] Call Trace: [Wed Feb 6 09:40:15 2019][1697189.425875] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1697189.431053] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.437931] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1697189.443716] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.450192] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.457768] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1697189.462851] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.469122] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1697189.475742] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.482019] mdt_io02_023 S ffffa08b5d156180 0 130117 2 0x00000080 [Wed Feb 6 09:40:15 2019][1697189.489380] Call Trace: [Wed Feb 6 09:40:15 2019][1697189.492010] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1697189.497191] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.504065] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1697189.509855] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.516338] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.523913] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1697189.528974] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.535250] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1697189.541870] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.548147] mdt_io02_024 S ffffa07b2cf88000 0 130203 2 0x00000080 [Wed Feb 6 09:40:15 2019][1697189.555507] Call Trace: [Wed Feb 6 09:40:15 2019][1697189.558139] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1697189.563319] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.570203] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1697189.575991] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.582502] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.590075] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1697189.595136] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.601413] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1697189.608032] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.614331] mdt_io02_025 S ffffa07b2cf89040 0 130204 2 0x00000080 [Wed Feb 6 09:40:15 2019][1697189.621686] Call Trace: [Wed Feb 6 09:40:15 2019][1697189.624318] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1697189.629499] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.636382] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1697189.642169] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.648646] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.656222] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1697189.661281] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.667559] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1697189.674178] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.680451] mdt_io00_023 S ffffa07b2cf8c100 0 130240 2 0x00000080 [Wed Feb 6 09:40:15 2019][1697189.687813] Call Trace: [Wed Feb 6 09:40:15 2019][1697189.690445] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1697189.695625] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.702503] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1697189.708287] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.714764] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.722339] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1697189.727401] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.733676] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1697189.740298] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.746571] ldlm_cn00_010 S ffffa07b42bdb0c0 0 130318 2 0x00000080 [Wed Feb 6 09:40:15 2019][1697189.753933] Call Trace: [Wed Feb 6 09:40:15 2019][1697189.756564] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1697189.761743] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.768621] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1697189.774407] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.780883] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.788459] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1697189.793518] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.799795] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1697189.806415] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.812712] ldlm_cn00_011 S ffffa07b42bdd140 0 130319 2 0x00000080 [Wed Feb 6 09:40:15 2019][1697189.820077] Call Trace: [Wed Feb 6 09:40:15 2019][1697189.822711] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1697189.827891] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.834776] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:40:15 2019][1697189.841742] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1697189.847557] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.854030] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.861605] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1697189.866667] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.872943] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1697189.879563] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.885892] ldlm_cn00_012 S ffffa07b42bd9040 0 130320 2 0x00000080 [Wed Feb 6 09:40:15 2019][1697189.893251] Call Trace: [Wed Feb 6 09:40:15 2019][1697189.895883] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1697189.901065] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.907949] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1697189.913733] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.920211] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.927786] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1697189.932846] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.939123] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1697189.945765] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697189.952033] ldlm_cn00_013 S ffffa07b42bda080 0 130321 2 0x00000080 [Wed Feb 6 09:40:15 2019][1697189.959396] Call Trace: [Wed Feb 6 09:40:15 2019][1697189.962028] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1697189.967208] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.974083] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1697189.979870] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.986347] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697189.993921] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1697189.998982] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697190.005258] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1697190.011880] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697190.018153] ldlm_cn00_014 S ffffa07b42bdc100 0 130322 2 0x00000080 [Wed Feb 6 09:40:15 2019][1697190.025525] Call Trace: [Wed Feb 6 09:40:15 2019][1697190.028156] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1697190.033336] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697190.040221] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1697190.046005] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697190.052485] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697190.060059] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1697190.065143] [] ? finish_task_switch+0x54/0x1c0 [Wed Feb 6 09:40:15 2019][1697190.071421] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697190.077693] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1697190.084317] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697190.090591] ldlm_cn00_015 S ffffa07b42bde180 0 130323 2 0x00000080 [Wed Feb 6 09:40:15 2019][1697190.097960] Call Trace: [Wed Feb 6 09:40:15 2019][1697190.100592] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1697190.105772] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697190.112648] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1697190.118434] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697190.124937] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1697190.132512] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1697190.137573] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697190.143849] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1697190.150470] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1697190.156743] ldlm_cn00_016 S ffffa08b2b388000 0 130324 2 0x00000080 [Wed Feb 6 09:40:15 2019][1697190.164106] Call Trace: [Wed Feb 6 09:40:15 2019][1697190.166738] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1697190.171920] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.178801] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1697190.184589] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.191073] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.198647] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1697190.203708] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.209987] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1697190.216607] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.222879] ldlm_cn00_017 S ffffa08b2b389040 0 130325 2 0x00000080 [Wed Feb 6 09:40:16 2019][1697190.230242] Call Trace: [Wed Feb 6 09:40:16 2019][1697190.232871] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1697190.238053] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.244952] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1697190.250740] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.257218] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.264793] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1697190.269854] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.276132] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1697190.282752] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.289025] ldlm_cn00_018 S ffffa08b2b38a080 0 130326 2 0x00000080 [Wed Feb 6 09:40:16 2019][1697190.296387] Call Trace: [Wed Feb 6 09:40:16 2019][1697190.299018] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1697190.304201] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.311082] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1697190.316868] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.323345] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.330920] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1697190.335981] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.342258] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1697190.348878] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.355152] ldlm_cn00_019 S ffffa08b2b38b0c0 0 130327 2 0x00000080 [Wed Feb 6 09:40:16 2019][1697190.362514] Call Trace: [Wed Feb 6 09:40:16 2019][1697190.365147] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1697190.370327] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.377209] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1697190.382997] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.389474] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.397050] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1697190.402109] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.408385] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1697190.415007] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.421280] ldlm_cn00_020 S ffffa08b2b38c100 0 130328 2 0x00000080 [Wed Feb 6 09:40:16 2019][1697190.428660] Call Trace: [Wed Feb 6 09:40:16 2019][1697190.431292] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1697190.436471] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.443345] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:40:16 2019][1697190.450317] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1697190.456102] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.462584] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.470152] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1697190.475216] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.481487] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1697190.488112] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.494409] ldlm_cn00_021 S ffffa08b2b38d140 0 130329 2 0x00000080 [Wed Feb 6 09:40:16 2019][1697190.501771] Call Trace: [Wed Feb 6 09:40:16 2019][1697190.504402] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1697190.509586] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.516469] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1697190.522254] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.528737] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.536306] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1697190.541365] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.547643] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1697190.554263] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.560540] ldlm_cn00_022 S ffffa08b2b38e180 0 130330 2 0x00000080 [Wed Feb 6 09:40:16 2019][1697190.567900] Call Trace: [Wed Feb 6 09:40:16 2019][1697190.570528] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1697190.575712] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.582595] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1697190.588401] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.594887] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.602460] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1697190.607552] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.613821] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1697190.620442] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.626718] ldlm_cn00_023 S ffffa08ab9ae8000 0 130331 2 0x00000080 [Wed Feb 6 09:40:16 2019][1697190.634106] Call Trace: [Wed Feb 6 09:40:16 2019][1697190.636737] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1697190.641917] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.648800] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1697190.654586] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.661069] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.668640] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1697190.673699] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.679976] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1697190.686596] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.692872] ldlm_cn00_024 S ffffa08ab9ae9040 0 130332 2 0x00000080 [Wed Feb 6 09:40:16 2019][1697190.700232] Call Trace: [Wed Feb 6 09:40:16 2019][1697190.702864] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1697190.708045] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.714927] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1697190.720715] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.727218] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.734793] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1697190.739854] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.746128] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1697190.752749] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.759024] ldlm_cn00_025 S ffffa08ab9aea080 0 130333 2 0x00000080 [Wed Feb 6 09:40:16 2019][1697190.766386] Call Trace: [Wed Feb 6 09:40:16 2019][1697190.769016] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1697190.774197] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.781097] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1697190.786884] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.793362] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.800938] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1697190.805997] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.812274] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1697190.818895] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.825171] ldlm_cn00_026 S ffffa08ab9aeb0c0 0 130336 2 0x00000080 [Wed Feb 6 09:40:16 2019][1697190.832541] Call Trace: [Wed Feb 6 09:40:16 2019][1697190.835170] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1697190.840352] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.847260] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1697190.853048] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.859539] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.867114] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1697190.872169] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.878445] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1697190.885067] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.891339] ldlm_cn00_027 S ffffa08ab9aec100 0 130337 2 0x00000080 [Wed Feb 6 09:40:16 2019][1697190.898701] Call Trace: [Wed Feb 6 09:40:16 2019][1697190.901333] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1697190.906515] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.913400] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1697190.919185] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.925670] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.933246] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1697190.938306] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.944581] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1697190.951201] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697190.957477] ldlm_cn00_028 S ffffa08ab9aed140 0 130342 2 0x00000080 [Wed Feb 6 09:40:16 2019][1697190.964838] Call Trace: [Wed Feb 6 09:40:16 2019][1697190.967471] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1697190.972651] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.979533] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1697190.985319] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.991802] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697190.999372] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1697191.004433] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697191.010711] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1697191.017329] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697191.023603] ldlm_cn00_029 S ffffa08ab9aee180 0 130343 2 0x00000080 [Wed Feb 6 09:40:16 2019][1697191.030993] Call Trace: [Wed Feb 6 09:40:16 2019][1697191.033624] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1697191.038802] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697191.045679] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1697191.051466] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697191.057957] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697191.065534] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1697191.070594] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697191.076872] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1697191.083492] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697191.089769] ldlm_cn00_030 S ffffa08b23bf8000 0 130345 2 0x00000080 [Wed Feb 6 09:40:16 2019][1697191.097128] Call Trace: [Wed Feb 6 09:40:16 2019][1697191.099758] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1697191.104940] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697191.111824] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1697191.117609] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697191.124088] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1697191.131664] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1697191.136723] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697191.142999] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1697191.149642] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1697191.155909] ldlm_cn00_031 S ffffa08b23bfa080 0 130353 2 0x00000080 [Wed Feb 6 09:40:16 2019][1697191.163271] Call Trace: [Wed Feb 6 09:40:16 2019][1697191.165903] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1697191.171082] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.177960] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1697191.183745] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.190227] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.197798] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1697191.202857] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.209135] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1697191.215754] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.222052] ldlm_cn02_008 S ffffa07b7c8830c0 0 130824 2 0x00000080 [Wed Feb 6 09:40:17 2019][1697191.229410] Call Trace: [Wed Feb 6 09:40:17 2019][1697191.232041] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1697191.237222] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.244103] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1697191.249890] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.256370] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.263944] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1697191.269027] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.275297] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1697191.281916] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.288192] mdt_io01_033 S ffffa08a91f52080 0 131047 2 0x00000080 [Wed Feb 6 09:40:17 2019][1697191.295553] Call Trace: [Wed Feb 6 09:40:17 2019][1697191.298184] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1697191.303364] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.310240] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1697191.316026] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.322503] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.330080] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1697191.335140] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.341416] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1697191.348036] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.354311] ldlm_cn01_009 S ffffa08a91f530c0 0 131061 2 0x00000080 [Wed Feb 6 09:40:17 2019][1697191.361680] Call Trace: [Wed Feb 6 09:40:17 2019][1697191.364311] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1697191.369494] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.376378] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1697191.382165] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.388692] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.396266] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1697191.401328] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.407603] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1697191.414225] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.420501] ldlm_cn01_010 S ffffa08a91f55140 0 131071 2 0x00000080 [Wed Feb 6 09:40:17 2019][1697191.427860] Call Trace: [Wed Feb 6 09:40:17 2019][1697191.430493] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1697191.435673] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.442558] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1697191.448343] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.454825] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.462396] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1697191.467455] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.473733] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1697191.480352] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.486628] ldlm_cn02_009 S ffffa08a91f56180 0 312 2 0x00000080 [Wed Feb 6 09:40:17 2019][1697191.493909] Call Trace: [Wed Feb 6 09:40:17 2019][1697191.496542] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1697191.501716] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.508597] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1697191.514382] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.520863] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.528435] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1697191.533496] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.539770] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1697191.546392] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.552668] mdt01_036 S ffffa05b73552080 0 847 2 0x00000080 [Wed Feb 6 09:40:17 2019][1697191.559942] Call Trace: [Wed Feb 6 09:40:17 2019][1697191.562573] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1697191.567755] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.574638] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1697191.580423] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.586902] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.594477] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1697191.599537] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.605814] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1697191.612433] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.618710] mdt_io00_024 S ffffa08a73e2a080 0 1012 2 0x00000080 [Wed Feb 6 09:40:17 2019][1697191.625982] Call Trace: [Wed Feb 6 09:40:17 2019][1697191.628616] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1697191.633793] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.640670] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1697191.646457] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.652943] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.660516] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1697191.665579] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.671853] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1697191.678473] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.684748] mdt_rdpg00_005 S ffffa07b77ec9040 0 1270 2 0x00000080 [Wed Feb 6 09:40:17 2019][1697191.692042] Call Trace: [Wed Feb 6 09:40:17 2019][1697191.694675] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1697191.699852] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.706729] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1697191.712516] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.719002] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.726576] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1697191.731638] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.737912] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1697191.744534] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.750807] mdt_rdpg00_006 S ffffa07b77ecb0c0 0 1271 2 0x00000080 [Wed Feb 6 09:40:17 2019][1697191.758082] Call Trace: [Wed Feb 6 09:40:17 2019][1697191.760714] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1697191.765893] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.772769] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1697191.778556] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.785033] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.792607] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1697191.797670] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.803944] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1697191.810565] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.816840] mdt_io00_025 S ffffa07b3dd7a080 0 1416 2 0x00000080 [Wed Feb 6 09:40:17 2019][1697191.824114] Call Trace: [Wed Feb 6 09:40:17 2019][1697191.826745] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1697191.831926] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.838810] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1697191.844597] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.851087] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.858657] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1697191.863718] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.869996] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1697191.876615] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.882891] mdt_io00_026 S ffffa07b3dd7e180 0 1417 2 0x00000080 [Wed Feb 6 09:40:17 2019][1697191.890166] Call Trace: [Wed Feb 6 09:40:17 2019][1697191.892795] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1697191.897975] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.904850] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1697191.910637] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.917114] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.924690] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1697191.929773] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.936045] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1697191.942664] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697191.948939] mdt_io00_027 S ffffa07b3dd78000 0 1418 2 0x00000080 [Wed Feb 6 09:40:17 2019][1697191.956213] Call Trace: [Wed Feb 6 09:40:17 2019][1697191.958845] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1697191.964023] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.970900] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1697191.976686] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.983164] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697191.990739] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1697191.995801] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697192.002076] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1697192.008696] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697192.014973] mdt_out03_002 S ffffa07b7415b0c0 0 1423 2 0x00000080 [Wed Feb 6 09:40:17 2019][1697192.022262] Call Trace: [Wed Feb 6 09:40:17 2019][1697192.024892] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1697192.030076] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697192.036958] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1697192.042743] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697192.049247] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697192.056823] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1697192.061885] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697192.068159] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1697192.074782] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697192.081054] ldlm_cn01_011 S ffffa07b7415c100 0 1442 2 0x00000080 [Wed Feb 6 09:40:17 2019][1697192.088338] Call Trace: [Wed Feb 6 09:40:17 2019][1697192.090972] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1697192.096149] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697192.103025] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:40:17 2019][1697192.109993] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1697192.115783] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697192.122261] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697192.129831] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1697192.134892] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697192.141166] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1697192.147791] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1697192.154064] ldlm_cn01_012 S ffffa07b74159040 0 1443 2 0x00000080 [Wed Feb 6 09:40:17 2019][1697192.161347] Call Trace: [Wed Feb 6 09:40:17 2019][1697192.163978] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1697192.169160] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1697192.176044] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1697192.181827] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.188306] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.195881] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697192.200944] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.207217] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1697192.213838] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.220111] ldlm_cn01_013 S ffffa07b7415a080 0 1444 2 0x00000080 [Wed Feb 6 09:40:18 2019][1697192.227378] Call Trace: [Wed Feb 6 09:40:18 2019][1697192.230013] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1697192.235190] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.242067] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1697192.247852] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.254329] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.261905] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697192.266965] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.273244] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1697192.279864] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.286136] mdt_out02_005 S ffffa07b7415e180 0 1445 2 0x00000080 [Wed Feb 6 09:40:18 2019][1697192.293411] Call Trace: [Wed Feb 6 09:40:18 2019][1697192.296043] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1697192.301224] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.308108] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1697192.313893] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.320372] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.327947] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697192.333007] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.339304] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1697192.345919] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.352194] mdt_out02_006 S ffffa0557c424100 0 2325 2 0x00000080 [Wed Feb 6 09:40:18 2019][1697192.359478] Call Trace: [Wed Feb 6 09:40:18 2019][1697192.362111] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1697192.367291] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.374175] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1697192.379960] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.386437] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.394013] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697192.399075] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.405371] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1697192.412013] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.418287] kworker/16:2 S ffffa05b72d4a080 0 2843 2 0x00000080 [Wed Feb 6 09:40:18 2019][1697192.425569] Call Trace: [Wed Feb 6 09:40:18 2019][1697192.428204] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:18 2019][1697192.434394] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1697192.439537] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:18 2019][1697192.445294] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:18 2019][1697192.452001] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697192.457065] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.463336] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1697192.469961] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.476233] mdt_seqm_0002 S ffffa08a5f63a080 0 2903 2 0x00000080 [Wed Feb 6 09:40:18 2019][1697192.483515] Call Trace: [Wed Feb 6 09:40:18 2019][1697192.486149] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1697192.491329] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.498214] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1697192.503999] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.510481] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.518053] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697192.523112] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.529389] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1697192.536033] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.542310] mdt00_025 S ffffa08a5f63b0c0 0 2937 2 0x00000080 [Wed Feb 6 09:40:18 2019][1697192.549583] Call Trace: [Wed Feb 6 09:40:18 2019][1697192.552215] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1697192.557395] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.564272] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1697192.570056] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.576536] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.584110] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697192.589219] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.595488] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1697192.602111] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.608385] mdt00_026 S ffffa08b76cc9040 0 2976 2 0x00000080 [Wed Feb 6 09:40:18 2019][1697192.615660] Call Trace: [Wed Feb 6 09:40:18 2019][1697192.618291] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1697192.623470] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.630347] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1697192.636133] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.642611] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.650185] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697192.655244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.661522] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1697192.668144] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.674419] mdt_rdpg02_005 S ffffa07b77ece180 0 3266 2 0x00000080 [Wed Feb 6 09:40:18 2019][1697192.681691] Call Trace: [Wed Feb 6 09:40:18 2019][1697192.684321] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1697192.689505] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.696388] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1697192.702175] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.708676] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.716250] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697192.721312] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.727589] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1697192.734210] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.740484] mdt_io01_034 S ffffa08a83a50000 0 3361 2 0x00000080 [Wed Feb 6 09:40:18 2019][1697192.747760] Call Trace: [Wed Feb 6 09:40:18 2019][1697192.750389] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1697192.755574] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.762461] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1697192.768278] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.774759] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.782326] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697192.787392] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.793663] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1697192.800286] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.806564] mdt_io01_035 S ffffa08a83a54100 0 3384 2 0x00000080 [Wed Feb 6 09:40:18 2019][1697192.813842] Call Trace: [Wed Feb 6 09:40:18 2019][1697192.816474] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1697192.821654] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.828559] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1697192.834343] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.840825] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.848396] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697192.853454] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.859732] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1697192.866352] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.872627] ldlm_cn02_010 S ffffa08a83a52080 0 3385 2 0x00000080 [Wed Feb 6 09:40:18 2019][1697192.879901] Call Trace: [Wed Feb 6 09:40:18 2019][1697192.882533] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1697192.887713] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.894588] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1697192.900376] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.906861] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.914436] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697192.919493] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.925764] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1697192.932385] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.938659] mdt_out03_003 S ffffa08a30215140 0 3908 2 0x00000080 [Wed Feb 6 09:40:18 2019][1697192.945932] Call Trace: [Wed Feb 6 09:40:18 2019][1697192.948564] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1697192.953744] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.960621] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1697192.966407] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.972892] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697192.980469] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697192.985527] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697192.991804] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1697192.998425] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697193.004699] ldlm_cn01_014 S ffffa05b7c6e30c0 0 4337 2 0x00000080 [Wed Feb 6 09:40:18 2019][1697193.011989] Call Trace: [Wed Feb 6 09:40:18 2019][1697193.014623] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1697193.019804] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697193.026687] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1697193.032474] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697193.038949] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697193.046526] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697193.051585] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697193.057864] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1697193.064482] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697193.070759] ldlm_cn01_015 S ffffa05b7c6e4100 0 4338 2 0x00000080 [Wed Feb 6 09:40:18 2019][1697193.078032] Call Trace: [Wed Feb 6 09:40:18 2019][1697193.080662] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1697193.085844] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697193.092728] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1697193.098511] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697193.104993] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1697193.112565] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697193.117627] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697193.123905] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1697193.130547] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1697193.136824] kworker/11:2 S ffffa08a73e2d140 0 11106 2 0x00000080 [Wed Feb 6 09:40:18 2019][1697193.144105] Call Trace: [Wed Feb 6 09:40:18 2019][1697193.146741] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:18 2019][1697193.152928] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1697193.158076] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:18 2019][1697193.163833] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:18 2019][1697193.170537] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1697193.175599] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.181875] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697193.188497] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.194772] mdt03_013 S ffffa05b7c70d140 0 14718 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697193.202047] Call Trace: [Wed Feb 6 09:40:19 2019][1697193.204677] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697193.209858] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.216739] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1697193.222526] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.229011] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.236590] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697193.241649] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.247926] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697193.254544] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.260820] mdt_rdpg01_007 S ffffa059eced0000 0 16573 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697193.268093] Call Trace: [Wed Feb 6 09:40:19 2019][1697193.270725] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697193.275906] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.282782] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1697193.288570] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.295045] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.302620] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697193.307684] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.313954] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697193.320579] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.326854] kworker/2:1 S ffffa07b4eb48000 0 16820 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697193.334132] Call Trace: [Wed Feb 6 09:40:19 2019][1697193.336766] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:19 2019][1697193.342955] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697193.348103] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:19 2019][1697193.353861] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:19 2019][1697193.360567] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697193.365629] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.371902] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697193.378525] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.384796] kworker/44:0 S ffffa05b7c6e1040 0 17593 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697193.392078] Call Trace: [Wed Feb 6 09:40:19 2019][1697193.394715] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:19 2019][1697193.400902] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697193.406048] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:19 2019][1697193.411806] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:19 2019][1697193.418510] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697193.423574] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.429847] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697193.436493] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.442769] mdt_rdpg01_008 S ffffa08b61331040 0 18321 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697193.450046] Call Trace: [Wed Feb 6 09:40:19 2019][1697193.452675] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697193.457862] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.464743] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1697193.470530] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.477013] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.484587] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697193.489649] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.495925] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697193.502545] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.508820] mdt_rdpg01_009 S ffffa08b61330000 0 18322 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697193.516095] Call Trace: [Wed Feb 6 09:40:19 2019][1697193.518726] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697193.523905] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.530781] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1697193.536571] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.543059] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.550627] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697193.555691] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.561962] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697193.568587] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.574860] ldlm_cb02_004 S ffffa07b76b4d140 0 18369 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697193.582135] Call Trace: [Wed Feb 6 09:40:19 2019][1697193.584766] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697193.589966] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.596848] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1697193.602634] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.609138] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.616713] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697193.621774] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.628049] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697193.634670] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.640943] kworker/22:2 S ffffa07b76b4b0c0 0 20891 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697193.648218] Call Trace: [Wed Feb 6 09:40:19 2019][1697193.650850] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:19 2019][1697193.657038] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697193.662211] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:19 2019][1697193.667962] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:19 2019][1697193.674668] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697193.679728] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.686004] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697193.692626] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.698899] kworker/3:0 S ffffa08b7ca9a080 0 33438 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697193.706195] Call Trace: [Wed Feb 6 09:40:19 2019][1697193.708831] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:19 2019][1697193.715022] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697193.720191] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:19 2019][1697193.725943] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:19 2019][1697193.732647] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697193.737710] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.743985] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697193.750607] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.756880] kworker/28:0 S ffffa059eced2080 0 34393 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697193.764160] Call Trace: [Wed Feb 6 09:40:19 2019][1697193.766794] [] ? start_worker+0x27/0x30 [Wed Feb 6 09:40:19 2019][1697193.772464] [] ? manage_workers.isra.25+0x189/0x2a0 [Wed Feb 6 09:40:19 2019][1697193.779196] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697193.784345] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:19 2019][1697193.790101] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:19 2019][1697193.796809] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697193.801870] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.808148] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697193.814765] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.821039] kworker/17:2 S ffffa06b7c0ad140 0 34465 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697193.828329] Call Trace: [Wed Feb 6 09:40:19 2019][1697193.830964] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:19 2019][1697193.837153] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697193.842299] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:19 2019][1697193.848057] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:19 2019][1697193.854764] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697193.859825] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.866098] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697193.872724] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.878994] kworker/25:2 S ffffa069597cd140 0 37144 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697193.886282] Call Trace: [Wed Feb 6 09:40:19 2019][1697193.888910] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697193.894058] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:19 2019][1697193.899812] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:19 2019][1697193.906522] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697193.911582] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.917859] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697193.924501] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.930770] mdt_out01_002 S ffffa06b17274100 0 38264 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697193.938047] Call Trace: [Wed Feb 6 09:40:19 2019][1697193.940677] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697193.945865] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.952765] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1697193.958552] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.965030] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697193.972605] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697193.977666] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.983942] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697193.990563] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697193.996837] kworker/33:1 S ffffa06b787ca080 0 39444 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697194.004125] Call Trace: [Wed Feb 6 09:40:19 2019][1697194.006761] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:19 2019][1697194.012949] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697194.018098] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:19 2019][1697194.023856] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:19 2019][1697194.030561] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697194.035619] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697194.041886] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697194.048510] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697194.054783] kworker/28:1 S ffffa05b6fadd140 0 39909 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697194.062071] Call Trace: [Wed Feb 6 09:40:19 2019][1697194.064706] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:19 2019][1697194.070895] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697194.076042] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:19 2019][1697194.081799] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:19 2019][1697194.088507] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697194.093571] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697194.099842] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697194.106466] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697194.112736] mdt_io03_003 S ffffa06b787ce180 0 40441 2 0x00000080 [Wed Feb 6 09:40:19 2019][1697194.120014] Call Trace: [Wed Feb 6 09:40:19 2019][1697194.122642] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1697194.127824] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697194.134700] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1697194.140485] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697194.146964] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1697194.154540] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1697194.159600] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697194.165877] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1697194.172495] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1697194.178771] mdt_io03_004 S ffffa06b787c9040 0 40442 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697194.186054] Call Trace: [Wed Feb 6 09:40:20 2019][1697194.188688] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697194.193864] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.200740] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1697194.206527] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.213007] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.220604] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697194.225666] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.231942] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697194.238565] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.244833] mdt_io03_005 S ffffa05c29a130c0 0 40451 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697194.252129] Call Trace: [Wed Feb 6 09:40:20 2019][1697194.254761] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697194.259943] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.266825] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1697194.272610] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.279088] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.286664] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697194.291725] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.297999] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697194.304622] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.310922] mdt_io03_006 S ffffa07c294430c0 0 40452 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697194.318196] Call Trace: [Wed Feb 6 09:40:20 2019][1697194.320827] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697194.326008] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.332885] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1697194.338670] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.345148] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.352721] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697194.357783] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.364060] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697194.370703] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.376978] mdt_io03_007 S ffffa07b3ca6a080 0 40453 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697194.384253] Call Trace: [Wed Feb 6 09:40:20 2019][1697194.386888] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697194.392066] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.398940] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1697194.404727] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.411205] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.418778] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697194.423842] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.430118] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697194.436736] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.443012] mdt_io03_008 S ffffa07b3ca6d140 0 40479 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697194.450294] Call Trace: [Wed Feb 6 09:40:20 2019][1697194.452925] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697194.458106] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.464981] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1697194.470771] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.477254] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.484830] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697194.489913] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.496185] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697194.502806] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.509080] ldlm_cn03_004 S ffffa07b7c881040 0 40888 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697194.516362] Call Trace: [Wed Feb 6 09:40:20 2019][1697194.518992] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697194.524175] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.531059] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1697194.536845] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.543332] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.550905] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697194.555967] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.562243] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697194.568863] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.575136] mdt_io03_009 S ffffa07b7c884100 0 41381 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697194.582413] Call Trace: [Wed Feb 6 09:40:20 2019][1697194.585041] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697194.590224] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.597133] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1697194.602922] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.609431] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.617007] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697194.622068] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.628344] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697194.634965] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.641238] kworker/12:0 S ffffa0521450d140 0 46894 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697194.648517] Call Trace: [Wed Feb 6 09:40:20 2019][1697194.651153] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:20 2019][1697194.657342] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697194.662488] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:20 2019][1697194.668247] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:20 2019][1697194.674951] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697194.680015] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.686287] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697194.692933] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.699202] ldlm_cn03_005 S ffffa06b76022080 0 51590 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697194.706485] Call Trace: [Wed Feb 6 09:40:20 2019][1697194.709116] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697194.714296] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.721172] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1697194.726956] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.733436] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.741010] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697194.746071] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.752348] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697194.758969] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.765245] ldlm_cn03_006 S ffffa06b760230c0 0 51608 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697194.772516] Call Trace: [Wed Feb 6 09:40:20 2019][1697194.775150] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697194.780332] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.787263] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1697194.793054] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.799536] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.807112] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697194.812174] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.818449] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697194.825070] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.831345] ldlm_cn03_007 S ffffa08b79f2e180 0 51807 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697194.838646] Call Trace: [Wed Feb 6 09:40:20 2019][1697194.841275] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697194.846458] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.853364] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1697194.859153] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.865635] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.873207] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697194.878268] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.884563] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697194.891178] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.897456] ldlm_cn03_008 S ffffa08a5f63d140 0 51825 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697194.904730] Call Trace: [Wed Feb 6 09:40:20 2019][1697194.907360] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697194.912544] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.919423] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1697194.925210] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.931716] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1697194.939289] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697194.944350] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.950647] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697194.957264] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697194.963540] kworker/29:0 S ffffa06b7807e180 0 54901 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697194.970836] Call Trace: [Wed Feb 6 09:40:20 2019][1697194.973471] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:20 2019][1697194.979662] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697194.984806] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:20 2019][1697194.990564] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:20 2019][1697194.997270] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697195.002332] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697195.008606] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697195.015229] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697195.021502] kworker/44:2 S ffffa05214508000 0 76594 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697195.028790] Call Trace: [Wed Feb 6 09:40:20 2019][1697195.031425] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697195.036602] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:20 2019][1697195.042354] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:20 2019][1697195.049064] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697195.054123] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697195.060399] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697195.067020] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697195.073295] kworker/21:2 S ffffa06b76156180 0 86721 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697195.080579] Call Trace: [Wed Feb 6 09:40:20 2019][1697195.083209] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:20 2019][1697195.089400] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697195.094544] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:20 2019][1697195.100302] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:20 2019][1697195.107007] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697195.112072] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697195.118345] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697195.124967] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697195.131240] kworker/5:2 S ffffa06b7974e180 0 95447 2 0x00000080 [Wed Feb 6 09:40:20 2019][1697195.138520] Call Trace: [Wed Feb 6 09:40:20 2019][1697195.141155] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:20 2019][1697195.147345] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1697195.152492] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:20 2019][1697195.158249] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:20 2019][1697195.164954] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1697195.170017] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1697195.176291] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1697195.182913] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.189187] kworker/45:0 S ffffa06b03cb30c0 0 103954 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697195.196561] Call Trace: [Wed Feb 6 09:40:21 2019][1697195.199196] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:21 2019][1697195.205385] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697195.210533] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:21 2019][1697195.216309] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:21 2019][1697195.223014] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697195.228077] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.234349] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697195.240973] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.247245] kworker/24:0 R running task 0 108674 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697195.254613] Call Trace: [Wed Feb 6 09:40:21 2019][1697195.257245] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:21 2019][1697195.263436] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697195.268583] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:21 2019][1697195.274341] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:21 2019][1697195.281045] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697195.286110] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.292382] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697195.299005] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.305279] kworker/13:2 S ffffa06a03efd140 0 110381 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697195.312645] Call Trace: [Wed Feb 6 09:40:21 2019][1697195.315278] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:21 2019][1697195.321483] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697195.326632] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:21 2019][1697195.332390] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:21 2019][1697195.339094] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697195.344158] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.350432] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697195.357054] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.363329] kworker/37:0 S ffffa0695eb9e180 0 111915 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697195.370694] Call Trace: [Wed Feb 6 09:40:21 2019][1697195.373330] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:21 2019][1697195.379519] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697195.384667] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:21 2019][1697195.390421] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:21 2019][1697195.397128] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697195.402193] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.408467] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697195.415088] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.421360] kworker/13:0 S ffffa06b7c6ce180 0 113688 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697195.428727] Call Trace: [Wed Feb 6 09:40:21 2019][1697195.431386] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697195.436530] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:21 2019][1697195.442282] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:21 2019][1697195.448992] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697195.454051] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.460329] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697195.466948] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.473223] kworker/29:1 S ffffa06b786bd140 0 115868 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697195.480589] Call Trace: [Wed Feb 6 09:40:21 2019][1697195.483225] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697195.488374] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:21 2019][1697195.494125] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:21 2019][1697195.500837] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697195.505895] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.512172] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697195.518793] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.525066] kworker/35:1 S ffffa0899ea20000 0 120937 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697195.532433] Call Trace: [Wed Feb 6 09:40:21 2019][1697195.535069] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:21 2019][1697195.541259] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697195.546430] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:21 2019][1697195.552187] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:21 2019][1697195.558892] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697195.563957] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.570231] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697195.576853] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.583126] kworker/24:2 S ffffa05b7c10c100 0 123849 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697195.590492] Call Trace: [Wed Feb 6 09:40:21 2019][1697195.593129] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697195.598278] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:21 2019][1697195.604082] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:21 2019][1697195.610791] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697195.615853] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.622127] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697195.628749] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.635021] kworker/20:1 S ffffa0521450c100 0 125816 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697195.642388] Call Trace: [Wed Feb 6 09:40:21 2019][1697195.645023] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:21 2019][1697195.651214] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697195.656360] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:21 2019][1697195.662139] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:21 2019][1697195.668840] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697195.673904] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.680176] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697195.686799] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.693071] kworker/1:0 S ffffa06b7b962080 0 4055 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697195.700362] Call Trace: [Wed Feb 6 09:40:21 2019][1697195.702995] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:21 2019][1697195.709186] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697195.714331] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:21 2019][1697195.720091] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:21 2019][1697195.726820] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697195.731885] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.738157] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697195.744781] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.751052] kworker/32:1 S ffffa05b6e316180 0 7231 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697195.758359] Call Trace: [Wed Feb 6 09:40:21 2019][1697195.760996] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:21 2019][1697195.767185] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697195.772331] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:21 2019][1697195.778089] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:21 2019][1697195.784818] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697195.789881] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.796157] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697195.802779] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.809053] kworker/9:0 S ffffa069597c9040 0 7406 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697195.816340] Call Trace: [Wed Feb 6 09:40:21 2019][1697195.818975] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:21 2019][1697195.825164] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697195.830311] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:21 2019][1697195.836067] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:21 2019][1697195.842773] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697195.847836] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.854109] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697195.860733] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.867030] kworker/41:0 S ffffa069597ca080 0 7616 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697195.874303] Call Trace: [Wed Feb 6 09:40:21 2019][1697195.876938] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:21 2019][1697195.883129] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697195.888275] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:21 2019][1697195.894032] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:21 2019][1697195.900760] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697195.905818] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.912090] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697195.918713] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697195.924988] python S ffffa06b04662080 0 8865 1 0x00000080 [Wed Feb 6 09:40:21 2019][1697195.932274] Call Trace: [Wed Feb 6 09:40:21 2019][1697195.934905] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:40:21 2019][1697195.941634] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697195.946783] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:40:21 2019][1697195.953930] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:40:21 2019][1697195.959855] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:40:21 2019][1697195.967170] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:21 2019][1697195.973704] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:40:21 2019][1697195.978940] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1697195.984694] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:40:21 2019][1697195.990451] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:21 2019][1697195.996636] kworker/6:1 S ffffa07b4eb4a080 0 10262 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697196.003924] Call Trace: [Wed Feb 6 09:40:21 2019][1697196.006559] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697196.011731] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:21 2019][1697196.017489] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:21 2019][1697196.024197] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697196.029257] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697196.035533] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697196.042153] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697196.048427] kworker/u264:0 S ffffa07b76479040 0 10636 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697196.055706] Call Trace: [Wed Feb 6 09:40:21 2019][1697196.058361] [] ? ib_cq_poll_work+0x48/0x70 [ib_core] [Wed Feb 6 09:40:21 2019][1697196.065159] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:21 2019][1697196.071366] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697196.076509] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:21 2019][1697196.082264] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:21 2019][1697196.088993] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697196.094049] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697196.100326] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697196.106947] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697196.113221] ldlm_bl_05 S ffffa07b7647b0c0 0 10646 2 0x00000080 [Wed Feb 6 09:40:21 2019][1697196.120496] Call Trace: [Wed Feb 6 09:40:21 2019][1697196.123128] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:21 2019][1697196.128999] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:21 2019][1697196.136396] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1697196.141602] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:21 2019][1697196.148657] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1697196.154440] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:21 2019][1697196.162016] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1697196.167075] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697196.173354] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1697196.179971] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1697196.186249] ldlm_bl_06 S ffffa07b7647c100 0 10647 2 0x00000080 [Wed Feb 6 09:40:22 2019][1697196.193541] Call Trace: [Wed Feb 6 09:40:22 2019][1697196.196173] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:22 2019][1697196.202034] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:22 2019][1697196.209431] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1697196.214611] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.221681] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1697196.227464] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.235043] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1697196.240102] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.246380] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1697196.253022] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.259314] ldlm_bl_07 S ffffa07b7647d140 0 10648 2 0x00000080 [Wed Feb 6 09:40:22 2019][1697196.266586] Call Trace: [Wed Feb 6 09:40:22 2019][1697196.269215] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:22 2019][1697196.275076] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:22 2019][1697196.282475] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1697196.287651] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.294706] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1697196.300490] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.308058] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1697196.313122] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.319417] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1697196.326036] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.332306] ldlm_bl_08 R running task 0 10649 2 0x00000080 [Wed Feb 6 09:40:22 2019][1697196.339586] Call Trace: [Wed Feb 6 09:40:22 2019][1697196.342215] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 09:40:22 2019][1697196.348085] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.354992] [] ? ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.363633] [] ? ldlm_cancel_locks_for_export_cb+0x3f/0x50 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.371732] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [Wed Feb 6 09:40:22 2019][1697196.379336] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.388006] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.396645] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [Wed Feb 6 09:40:22 2019][1697196.404164] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.411771] [] ? ldlm_bl_thread_main+0x4c8/0x700 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.418994] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1697196.424777] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.432350] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1697196.437588] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.443859] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1697196.450656] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.456953] ldlm_bl_09 R running task 0 10650 2 0x00000080 [Wed Feb 6 09:40:22 2019][1697196.464239] Call Trace: [Wed Feb 6 09:40:22 2019][1697196.466871] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 09:40:22 2019][1697196.472755] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.479667] [] ? ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.488303] [] ? ldlm_cancel_locks_for_export_cb+0x3f/0x50 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.496403] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [Wed Feb 6 09:40:22 2019][1697196.504008] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.512652] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.521271] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [Wed Feb 6 09:40:22 2019][1697196.528831] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.536437] [] ? ldlm_bl_thread_main+0x4c8/0x700 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.543666] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1697196.549478] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.557048] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1697196.562281] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.568557] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1697196.575352] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.581625] ldlm_bl_10 S ffffa0516a6130c0 0 10651 2 0x00000080 [Wed Feb 6 09:40:22 2019][1697196.588940] Call Trace: [Wed Feb 6 09:40:22 2019][1697196.591574] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:22 2019][1697196.597437] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:22 2019][1697196.604835] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1697196.610015] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.617064] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1697196.622849] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.630419] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1697196.635483] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.641757] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1697196.648379] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.654675] ldlm_bl_11 S ffffa0516a615140 0 10652 2 0x00000080 [Wed Feb 6 09:40:22 2019][1697196.661944] Call Trace: [Wed Feb 6 09:40:22 2019][1697196.664577] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:22 2019][1697196.670436] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:22 2019][1697196.677836] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1697196.683014] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.690065] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1697196.695850] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.703419] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1697196.708483] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.714757] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1697196.721379] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.727653] ldlm_bl_12 S ffffa06b17271040 0 10653 2 0x00000080 [Wed Feb 6 09:40:22 2019][1697196.734928] Call Trace: [Wed Feb 6 09:40:22 2019][1697196.737559] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:22 2019][1697196.743421] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:22 2019][1697196.750820] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1697196.755996] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.763048] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1697196.768831] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.776427] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1697196.781491] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.787766] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1697196.794388] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.800661] ldlm_bl_13 S ffffa06b17272080 0 10654 2 0x00000080 [Wed Feb 6 09:40:22 2019][1697196.807946] Call Trace: [Wed Feb 6 09:40:22 2019][1697196.810575] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:22 2019][1697196.816436] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:22 2019][1697196.823836] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1697196.829016] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.836088] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1697196.841866] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.849438] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1697196.854500] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.860774] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1697196.867396] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.873669] ldlm_bl_14 S ffffa06b17276180 0 10655 2 0x00000080 [Wed Feb 6 09:40:22 2019][1697196.880945] Call Trace: [Wed Feb 6 09:40:22 2019][1697196.883577] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:22 2019][1697196.889438] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:22 2019][1697196.896861] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1697196.902042] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.909092] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1697196.914874] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.922447] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1697196.927508] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.933782] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1697196.940405] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697196.946679] ldlm_bl_15 S ffffa05b6e3130c0 0 10656 2 0x00000080 [Wed Feb 6 09:40:22 2019][1697196.953954] Call Trace: [Wed Feb 6 09:40:22 2019][1697196.956584] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:22 2019][1697196.962466] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:22 2019][1697196.969864] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1697196.975042] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.982093] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1697196.987876] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697196.995447] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1697197.000509] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697197.006784] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1697197.013406] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697197.019702] ldlm_bl_16 S ffffa05b6e315140 0 10657 2 0x00000080 [Wed Feb 6 09:40:22 2019][1697197.026971] Call Trace: [Wed Feb 6 09:40:22 2019][1697197.029603] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:22 2019][1697197.035462] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:22 2019][1697197.042863] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1697197.048043] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697197.055092] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1697197.060875] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697197.068449] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1697197.073509] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697197.079785] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1697197.086406] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697197.092679] ldlm_bl_17 S ffffa05b6e314100 0 10658 2 0x00000080 [Wed Feb 6 09:40:22 2019][1697197.099954] Call Trace: [Wed Feb 6 09:40:22 2019][1697197.102584] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:22 2019][1697197.108443] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:22 2019][1697197.115850] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1697197.121026] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697197.128075] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1697197.133859] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:22 2019][1697197.141453] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1697197.146510] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697197.152785] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1697197.159406] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1697197.165679] kworker/46:1 S ffffa072fc09d140 0 10785 2 0x00000080 [Wed Feb 6 09:40:22 2019][1697197.172960] Call Trace: [Wed Feb 6 09:40:22 2019][1697197.175596] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:22 2019][1697197.181783] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1697197.186929] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:23 2019][1697197.192686] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:23 2019][1697197.199394] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697197.204455] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.210731] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1697197.217351] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.223648] kworker/u264:2 S ffffa07b76b4c100 0 10936 2 0x00000080 [Wed Feb 6 09:40:23 2019][1697197.230924] Call Trace: [Wed Feb 6 09:40:23 2019][1697197.233574] [] ? ib_cq_poll_work+0x48/0x70 [ib_core] [Wed Feb 6 09:40:23 2019][1697197.240371] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:23 2019][1697197.246560] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1697197.251729] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:23 2019][1697197.257476] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:23 2019][1697197.264188] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697197.269247] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.275527] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1697197.282144] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.288419] mdt_fld_0002 S ffffa05b78990000 0 11286 2 0x00000080 [Wed Feb 6 09:40:23 2019][1697197.295692] Call Trace: [Wed Feb 6 09:40:23 2019][1697197.298324] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1697197.303508] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.310441] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1697197.316233] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.322713] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.330288] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697197.335351] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.341625] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1697197.348247] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.354520] mdt_fld_0003 S ffffa06b789b9040 0 11594 2 0x00000080 [Wed Feb 6 09:40:23 2019][1697197.361804] Call Trace: [Wed Feb 6 09:40:23 2019][1697197.364434] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1697197.369618] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.376498] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1697197.382289] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.388787] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.396363] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697197.401426] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.407699] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1697197.414322] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.420599] mdt_io03_010 S ffffa05b7c252080 0 12793 2 0x00000080 [Wed Feb 6 09:40:23 2019][1697197.427871] Call Trace: [Wed Feb 6 09:40:23 2019][1697197.430503] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1697197.435685] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.442566] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1697197.448354] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.454865] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.462439] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697197.467501] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.473798] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1697197.480413] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.486689] mdt_rdpg03_003 S ffffa05b79648000 0 12913 2 0x00000080 [Wed Feb 6 09:40:23 2019][1697197.493983] Call Trace: [Wed Feb 6 09:40:23 2019][1697197.496612] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1697197.501794] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.508680] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1697197.514465] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.520950] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.528524] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697197.533585] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.539861] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1697197.546483] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.552757] ldlm_cn03_009 S ffffa05b7964a080 0 12924 2 0x00000080 [Wed Feb 6 09:40:23 2019][1697197.560057] Call Trace: [Wed Feb 6 09:40:23 2019][1697197.562687] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1697197.567869] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.574751] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1697197.580540] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.587024] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.594622] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697197.599677] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.605953] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1697197.612602] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.618874] ldlm_cn03_010 S ffffa05b7964c100 0 12925 2 0x00000080 [Wed Feb 6 09:40:23 2019][1697197.626149] Call Trace: [Wed Feb 6 09:40:23 2019][1697197.628780] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1697197.633962] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.640848] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1697197.646631] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.653115] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.660694] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697197.665752] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.672030] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1697197.678649] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.684944] ldlm_cn02_011 S ffffa05b7964e180 0 12935 2 0x00000080 [Wed Feb 6 09:40:23 2019][1697197.692226] Call Trace: [Wed Feb 6 09:40:23 2019][1697197.694857] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1697197.700037] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.706921] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1697197.712707] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.719190] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.726759] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697197.731843] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.738115] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1697197.744732] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.751006] ldlm_cn03_011 S ffffa05b72d4e180 0 12936 2 0x00000080 [Wed Feb 6 09:40:23 2019][1697197.758274] Call Trace: [Wed Feb 6 09:40:23 2019][1697197.760907] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1697197.766087] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.772971] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1697197.778756] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.785236] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.792809] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697197.797870] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.804145] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1697197.810765] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.817040] ldlm_cn03_012 S ffffa05b72d49040 0 12937 2 0x00000080 [Wed Feb 6 09:40:23 2019][1697197.824316] Call Trace: [Wed Feb 6 09:40:23 2019][1697197.826945] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1697197.832127] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.839011] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1697197.844797] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.851276] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.858850] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697197.863911] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.870186] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1697197.876807] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.883081] ldlm_cn02_012 S ffffa05b753c5140 0 12966 2 0x00000080 [Wed Feb 6 09:40:23 2019][1697197.890355] Call Trace: [Wed Feb 6 09:40:23 2019][1697197.892988] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1697197.898166] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.905045] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1697197.910832] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.917318] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.924890] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697197.929952] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.936227] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1697197.942848] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697197.949123] ldlm_cn02_013 S ffffa0516a611040 0 13011 2 0x00000080 [Wed Feb 6 09:40:23 2019][1697197.956398] Call Trace: [Wed Feb 6 09:40:23 2019][1697197.959030] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1697197.964208] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.971108] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1697197.976897] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.983380] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697197.990957] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697197.996016] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697198.002293] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1697198.008915] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697198.015188] ldlm_cn03_013 S ffffa05b753c2080 0 13042 2 0x00000080 [Wed Feb 6 09:40:23 2019][1697198.022471] Call Trace: [Wed Feb 6 09:40:23 2019][1697198.025103] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1697198.030283] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697198.037160] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1697198.042944] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697198.049422] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697198.056999] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697198.062058] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697198.068333] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1697198.074957] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697198.081231] kworker/25:1 S ffffa06b7c0ae180 0 13104 2 0x00000080 [Wed Feb 6 09:40:23 2019][1697198.088510] Call Trace: [Wed Feb 6 09:40:23 2019][1697198.091146] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:23 2019][1697198.097332] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1697198.102482] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:23 2019][1697198.108237] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:23 2019][1697198.114942] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697198.120007] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697198.126280] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1697198.132903] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1697198.139175] mdt_rdpg03_004 S ffffa04ca96c0000 0 13294 2 0x00000080 [Wed Feb 6 09:40:23 2019][1697198.146449] Call Trace: [Wed Feb 6 09:40:23 2019][1697198.149083] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1697198.154264] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697198.161146] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1697198.166933] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697198.173411] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1697198.180985] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1697198.186070] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.192349] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1697198.198966] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.205242] ldlm_cn02_014 S ffffa08a83a51040 0 13387 2 0x00000080 [Wed Feb 6 09:40:24 2019][1697198.212535] Call Trace: [Wed Feb 6 09:40:24 2019][1697198.215165] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1697198.220347] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.227232] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1697198.233017] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.239495] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.247069] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1697198.252128] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.258406] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1697198.265028] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.271300] ldlm_cn01_016 S ffffa0695eb98000 0 13458 2 0x00000080 [Wed Feb 6 09:40:24 2019][1697198.278575] Call Trace: [Wed Feb 6 09:40:24 2019][1697198.281207] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1697198.286388] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.293271] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1697198.299057] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.305537] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.313111] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1697198.318171] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.324446] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1697198.331089] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.337357] ldlm_cn03_014 S ffffa0695eb9d140 0 13459 2 0x00000080 [Wed Feb 6 09:40:24 2019][1697198.344634] Call Trace: [Wed Feb 6 09:40:24 2019][1697198.347265] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1697198.352446] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.359330] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1697198.365136] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.371619] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.379208] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1697198.384264] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.390541] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1697198.397159] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.403435] ldlm_cn03_015 S ffffa0695eb9a080 0 13460 2 0x00000080 [Wed Feb 6 09:40:24 2019][1697198.410709] Call Trace: [Wed Feb 6 09:40:24 2019][1697198.413366] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1697198.418546] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.425422] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1697198.431211] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.437697] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.445270] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1697198.450356] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.456653] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1697198.463270] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.469545] ldlm_cn02_015 S ffffa07b7922e180 0 14199 2 0x00000080 [Wed Feb 6 09:40:24 2019][1697198.476819] Call Trace: [Wed Feb 6 09:40:24 2019][1697198.479453] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1697198.484638] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.491541] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1697198.497326] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.503805] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.511401] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1697198.516458] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.522736] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1697198.529355] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.535629] ldlm_cn03_016 S ffffa07b7922a080 0 14200 2 0x00000080 [Wed Feb 6 09:40:24 2019][1697198.542905] Call Trace: [Wed Feb 6 09:40:24 2019][1697198.545536] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1697198.550716] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.557599] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1697198.563381] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.569888] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.577464] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1697198.582524] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.588800] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1697198.595421] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.601696] ldlm_cn01_017 S ffffa07b79228000 0 14201 2 0x00000080 [Wed Feb 6 09:40:24 2019][1697198.608971] Call Trace: [Wed Feb 6 09:40:24 2019][1697198.611607] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1697198.616781] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.623659] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1697198.629443] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.635922] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.643496] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1697198.648557] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.654834] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1697198.661453] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.667727] kworker/43:2 S ffffa08b76075140 0 16827 2 0x00000080 [Wed Feb 6 09:40:24 2019][1697198.675007] Call Trace: [Wed Feb 6 09:40:24 2019][1697198.677642] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:24 2019][1697198.683831] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1697198.689003] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:24 2019][1697198.694784] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:24 2019][1697198.701495] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1697198.706555] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.712830] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1697198.719452] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.725728] kworker/u257:1 S ffffa0516a610000 0 17401 2 0x00000080 [Wed Feb 6 09:40:24 2019][1697198.733004] Call Trace: [Wed Feb 6 09:40:24 2019][1697198.735641] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:24 2019][1697198.741831] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1697198.747001] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:24 2019][1697198.752779] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:24 2019][1697198.759481] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1697198.764547] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.770818] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1697198.777441] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.783715] mdt03_014 S ffffa07b7922c100 0 17804 2 0x00000080 [Wed Feb 6 09:40:24 2019][1697198.790992] Call Trace: [Wed Feb 6 09:40:24 2019][1697198.793620] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1697198.798801] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.805699] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1697198.811484] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.817967] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.825542] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1697198.830602] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.836880] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1697198.843500] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.849773] mdt03_015 S ffffa07b7922d140 0 17805 2 0x00000080 [Wed Feb 6 09:40:24 2019][1697198.857049] Call Trace: [Wed Feb 6 09:40:24 2019][1697198.859679] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1697198.864859] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.871736] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1697198.877522] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.884000] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.891574] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1697198.896635] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.902911] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1697198.909533] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.915835] mdt03_016 S ffffa07b75efd140 0 17806 2 0x00000080 [Wed Feb 6 09:40:24 2019][1697198.923116] Call Trace: [Wed Feb 6 09:40:24 2019][1697198.925748] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1697198.930947] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.937826] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1697198.943616] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.950092] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697198.957692] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1697198.962753] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.969028] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1697198.975650] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697198.981925] mdt03_017 S ffffa07b75a55140 0 17807 2 0x00000080 [Wed Feb 6 09:40:24 2019][1697198.989199] Call Trace: [Wed Feb 6 09:40:24 2019][1697198.991855] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1697198.997036] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697199.003911] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1697199.009697] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697199.016176] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697199.023751] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1697199.028811] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697199.035087] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1697199.041708] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697199.047982] mdt03_018 S ffffa04ead3a9040 0 17810 2 0x00000080 [Wed Feb 6 09:40:24 2019][1697199.055258] Call Trace: [Wed Feb 6 09:40:24 2019][1697199.057889] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1697199.063068] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697199.069944] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1697199.075732] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697199.082209] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697199.089785] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1697199.094868] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697199.101146] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1697199.107790] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697199.114067] mdt03_019 S ffffa04ead3ae180 0 17811 2 0x00000080 [Wed Feb 6 09:40:24 2019][1697199.121349] Call Trace: [Wed Feb 6 09:40:24 2019][1697199.123982] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1697199.129163] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697199.136046] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1697199.141830] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697199.148309] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1697199.155885] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1697199.160947] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1697199.167221] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697199.173843] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.180115] mdt01_037 S ffffa04ead3ad140 0 17815 2 0x00000080 [Wed Feb 6 09:40:25 2019][1697199.187390] Call Trace: [Wed Feb 6 09:40:25 2019][1697199.190022] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1697199.195203] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.202077] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1697199.207865] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.214342] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.221919] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1697199.227000] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.233272] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697199.239891] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.246164] mdt03_020 S ffffa04ead3ac100 0 17822 2 0x00000080 [Wed Feb 6 09:40:25 2019][1697199.253432] Call Trace: [Wed Feb 6 09:40:25 2019][1697199.256064] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1697199.261243] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.268119] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1697199.273904] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.280382] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.287958] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1697199.293019] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.299294] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697199.305916] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.312188] mdt01_038 S ffffa04ead3aa080 0 17823 2 0x00000080 [Wed Feb 6 09:40:25 2019][1697199.319465] Call Trace: [Wed Feb 6 09:40:25 2019][1697199.322122] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1697199.327302] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.334177] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1697199.339963] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.346466] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.354045] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1697199.359103] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.365379] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697199.371999] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.378274] mdt03_021 S ffffa08b7ca9b0c0 0 17827 2 0x00000080 [Wed Feb 6 09:40:25 2019][1697199.385548] Call Trace: [Wed Feb 6 09:40:25 2019][1697199.388180] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1697199.393359] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.400235] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1697199.406022] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.412506] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.420085] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1697199.425142] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.431420] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697199.438039] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.444315] mdt03_022 S ffffa08b7ca9c100 0 17828 2 0x00000080 [Wed Feb 6 09:40:25 2019][1697199.451591] Call Trace: [Wed Feb 6 09:40:25 2019][1697199.454222] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1697199.459400] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.466298] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1697199.472079] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.478557] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.486132] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1697199.491192] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.497469] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697199.504112] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.510383] mdt03_023 S ffffa08b7ca98000 0 17829 2 0x00000080 [Wed Feb 6 09:40:25 2019][1697199.517656] Call Trace: [Wed Feb 6 09:40:25 2019][1697199.520286] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1697199.525468] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.532346] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1697199.538127] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.544636] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.552215] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1697199.557277] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.563553] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697199.570173] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.576449] mdt02_027 S ffffa07b75a54100 0 17830 2 0x00000080 [Wed Feb 6 09:40:25 2019][1697199.583724] Call Trace: [Wed Feb 6 09:40:25 2019][1697199.586354] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1697199.591533] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.598409] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1697199.604197] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.610684] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.618259] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1697199.623318] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.629596] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697199.636215] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.642488] mdt01_039 S ffffa07b75a56180 0 17831 2 0x00000080 [Wed Feb 6 09:40:25 2019][1697199.649808] Call Trace: [Wed Feb 6 09:40:25 2019][1697199.652437] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1697199.657621] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.664501] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1697199.670293] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.676774] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.684349] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1697199.689413] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.695685] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697199.702308] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.708604] mdt01_040 S ffffa07b75a52080 0 17832 2 0x00000080 [Wed Feb 6 09:40:25 2019][1697199.715881] Call Trace: [Wed Feb 6 09:40:25 2019][1697199.718513] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1697199.723694] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.730570] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1697199.736355] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.742840] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.750417] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1697199.755478] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.761755] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697199.768400] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.774675] mdt02_028 S ffffa07b75a530c0 0 17834 2 0x00000080 [Wed Feb 6 09:40:25 2019][1697199.781950] Call Trace: [Wed Feb 6 09:40:25 2019][1697199.784605] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1697199.789786] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.796664] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:40:25 2019][1697199.803631] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1697199.809421] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.815919] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.823494] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1697199.828556] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.834830] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697199.841455] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.847725] mdt02_029 S ffffa05b785f8000 0 17838 2 0x00000080 [Wed Feb 6 09:40:25 2019][1697199.855011] Call Trace: [Wed Feb 6 09:40:25 2019][1697199.857643] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1697199.862823] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.869705] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1697199.875491] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.881974] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.889545] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1697199.894606] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.900887] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697199.907511] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.913784] mdt02_030 S ffffa05b785f9040 0 17839 2 0x00000080 [Wed Feb 6 09:40:25 2019][1697199.921059] Call Trace: [Wed Feb 6 09:40:25 2019][1697199.923716] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1697199.928898] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.935774] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1697199.941559] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.948061] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697199.955637] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1697199.960698] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.966974] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697199.973595] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697199.979870] mdt02_031 S ffffa05b785fc100 0 17840 2 0x00000080 [Wed Feb 6 09:40:25 2019][1697199.987151] Call Trace: [Wed Feb 6 09:40:25 2019][1697199.989783] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1697199.994964] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697200.001839] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1697200.007626] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697200.014106] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697200.021678] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1697200.026759] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697200.033031] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697200.039653] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697200.045928] mdt03_024 S ffffa072fc09a080 0 17848 2 0x00000080 [Wed Feb 6 09:40:25 2019][1697200.053203] Call Trace: [Wed Feb 6 09:40:25 2019][1697200.055832] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1697200.061031] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697200.067930] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1697200.073708] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697200.080188] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697200.087764] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1697200.092824] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697200.099099] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697200.105720] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697200.111994] mdt02_032 S ffffa072fc09e180 0 17849 2 0x00000080 [Wed Feb 6 09:40:25 2019][1697200.119278] Call Trace: [Wed Feb 6 09:40:25 2019][1697200.121911] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1697200.127090] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697200.133975] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1697200.139760] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697200.146244] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1697200.153820] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1697200.158881] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1697200.165159] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1697200.171778] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.178054] mdt03_025 S ffffa072fc09b0c0 0 17850 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697200.185328] Call Trace: [Wed Feb 6 09:40:26 2019][1697200.187958] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1697200.193139] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.200014] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1697200.205800] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.212280] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.219853] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1697200.224914] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.231191] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1697200.237811] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.244085] mdt01_041 S ffffa072fc098000 0 17851 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697200.251377] Call Trace: [Wed Feb 6 09:40:26 2019][1697200.254008] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1697200.259188] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.266065] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1697200.271849] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.278333] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.285904] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1697200.290963] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.297239] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1697200.303861] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.310155] mdt03_026 S ffffa072fc099040 0 17853 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697200.317425] Call Trace: [Wed Feb 6 09:40:26 2019][1697200.320056] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1697200.325238] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.332115] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1697200.337898] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.344377] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.351951] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1697200.357012] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.363291] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1697200.369931] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.376201] mdt03_027 S ffffa05b7c6e0000 0 17854 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697200.383475] Call Trace: [Wed Feb 6 09:40:26 2019][1697200.386107] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1697200.391288] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.398169] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1697200.403957] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.410443] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.418020] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1697200.423079] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.429356] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1697200.435974] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.442249] mdt02_033 S ffffa05b7c251040 0 17855 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697200.449525] Call Trace: [Wed Feb 6 09:40:26 2019][1697200.452156] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1697200.457337] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.464213] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1697200.469998] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.476477] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.484051] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1697200.489136] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.495406] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1697200.502024] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.508300] mdt02_034 S ffffa06c29ab0000 0 17859 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697200.515576] Call Trace: [Wed Feb 6 09:40:26 2019][1697200.518204] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1697200.523386] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.530264] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1697200.536049] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.542531] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.550100] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1697200.555162] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.561458] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1697200.568076] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.574348] mdt00_027 S ffffa06c29ab6180 0 17860 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697200.581624] Call Trace: [Wed Feb 6 09:40:26 2019][1697200.584257] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1697200.589437] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.596319] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1697200.602106] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.608611] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.616183] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1697200.621270] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.627549] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1697200.634168] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.640441] mdt00_028 S ffffa06b7b961040 0 17870 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697200.647718] Call Trace: [Wed Feb 6 09:40:26 2019][1697200.650349] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1697200.655527] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.662405] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1697200.668190] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.674675] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.682251] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1697200.687311] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.693588] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1697200.700208] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.706484] mdt00_029 S ffffa06b7b960000 0 17875 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697200.713765] Call Trace: [Wed Feb 6 09:40:26 2019][1697200.716401] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1697200.721578] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.728479] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1697200.734264] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.740744] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.748319] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1697200.753379] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.759656] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1697200.766275] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.772551] mdt00_030 S ffffa072fc0aa080 0 17877 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697200.779825] Call Trace: [Wed Feb 6 09:40:26 2019][1697200.782458] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1697200.787640] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.794517] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1697200.800302] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.806781] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.814349] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1697200.819412] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.825686] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1697200.832309] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.838581] mdt00_031 S ffffa072fc0ae180 0 17885 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697200.845866] Call Trace: [Wed Feb 6 09:40:26 2019][1697200.848498] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1697200.853678] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.860560] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1697200.866348] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.872825] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.880399] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1697200.885460] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.891736] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1697200.898355] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.904634] mdt00_032 S ffffa072fc0a9040 0 17889 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697200.911925] Call Trace: [Wed Feb 6 09:40:26 2019][1697200.914554] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1697200.919737] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.926625] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1697200.932406] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.938889] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.946460] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1697200.951519] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.957796] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1697200.964416] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697200.970692] mdt03_028 S ffffa072fc0ab0c0 0 17890 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697200.977965] Call Trace: [Wed Feb 6 09:40:26 2019][1697200.980595] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1697200.985775] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697200.992652] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1697200.998438] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697201.004941] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697201.012518] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1697201.017578] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697201.023853] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1697201.030496] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697201.036765] mdt00_033 S ffffa072fc0ac100 0 17893 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697201.044042] Call Trace: [Wed Feb 6 09:40:26 2019][1697201.046672] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1697201.051870] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697201.058752] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1697201.064538] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697201.071024] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697201.078601] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1697201.083661] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697201.089936] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1697201.096557] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697201.102834] mdt00_034 S ffffa072fc0ad140 0 17894 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697201.110107] Call Trace: [Wed Feb 6 09:40:26 2019][1697201.112738] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1697201.117918] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697201.124794] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1697201.130580] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697201.137059] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1697201.144634] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1697201.149715] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697201.155986] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1697201.162608] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1697201.168882] mdt00_035 S ffffa07b7b2bc100 0 17897 2 0x00000080 [Wed Feb 6 09:40:26 2019][1697201.176156] Call Trace: [Wed Feb 6 09:40:27 2019][1697201.178824] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.186573] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1697201.191750] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.198830] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.205935] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:27 2019][1697201.213074] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1697201.218861] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.226209] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.234246] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.242014] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Feb 6 09:40:27 2019][1697201.248932] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.256707] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.264723] [] mdt_reint_unlink+0x331/0x14a0 [mdt] [Wed Feb 6 09:40:27 2019][1697201.271354] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 09:40:27 2019][1697201.277551] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 09:40:27 2019][1697201.284268] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Wed Feb 6 09:40:27 2019][1697201.291242] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 09:40:27 2019][1697201.297124] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.304219] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.311973] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:27 2019][1697201.319234] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.327101] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.334070] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:27 2019][1697201.340523] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:27 2019][1697201.346569] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.353049] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.360627] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1697201.365689] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.371987] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1697201.378610] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.384883] mdt00_036 S ffffa07b7a272080 0 17900 2 0x00000080 [Wed Feb 6 09:40:27 2019][1697201.392158] Call Trace: [Wed Feb 6 09:40:27 2019][1697201.394790] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1697201.399970] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.406846] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1697201.412634] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.419110] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.426684] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1697201.431748] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.438022] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1697201.444644] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.450914] mdt03_029 S ffffa069597cc100 0 17903 2 0x00000080 [Wed Feb 6 09:40:27 2019][1697201.458190] Call Trace: [Wed Feb 6 09:40:27 2019][1697201.460823] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1697201.466003] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.472900] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1697201.478683] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.485166] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.492743] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1697201.497801] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.504070] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1697201.510690] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.516966] mdt00_037 S ffffa069597c8000 0 17904 2 0x00000080 [Wed Feb 6 09:40:27 2019][1697201.524239] Call Trace: [Wed Feb 6 09:40:27 2019][1697201.526871] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1697201.532052] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.538927] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1697201.544713] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.551189] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.558765] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1697201.563825] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.570103] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1697201.576723] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.582997] mdt03_030 S ffffa069597ce180 0 17905 2 0x00000080 [Wed Feb 6 09:40:27 2019][1697201.590288] Call Trace: [Wed Feb 6 09:40:27 2019][1697201.592921] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1697201.598101] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.604976] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1697201.610763] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.617241] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.624817] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1697201.629874] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.636153] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1697201.642773] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.649047] mdt02_035 S ffffa07b7778b0c0 0 17906 2 0x00000080 [Wed Feb 6 09:40:27 2019][1697201.656340] Call Trace: [Wed Feb 6 09:40:27 2019][1697201.658970] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1697201.664153] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.671035] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1697201.676821] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.683300] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.690874] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1697201.695935] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.702211] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1697201.708829] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.715108] mdt03_031 S ffffa07b7778a080 0 17907 2 0x00000080 [Wed Feb 6 09:40:27 2019][1697201.722381] Call Trace: [Wed Feb 6 09:40:27 2019][1697201.725015] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1697201.730192] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.737070] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1697201.742853] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.749337] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.756905] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1697201.761969] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.768242] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1697201.774887] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.781154] mdt02_036 S ffffa07b78141040 0 17908 2 0x00000080 [Wed Feb 6 09:40:27 2019][1697201.788431] Call Trace: [Wed Feb 6 09:40:27 2019][1697201.791062] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1697201.796241] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.803118] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1697201.808903] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.815380] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.822956] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1697201.828017] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.834292] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1697201.840938] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.847214] mdt02_037 S ffffa07b3ca6c100 0 17909 2 0x00000080 [Wed Feb 6 09:40:27 2019][1697201.854490] Call Trace: [Wed Feb 6 09:40:27 2019][1697201.857119] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1697201.862300] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.869176] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1697201.874962] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.881464] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.889042] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1697201.894124] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.900394] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1697201.907014] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.913289] mdt00_038 S ffffa07b3ca69040 0 17910 2 0x00000080 [Wed Feb 6 09:40:27 2019][1697201.920573] Call Trace: [Wed Feb 6 09:40:27 2019][1697201.923203] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1697201.928382] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.935260] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1697201.941047] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.947531] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697201.955106] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1697201.960168] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.966445] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1697201.973063] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697201.979342] mdt00_039 S ffffa07b74acb0c0 0 17911 2 0x00000080 [Wed Feb 6 09:40:27 2019][1697201.986612] Call Trace: [Wed Feb 6 09:40:27 2019][1697201.989245] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1697201.994424] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697202.001301] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1697202.007088] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697202.013598] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697202.021174] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1697202.026233] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697202.032512] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1697202.039131] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697202.045405] mdt00_040 S ffffa06c29950000 0 17915 2 0x00000080 [Wed Feb 6 09:40:27 2019][1697202.052688] Call Trace: [Wed Feb 6 09:40:27 2019][1697202.055321] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1697202.060499] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697202.067376] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1697202.073162] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697202.079641] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697202.087212] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1697202.092266] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697202.098543] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1697202.105162] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697202.111438] mdt00_041 S ffffa07b741b0000 0 17916 2 0x00000080 [Wed Feb 6 09:40:27 2019][1697202.118713] Call Trace: [Wed Feb 6 09:40:27 2019][1697202.121344] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1697202.126524] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697202.133434] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1697202.139220] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697202.145698] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1697202.153273] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1697202.158334] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697202.164609] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1697202.171231] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1697202.177504] mdt00_042 S ffffa07b741b30c0 0 17919 2 0x00000080 [Wed Feb 6 09:40:28 2019][1697202.184789] Call Trace: [Wed Feb 6 09:40:28 2019][1697202.187420] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1697202.192600] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.199483] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1697202.205269] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.211772] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.219349] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1697202.224410] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.230686] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1697202.237307] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.243602] mdt03_032 S ffffa07b741b1040 0 17925 2 0x00000080 [Wed Feb 6 09:40:28 2019][1697202.250873] Call Trace: [Wed Feb 6 09:40:28 2019][1697202.253503] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1697202.258683] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.265559] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1697202.271360] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.277840] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.285415] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1697202.290476] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.296753] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1697202.303373] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.309647] mdt02_038 S ffffa07b741b4100 0 17930 2 0x00000080 [Wed Feb 6 09:40:28 2019][1697202.316938] Call Trace: [Wed Feb 6 09:40:28 2019][1697202.319571] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1697202.324753] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.331636] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1697202.337420] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.343903] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.351473] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1697202.356532] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.362810] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1697202.369453] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.375724] mdt00_043 S ffffa07b741b2080 0 17931 2 0x00000080 [Wed Feb 6 09:40:28 2019][1697202.383011] Call Trace: [Wed Feb 6 09:40:28 2019][1697202.385647] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1697202.390825] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.397703] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1697202.403487] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.409973] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.417550] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1697202.422607] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.428886] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1697202.435504] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.441780] mdt00_044 S ffffa07b775cb0c0 0 17932 2 0x00000080 [Wed Feb 6 09:40:28 2019][1697202.449063] Call Trace: [Wed Feb 6 09:40:28 2019][1697202.451697] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1697202.456875] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.463751] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1697202.469537] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.476014] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.483589] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1697202.488650] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.494947] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1697202.501563] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.507840] mdt03_033 S ffffa07b2cf8b0c0 0 17934 2 0x00000080 [Wed Feb 6 09:40:28 2019][1697202.515113] Call Trace: [Wed Feb 6 09:40:28 2019][1697202.517744] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1697202.522924] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.529799] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1697202.535587] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.542064] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.549642] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1697202.554725] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.561002] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1697202.567621] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.573897] mdt00_045 S ffffa07b2cf8e180 0 17935 2 0x00000080 [Wed Feb 6 09:40:28 2019][1697202.581172] Call Trace: [Wed Feb 6 09:40:28 2019][1697202.583802] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1697202.588981] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.595882] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1697202.601677] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.608153] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.615745] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1697202.620802] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.627077] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1697202.633716] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.639991] mdt00_046 S ffffa07b2cf8a080 0 17939 2 0x00000080 [Wed Feb 6 09:40:28 2019][1697202.647274] Call Trace: [Wed Feb 6 09:40:28 2019][1697202.649903] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1697202.655086] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.661969] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1697202.667754] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.674256] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.681835] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1697202.686894] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.693170] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1697202.699790] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.706065] mdt00_047 S ffffa06c29a1b0c0 0 17940 2 0x00000080 [Wed Feb 6 09:40:28 2019][1697202.713340] Call Trace: [Wed Feb 6 09:40:28 2019][1697202.715979] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:40:28 2019][1697202.722889] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.730638] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1697202.735812] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.742920] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.749999] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:28 2019][1697202.757138] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1697202.762922] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.770273] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.778289] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:28 2019][1697202.785006] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:40:28 2019][1697202.791981] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:28 2019][1697202.798719] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.806763] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.813858] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.820920] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:28 2019][1697202.828067] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Feb 6 09:40:28 2019][1697202.834699] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Feb 6 09:40:28 2019][1697202.841876] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.848529] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.855618] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.862773] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:40:28 2019][1697202.869487] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:28 2019][1697202.876120] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:40:28 2019][1697202.882941] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.889827] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:28 2019][1697202.897150] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:28 2019][1697202.903717] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.910978] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.918681] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.924995] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.932087] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.939841] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:28 2019][1697202.947104] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.954976] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.961944] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:28 2019][1697202.968391] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:28 2019][1697202.974439] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.980928] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697202.988500] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1697202.993560] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697202.999836] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1697203.006456] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697203.012728] mdt00_048 S ffffa07b3dd7d140 0 17952 2 0x00000080 [Wed Feb 6 09:40:28 2019][1697203.020012] Call Trace: [Wed Feb 6 09:40:28 2019][1697203.022643] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1697203.027826] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697203.034712] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1697203.040498] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697203.046982] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697203.054558] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1697203.059616] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697203.065893] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1697203.072537] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697203.078808] mdt00_049 S ffffa07b3dd7b0c0 0 17953 2 0x00000080 [Wed Feb 6 09:40:28 2019][1697203.086080] Call Trace: [Wed Feb 6 09:40:28 2019][1697203.088713] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1697203.093891] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697203.100793] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1697203.106578] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697203.113058] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697203.120633] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1697203.125715] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697203.132009] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1697203.138633] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1697203.144913] mdt03_034 S ffffa07b3dd79040 0 17957 2 0x00000080 [Wed Feb 6 09:40:28 2019][1697203.152181] Call Trace: [Wed Feb 6 09:40:28 2019][1697203.154812] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1697203.159993] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697203.166871] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1697203.172656] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1697203.179132] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.186706] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697203.191770] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.198043] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1697203.204664] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.210937] mdt01_042 S ffffa07b3dd7c100 0 17958 2 0x00000080 [Wed Feb 6 09:40:29 2019][1697203.218223] Call Trace: [Wed Feb 6 09:40:29 2019][1697203.220855] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1697203.226034] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.232909] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1697203.238695] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.245173] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.252747] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697203.257810] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.264085] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1697203.270706] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.276979] mdt03_035 S ffffa06c29ab5140 0 17959 2 0x00000080 [Wed Feb 6 09:40:29 2019][1697203.284255] Call Trace: [Wed Feb 6 09:40:29 2019][1697203.286885] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1697203.292066] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.298943] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1697203.304728] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.311233] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.318806] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697203.323867] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.330145] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1697203.336765] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.343039] mdt03_036 S ffffa06c29ab30c0 0 17963 2 0x00000080 [Wed Feb 6 09:40:29 2019][1697203.350315] Call Trace: [Wed Feb 6 09:40:29 2019][1697203.352943] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1697203.358124] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.365000] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1697203.370787] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.377266] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.384837] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697203.389899] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.396178] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1697203.402794] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.409069] mdt00_050 S ffffa06c29ab4100 0 17967 2 0x00000080 [Wed Feb 6 09:40:29 2019][1697203.416346] Call Trace: [Wed Feb 6 09:40:29 2019][1697203.418978] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1697203.424157] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.431058] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1697203.436844] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.443322] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.450898] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697203.455957] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.462234] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1697203.468878] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.475146] mdt03_037 S ffffa075af270000 0 17968 2 0x00000080 [Wed Feb 6 09:40:29 2019][1697203.482421] Call Trace: [Wed Feb 6 09:40:29 2019][1697203.485053] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1697203.490231] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.497109] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1697203.502893] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.509371] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.516945] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697203.522008] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.528285] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1697203.534903] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.541179] mdt03_038 S ffffa075af271040 0 17969 2 0x00000080 [Wed Feb 6 09:40:29 2019][1697203.548454] Call Trace: [Wed Feb 6 09:40:29 2019][1697203.551084] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1697203.556265] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.563165] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1697203.568948] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.575429] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.583006] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697203.588065] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.594362] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1697203.600979] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.607254] mdt00_051 S ffffa075af272080 0 17970 2 0x00000080 [Wed Feb 6 09:40:29 2019][1697203.614545] Call Trace: [Wed Feb 6 09:40:29 2019][1697203.617178] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1697203.622357] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.629236] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1697203.635021] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.641512] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.649089] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697203.654148] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.660426] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1697203.667046] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.673321] mdt00_052 S ffffa075af2730c0 0 17971 2 0x00000080 [Wed Feb 6 09:40:29 2019][1697203.680596] Call Trace: [Wed Feb 6 09:40:29 2019][1697203.683225] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1697203.688408] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.695294] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1697203.701095] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.707575] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.715148] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697203.720207] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.726485] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1697203.733105] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.739399] mdt01_043 S ffffa075af274100 0 17972 2 0x00000080 [Wed Feb 6 09:40:29 2019][1697203.746672] Call Trace: [Wed Feb 6 09:40:29 2019][1697203.749302] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1697203.754481] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.761358] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1697203.767145] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.773620] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.781197] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697203.786257] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.792535] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1697203.799154] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.805428] mdt01_044 S ffffa075af275140 0 17973 2 0x00000080 [Wed Feb 6 09:40:29 2019][1697203.812703] Call Trace: [Wed Feb 6 09:40:29 2019][1697203.815334] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1697203.820514] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.827393] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1697203.833176] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.839653] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.847227] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697203.852290] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.858566] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1697203.865185] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.871459] mdt01_045 S ffffa075af276180 0 17974 2 0x00000080 [Wed Feb 6 09:40:29 2019][1697203.878735] Call Trace: [Wed Feb 6 09:40:29 2019][1697203.881365] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1697203.886549] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.893432] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1697203.899218] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.905697] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.913296] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697203.918356] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.924631] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1697203.931251] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.937528] mdt00_053 S ffffa073079f0000 0 17975 2 0x00000080 [Wed Feb 6 09:40:29 2019][1697203.944830] Call Trace: [Wed Feb 6 09:40:29 2019][1697203.947458] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1697203.952643] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.959523] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1697203.965314] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.971794] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697203.979360] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697203.984446] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697203.990714] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1697203.997339] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697204.003613] mdt01_046 S ffffa073079f1040 0 17976 2 0x00000080 [Wed Feb 6 09:40:29 2019][1697204.010894] Call Trace: [Wed Feb 6 09:40:29 2019][1697204.013525] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1697204.018707] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697204.025592] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1697204.031376] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697204.037853] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697204.045429] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697204.050488] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697204.056767] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1697204.063388] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697204.069682] mdt03_039 S ffffa073079f2080 0 17977 2 0x00000080 [Wed Feb 6 09:40:29 2019][1697204.076953] Call Trace: [Wed Feb 6 09:40:29 2019][1697204.079586] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1697204.084765] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697204.091651] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1697204.097435] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697204.103936] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697204.111513] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697204.116575] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697204.122849] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1697204.129474] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1697204.135747] mdt01_047 S ffffa073079f30c0 0 17978 2 0x00000080 [Wed Feb 6 09:40:29 2019][1697204.143023] Call Trace: [Wed Feb 6 09:40:29 2019][1697204.145651] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1697204.150834] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697204.157717] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1697204.163503] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697204.169979] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1697204.177553] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1697204.182615] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.188893] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1697204.195512] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.201787] mdt03_040 S ffffa073079f4100 0 17982 2 0x00000080 [Wed Feb 6 09:40:30 2019][1697204.209069] Call Trace: [Wed Feb 6 09:40:30 2019][1697204.211702] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1697204.216880] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.223760] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1697204.229545] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.236054] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.243628] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1697204.248691] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.254968] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1697204.261586] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.267862] mdt00_054 S ffffa073079f5140 0 17983 2 0x00000080 [Wed Feb 6 09:40:30 2019][1697204.275146] Call Trace: [Wed Feb 6 09:40:30 2019][1697204.277777] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1697204.282956] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.289831] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1697204.295619] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.302097] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.309673] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1697204.314730] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.321006] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1697204.327627] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.333902] mdt02_039 S ffffa073079f6180 0 17984 2 0x00000080 [Wed Feb 6 09:40:30 2019][1697204.341177] Call Trace: [Wed Feb 6 09:40:30 2019][1697204.343808] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1697204.348988] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.355868] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1697204.361652] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.368128] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.375701] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1697204.380766] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.387037] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1697204.393663] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.399933] mdt01_048 S ffffa0749fb80000 0 17991 2 0x00000080 [Wed Feb 6 09:40:30 2019][1697204.407210] Call Trace: [Wed Feb 6 09:40:30 2019][1697204.409842] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1697204.415020] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.421896] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1697204.427684] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.434172] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.441745] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1697204.446805] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.453082] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1697204.459700] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.465976] mdt01_049 S ffffa0749fb81040 0 17992 2 0x00000080 [Wed Feb 6 09:40:30 2019][1697204.473251] Call Trace: [Wed Feb 6 09:40:30 2019][1697204.475881] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1697204.481062] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.487941] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1697204.493743] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.500226] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.507802] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1697204.512862] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.519140] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1697204.525758] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.532032] mdt02_040 S ffffa059a67ac100 0 17993 2 0x00000080 [Wed Feb 6 09:40:30 2019][1697204.539309] Call Trace: [Wed Feb 6 09:40:30 2019][1697204.541939] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1697204.547120] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.554056] [] ? ptlrpc_at_set_timer+0x5b/0x110 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.561199] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1697204.566987] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.573468] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.581043] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1697204.586108] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.592406] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1697204.599030] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.605304] mdt01_050 S ffffa059a67ad140 0 17994 2 0x00000080 [Wed Feb 6 09:40:30 2019][1697204.612579] Call Trace: [Wed Feb 6 09:40:30 2019][1697204.615243] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.622995] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1697204.628202] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.635282] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.642392] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:30 2019][1697204.649531] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1697204.655316] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.662663] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.670700] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.678467] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Feb 6 09:40:30 2019][1697204.685379] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.693161] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.701178] [] mdt_reint_unlink+0x331/0x14a0 [mdt] [Wed Feb 6 09:40:30 2019][1697204.707811] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 09:40:30 2019][1697204.714014] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 09:40:30 2019][1697204.720729] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Wed Feb 6 09:40:30 2019][1697204.727706] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 09:40:30 2019][1697204.733593] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.740679] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.748430] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:30 2019][1697204.755692] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.763555] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.770517] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:30 2019][1697204.776968] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:30 2019][1697204.783017] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.789497] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.797073] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1697204.802136] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.808410] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1697204.815031] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.821304] mdt01_051 S ffffa059a67aa080 0 17995 2 0x00000080 [Wed Feb 6 09:40:30 2019][1697204.828605] Call Trace: [Wed Feb 6 09:40:30 2019][1697204.831239] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1697204.836416] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.843291] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1697204.849080] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.855563] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.863140] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1697204.868226] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.874501] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1697204.881123] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.887398] mdt02_041 S ffffa08a5f63c100 0 17997 2 0x00000080 [Wed Feb 6 09:40:30 2019][1697204.894682] Call Trace: [Wed Feb 6 09:40:30 2019][1697204.897311] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1697204.902494] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.909377] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1697204.915183] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.921668] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.929238] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1697204.934295] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.940591] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1697204.947207] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697204.953482] mdt01_052 S ffffa08a5f63e180 0 17998 2 0x00000080 [Wed Feb 6 09:40:30 2019][1697204.960757] Call Trace: [Wed Feb 6 09:40:30 2019][1697204.963387] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1697204.968568] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.975446] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1697204.981231] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.987716] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697204.995292] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1697205.000350] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697205.006628] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1697205.013248] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697205.019524] mdt02_042 S ffffa076d9df8000 0 17999 2 0x00000080 [Wed Feb 6 09:40:30 2019][1697205.026806] Call Trace: [Wed Feb 6 09:40:30 2019][1697205.029438] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1697205.034618] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697205.041493] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1697205.047280] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697205.053755] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697205.061333] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1697205.066394] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697205.072671] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1697205.079289] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697205.085564] mdt00_055 S ffffa076d9df9040 0 18001 2 0x00000080 [Wed Feb 6 09:40:30 2019][1697205.092839] Call Trace: [Wed Feb 6 09:40:30 2019][1697205.095471] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1697205.100654] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697205.107532] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1697205.113328] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697205.119802] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697205.127373] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1697205.132433] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697205.138708] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1697205.145332] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1697205.151605] mdt01_053 S ffffa08b61332080 0 18044 2 0x00000080 [Wed Feb 6 09:40:30 2019][1697205.158886] Call Trace: [Wed Feb 6 09:40:30 2019][1697205.161519] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1697205.166699] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697205.173574] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1697205.179363] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:30 2019][1697205.185848] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.193422] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1697205.198482] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.204760] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1697205.211379] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.217659] mdt03_041 S ffffa07b77ecd140 0 18154 2 0x00000080 [Wed Feb 6 09:40:31 2019][1697205.224938] Call Trace: [Wed Feb 6 09:40:31 2019][1697205.227568] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1697205.232748] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.239624] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:31 2019][1697205.245411] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.251887] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.259487] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1697205.264549] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.270827] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1697205.277445] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.283722] mdt03_042 S ffffa07b77eca080 0 18164 2 0x00000080 [Wed Feb 6 09:40:31 2019][1697205.291022] Call Trace: [Wed Feb 6 09:40:31 2019][1697205.293654] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1697205.298828] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.305710] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:31 2019][1697205.311495] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.317971] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.325547] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1697205.330608] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.336884] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1697205.343506] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.349777] mdt02_043 S ffffa0749fb82080 0 18165 2 0x00000080 [Wed Feb 6 09:40:31 2019][1697205.357054] Call Trace: [Wed Feb 6 09:40:31 2019][1697205.359684] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1697205.364865] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.371741] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:31 2019][1697205.377526] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.384006] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.391579] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1697205.396642] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.402916] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1697205.409536] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.415813] mdt03_043 S ffffa0749fb830c0 0 18166 2 0x00000080 [Wed Feb 6 09:40:31 2019][1697205.423085] Call Trace: [Wed Feb 6 09:40:31 2019][1697205.425717] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1697205.430897] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.437774] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:31 2019][1697205.443557] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.450036] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.457610] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1697205.462672] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.468949] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1697205.475567] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.481844] mdt03_044 S ffffa0749fb85140 0 18177 2 0x00000080 [Wed Feb 6 09:40:31 2019][1697205.489128] Call Trace: [Wed Feb 6 09:40:31 2019][1697205.491757] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1697205.496938] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.503812] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:31 2019][1697205.509601] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.516077] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.523653] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1697205.528719] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.534990] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1697205.541613] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.547886] mdt02_044 S ffffa08b613330c0 0 18178 2 0x00000080 [Wed Feb 6 09:40:31 2019][1697205.555168] Call Trace: [Wed Feb 6 09:40:31 2019][1697205.557800] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1697205.562980] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.569886] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:31 2019][1697205.575670] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.582143] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.589719] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1697205.594781] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.601055] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1697205.607677] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.613949] mdt03_045 S ffffa076d9dfe180 0 18186 2 0x00000080 [Wed Feb 6 09:40:31 2019][1697205.621235] Call Trace: [Wed Feb 6 09:40:31 2019][1697205.623866] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1697205.629047] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.635929] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:31 2019][1697205.641715] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.648196] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.655790] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1697205.660847] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.667120] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1697205.673741] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.680018] mdt02_045 S ffffa059a67a8000 0 18223 2 0x00000080 [Wed Feb 6 09:40:31 2019][1697205.687294] Call Trace: [Wed Feb 6 09:40:31 2019][1697205.689923] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1697205.695104] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.701980] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:31 2019][1697205.707767] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.714259] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.721838] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1697205.726896] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.733172] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1697205.739793] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.746066] mdt02_046 S ffffa059a67a9040 0 18224 2 0x00000080 [Wed Feb 6 09:40:31 2019][1697205.753358] Call Trace: [Wed Feb 6 09:40:31 2019][1697205.755991] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1697205.761172] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.768055] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:31 2019][1697205.773841] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.780318] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.787895] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1697205.792955] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.799252] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1697205.805868] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.812145] mdt02_047 S ffffa059a67ab0c0 0 18238 2 0x00000080 [Wed Feb 6 09:40:31 2019][1697205.819426] Call Trace: [Wed Feb 6 09:40:31 2019][1697205.822058] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1697205.827237] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.834136] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:31 2019][1697205.839917] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.846394] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.853970] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1697205.859030] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.865326] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1697205.871944] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.878218] mdt03_046 S ffffa08a91f50000 0 18250 2 0x00000080 [Wed Feb 6 09:40:31 2019][1697205.885501] Call Trace: [Wed Feb 6 09:40:31 2019][1697205.888133] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1697205.893314] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.900197] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:31 2019][1697205.905982] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.912460] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.920038] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1697205.925097] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.931373] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1697205.937994] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.944270] mdt03_047 S ffffa08a91f54100 0 18260 2 0x00000080 [Wed Feb 6 09:40:31 2019][1697205.951551] Call Trace: [Wed Feb 6 09:40:31 2019][1697205.954183] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1697205.959362] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.966238] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:31 2019][1697205.972025] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.978501] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697205.986077] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1697205.991137] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697205.997415] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1697206.004034] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697206.010308] mdt03_048 S ffffa08a91f51040 0 18261 2 0x00000080 [Wed Feb 6 09:40:31 2019][1697206.017585] Call Trace: [Wed Feb 6 09:40:31 2019][1697206.020215] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1697206.025395] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697206.032271] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:31 2019][1697206.038058] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697206.044544] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697206.052120] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1697206.057205] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697206.063478] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1697206.070099] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697206.076379] mdt02_048 S ffffa05b7c256180 0 18262 2 0x00000080 [Wed Feb 6 09:40:31 2019][1697206.083664] Call Trace: [Wed Feb 6 09:40:31 2019][1697206.086298] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1697206.091477] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697206.098355] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:31 2019][1697206.104141] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697206.110619] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697206.118193] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1697206.123254] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697206.129531] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1697206.136151] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1697206.142423] mdt02_049 S ffffa05b7c254100 0 18265 2 0x00000080 [Wed Feb 6 09:40:31 2019][1697206.149702] Call Trace: [Wed Feb 6 09:40:31 2019][1697206.152331] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1697206.157511] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697206.164387] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:40:31 2019][1697206.171356] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:31 2019][1697206.177145] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:31 2019][1697206.183624] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.191193] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1697206.196256] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.202528] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1697206.209153] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.215426] mdt02_050 S ffffa05b7c696180 0 18266 2 0x00000080 [Wed Feb 6 09:40:32 2019][1697206.222700] Call Trace: [Wed Feb 6 09:40:32 2019][1697206.225331] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1697206.230513] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.237398] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1697206.243183] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.249667] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.257241] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1697206.262303] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.268581] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1697206.275200] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.281475] mdt02_051 S ffffa0521450e180 0 18267 2 0x00000080 [Wed Feb 6 09:40:32 2019][1697206.288748] Call Trace: [Wed Feb 6 09:40:32 2019][1697206.291381] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1697206.296561] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.303437] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1697206.309223] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.315708] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.323284] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1697206.328343] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.334621] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1697206.341244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.347516] mdt03_049 S ffffa0521450a080 0 18268 2 0x00000080 [Wed Feb 6 09:40:32 2019][1697206.354818] Call Trace: [Wed Feb 6 09:40:32 2019][1697206.357446] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1697206.362627] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.369504] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1697206.375290] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.381772] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.389343] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1697206.394403] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.400680] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1697206.407299] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.413573] mdt02_052 S ffffa0521450b0c0 0 18269 2 0x00000080 [Wed Feb 6 09:40:32 2019][1697206.420850] Call Trace: [Wed Feb 6 09:40:32 2019][1697206.423481] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1697206.428661] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.435543] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1697206.441334] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.447814] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.455391] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1697206.460452] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.466727] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1697206.473351] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.479622] mdt02_053 S ffffa05214509040 0 18270 2 0x00000080 [Wed Feb 6 09:40:32 2019][1697206.486897] Call Trace: [Wed Feb 6 09:40:32 2019][1697206.489530] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1697206.494711] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.501594] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1697206.507378] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.513858] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.521431] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1697206.526493] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.532771] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1697206.539389] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.545665] mdt03_050 S ffffa05b6e312080 0 18271 2 0x00000080 [Wed Feb 6 09:40:32 2019][1697206.552940] Call Trace: [Wed Feb 6 09:40:32 2019][1697206.555571] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1697206.560751] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.567626] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1697206.573412] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.579891] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.587465] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1697206.592524] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.598803] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1697206.605425] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.611695] mdt02_054 S ffffa05b753c0000 0 18272 2 0x00000080 [Wed Feb 6 09:40:32 2019][1697206.618980] Call Trace: [Wed Feb 6 09:40:32 2019][1697206.621611] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1697206.626791] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.633670] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1697206.639454] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.645929] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.653506] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1697206.658565] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.664843] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1697206.671463] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.677737] mdt03_051 S ffffa076d9dfc100 0 18277 2 0x00000080 [Wed Feb 6 09:40:32 2019][1697206.685012] Call Trace: [Wed Feb 6 09:40:32 2019][1697206.687643] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1697206.692824] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.699706] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1697206.705493] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.711971] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.719568] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1697206.724625] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.730902] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1697206.737523] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.743796] mdt02_055 S ffffa06b7a34c100 0 18335 2 0x00000080 [Wed Feb 6 09:40:32 2019][1697206.751070] Call Trace: [Wed Feb 6 09:40:32 2019][1697206.753701] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1697206.758880] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.765759] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1697206.771543] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.778022] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.785596] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1697206.790656] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.796932] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1697206.803551] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.809827] mdt02_056 S ffffa05b78b60000 0 18566 2 0x00000080 [Wed Feb 6 09:40:32 2019][1697206.817103] Call Trace: [Wed Feb 6 09:40:32 2019][1697206.819735] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1697206.824914] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.831788] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1697206.837577] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.844064] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.851638] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1697206.856698] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.862973] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1697206.869620] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.875894] mdt02_057 S ffffa05b78b65140 0 18647 2 0x00000080 [Wed Feb 6 09:40:32 2019][1697206.883169] Call Trace: [Wed Feb 6 09:40:32 2019][1697206.885801] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1697206.890982] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.897865] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1697206.903651] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.910129] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.917703] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1697206.922764] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.929040] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1697206.935660] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.941935] mdt02_058 S ffffa08a15309040 0 18653 2 0x00000080 [Wed Feb 6 09:40:32 2019][1697206.949209] Call Trace: [Wed Feb 6 09:40:32 2019][1697206.951841] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1697206.957022] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.963905] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1697206.969694] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.976170] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697206.983743] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1697206.988804] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697206.995081] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1697207.001725] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697207.007994] mdt02_059 S ffffa08a1530a080 0 18661 2 0x00000080 [Wed Feb 6 09:40:32 2019][1697207.015277] Call Trace: [Wed Feb 6 09:40:32 2019][1697207.017908] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1697207.023088] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697207.029962] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1697207.035749] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697207.042254] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697207.049829] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1697207.054889] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697207.061167] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1697207.067785] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697207.074062] mdt02_060 S ffffa08a1530c100 0 18678 2 0x00000080 [Wed Feb 6 09:40:32 2019][1697207.081334] Call Trace: [Wed Feb 6 09:40:32 2019][1697207.083966] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1697207.089146] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697207.096022] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1697207.101808] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697207.108286] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697207.115861] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1697207.120921] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697207.127198] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1697207.133818] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1697207.140094] mdt02_061 S ffffa07194369040 0 18682 2 0x00000080 [Wed Feb 6 09:40:32 2019][1697207.147366] Call Trace: [Wed Feb 6 09:40:32 2019][1697207.149999] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1697207.155179] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:32 2019][1697207.162053] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1697207.167842] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.174326] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.181902] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697207.186963] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.193238] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697207.199860] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.206132] kworker/5:1 S ffffa06b17270000 0 18809 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697207.213415] Call Trace: [Wed Feb 6 09:40:33 2019][1697207.216048] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697207.221198] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:33 2019][1697207.226951] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:33 2019][1697207.233661] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697207.238720] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.244995] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697207.251616] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.257892] mdt00_056 S ffffa0719436c100 0 18907 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697207.265182] Call Trace: [Wed Feb 6 09:40:33 2019][1697207.267814] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697207.272996] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.279878] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:33 2019][1697207.285665] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.292142] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.299719] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697207.304780] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.311055] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697207.317676] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.323949] mdt01_054 S ffffa086d5c5b0c0 0 19059 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697207.331225] Call Trace: [Wed Feb 6 09:40:33 2019][1697207.333855] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697207.339037] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.345919] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:33 2019][1697207.351709] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.358198] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.365768] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697207.370827] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.377104] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697207.383723] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.390001] mdt01_055 S ffffa086d5c5c100 0 19060 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697207.397282] Call Trace: [Wed Feb 6 09:40:33 2019][1697207.399914] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697207.405094] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.411970] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:33 2019][1697207.417756] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.424233] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.431809] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697207.436870] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.443145] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697207.449767] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.456039] mdt01_056 S ffffa05b78b66180 0 19083 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697207.463316] Call Trace: [Wed Feb 6 09:40:33 2019][1697207.465946] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697207.471126] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.478000] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:33 2019][1697207.483789] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.490279] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.497851] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697207.502910] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.509186] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697207.515808] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.522081] mdt01_057 S ffffa08b7a82d140 0 19086 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697207.529371] Call Trace: [Wed Feb 6 09:40:33 2019][1697207.532003] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697207.537185] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.544060] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:33 2019][1697207.549846] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.556325] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.563900] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697207.568958] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.575236] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697207.581856] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.588131] mdt01_058 S ffffa05b73556180 0 19087 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697207.595413] Call Trace: [Wed Feb 6 09:40:33 2019][1697207.598045] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697207.603226] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.610110] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:33 2019][1697207.615897] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.622372] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.629948] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697207.635008] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.641285] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697207.647905] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.654181] mdt01_059 S ffffa05b73555140 0 19097 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697207.661454] Call Trace: [Wed Feb 6 09:40:33 2019][1697207.664088] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697207.669266] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.676141] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:33 2019][1697207.681929] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.688410] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.696002] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697207.701057] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.707336] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697207.713954] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.720230] mdt01_060 S ffffa08b7a2530c0 0 19113 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697207.727502] Call Trace: [Wed Feb 6 09:40:33 2019][1697207.730135] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697207.735316] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.742200] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:33 2019][1697207.747987] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.754471] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.762048] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697207.767108] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.773384] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697207.780006] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.786279] mdt01_061 S ffffa08b7a256180 0 19115 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697207.793562] Call Trace: [Wed Feb 6 09:40:33 2019][1697207.796194] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697207.801375] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.808257] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:33 2019][1697207.814046] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.820536] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.828105] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697207.833189] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.839459] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697207.846078] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.852353] mdt01_062 S ffffa08b7a251040 0 19136 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697207.859629] Call Trace: [Wed Feb 6 09:40:33 2019][1697207.862260] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697207.867441] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.874315] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:33 2019][1697207.880101] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.886579] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.894154] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697207.899216] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.905493] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697207.912112] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.918388] mdt01_063 S ffffa08b7a255140 0 19141 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697207.925672] Call Trace: [Wed Feb 6 09:40:33 2019][1697207.928301] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697207.933482] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.940357] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:33 2019][1697207.946143] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.952621] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:33 2019][1697207.960195] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697207.965256] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.971535] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697207.978152] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697207.984426] kworker/27:2 S ffffa089e1f84100 0 19266 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697207.991708] Call Trace: [Wed Feb 6 09:40:33 2019][1697207.994340] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697207.999483] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:33 2019][1697208.005236] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:33 2019][1697208.011947] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697208.017003] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697208.023273] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697208.029895] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697208.036167] kworker/8:2 S ffffa05b735530c0 0 19466 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697208.043447] Call Trace: [Wed Feb 6 09:40:33 2019][1697208.046086] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:33 2019][1697208.052273] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697208.057419] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:33 2019][1697208.063177] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:33 2019][1697208.069882] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697208.074943] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697208.081217] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697208.087842] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697208.094112] kworker/16:0 S ffffa05b7abc4100 0 19775 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697208.101402] Call Trace: [Wed Feb 6 09:40:33 2019][1697208.104035] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:33 2019][1697208.110227] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697208.115374] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:33 2019][1697208.121129] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:33 2019][1697208.127836] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:33 2019][1697208.132897] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697208.139171] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:33 2019][1697208.145796] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:33 2019][1697208.152068] kworker/u259:1 S ffffa07b75ef2080 0 20339 2 0x00000080 [Wed Feb 6 09:40:33 2019][1697208.159348] Call Trace: [Wed Feb 6 09:40:33 2019][1697208.161981] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:33 2019][1697208.168172] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1697208.173319] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:34 2019][1697208.179077] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:34 2019][1697208.185783] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697208.190847] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.197145] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697208.203769] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.210063] kworker/u258:1 S ffffa06b03cb5140 0 22145 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697208.217338] Call Trace: [Wed Feb 6 09:40:34 2019][1697208.219973] [] ? __call_usermodehelper+0x4e/0x90 [Wed Feb 6 09:40:34 2019][1697208.226423] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:34 2019][1697208.232611] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697208.237758] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:34 2019][1697208.243512] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:34 2019][1697208.250219] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697208.255281] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.261558] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697208.268178] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.274452] kworker/9:1 S ffffa06b03cb6180 0 22207 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697208.281733] Call Trace: [Wed Feb 6 09:40:34 2019][1697208.284368] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697208.289515] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:34 2019][1697208.295272] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:34 2019][1697208.301978] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697208.307042] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.313313] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697208.319934] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.326212] kworker/u260:0 S ffffa08a5f639040 0 22241 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697208.333489] Call Trace: [Wed Feb 6 09:40:34 2019][1697208.336118] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:34 2019][1697208.342306] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697208.347453] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:34 2019][1697208.353210] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:34 2019][1697208.359915] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697208.364979] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.371250] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697208.377874] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.384148] kworker/u257:2 S ffffa0557c426180 0 22262 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697208.391426] Call Trace: [Wed Feb 6 09:40:34 2019][1697208.394054] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697208.399203] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:34 2019][1697208.404955] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:34 2019][1697208.411665] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697208.416726] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.423002] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697208.429620] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.435898] kworker/41:2 S ffffa06a03efb0c0 0 22569 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697208.443176] Call Trace: [Wed Feb 6 09:40:34 2019][1697208.445813] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:34 2019][1697208.452001] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697208.457149] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:34 2019][1697208.462904] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:34 2019][1697208.469635] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697208.474692] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.480963] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697208.487588] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.493860] kworker/12:1 S ffffa0557c422080 0 23245 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697208.501149] Call Trace: [Wed Feb 6 09:40:34 2019][1697208.503786] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697208.508932] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:34 2019][1697208.514689] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:34 2019][1697208.521395] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697208.526457] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.532731] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697208.539351] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.545625] mdt_out03_004 S ffffa05b789930c0 0 23366 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697208.552902] Call Trace: [Wed Feb 6 09:40:34 2019][1697208.555533] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697208.560727] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.567609] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:34 2019][1697208.573391] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.579870] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.587446] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697208.592528] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.598799] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697208.605419] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.611693] kworker/4:2 S ffffa05b7abc1040 0 23597 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697208.618974] Call Trace: [Wed Feb 6 09:40:34 2019][1697208.621611] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:34 2019][1697208.627798] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697208.632945] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:34 2019][1697208.638701] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:34 2019][1697208.645406] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697208.650470] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.656744] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697208.663367] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.669641] ldlm_cn02_016 S ffffa05b7abc2080 0 23607 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697208.676915] Call Trace: [Wed Feb 6 09:40:34 2019][1697208.679546] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697208.684728] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.691611] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:34 2019][1697208.697398] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.703873] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.711452] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697208.716510] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.722786] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697208.729406] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.735680] ldlm_cn01_018 S ffffa05b7abc30c0 0 23608 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697208.742966] Call Trace: [Wed Feb 6 09:40:34 2019][1697208.745596] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697208.750777] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.757660] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:34 2019][1697208.763448] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.769957] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.777534] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697208.782596] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.788871] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697208.795489] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.801766] ldlm_cn01_019 S ffffa06b03cb1040 0 23652 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697208.809041] Call Trace: [Wed Feb 6 09:40:34 2019][1697208.811669] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697208.816852] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.823736] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:34 2019][1697208.829523] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.836008] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.843584] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697208.848645] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.854918] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697208.861540] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.867814] mdt_seqm_0003 S ffffa05b69e82080 0 23717 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697208.875091] Call Trace: [Wed Feb 6 09:40:34 2019][1697208.877719] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697208.882901] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.889776] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:34 2019][1697208.895561] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.902040] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.909615] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697208.914716] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.920986] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697208.927632] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.933908] ldlm_cn02_017 S ffffa08b79f2d140 0 23719 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697208.941190] Call Trace: [Wed Feb 6 09:40:34 2019][1697208.943823] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697208.949003] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.955877] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:34 2019][1697208.961667] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.968156] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697208.975725] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697208.980787] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.987062] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697208.993684] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697208.999955] ldlm_cn02_018 S ffffa08b79f2a080 0 23720 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697209.007232] Call Trace: [Wed Feb 6 09:40:34 2019][1697209.009864] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697209.015063] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697209.021945] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:34 2019][1697209.027731] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697209.034215] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697209.041786] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697209.046845] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697209.053121] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697209.059740] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697209.066014] ldlm_cn01_020 S ffffa08b79f2b0c0 0 23721 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697209.073290] Call Trace: [Wed Feb 6 09:40:34 2019][1697209.075923] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697209.081101] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697209.087975] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:34 2019][1697209.093766] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697209.100250] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697209.107823] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:34 2019][1697209.112884] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697209.119162] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:34 2019][1697209.125783] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:34 2019][1697209.132056] ldlm_cn02_019 S ffffa08b79f2c100 0 23722 2 0x00000080 [Wed Feb 6 09:40:34 2019][1697209.139331] Call Trace: [Wed Feb 6 09:40:34 2019][1697209.141963] [] schedule+0x29/0x70 [Wed Feb 6 09:40:34 2019][1697209.147142] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697209.154018] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:34 2019][1697209.159805] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697209.166285] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:34 2019][1697209.173856] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697209.178917] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.185193] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:35 2019][1697209.191813] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.198089] ldlm_cn01_021 S ffffa08b5d629040 0 23723 2 0x00000080 [Wed Feb 6 09:40:35 2019][1697209.205363] Call Trace: [Wed Feb 6 09:40:35 2019][1697209.207994] [] schedule+0x29/0x70 [Wed Feb 6 09:40:35 2019][1697209.213175] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.220052] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:35 2019][1697209.225838] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.232324] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.239899] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697209.244959] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.251233] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:35 2019][1697209.257853] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.264129] ldlm_cn02_020 S ffffa07c2948b0c0 0 23724 2 0x00000080 [Wed Feb 6 09:40:35 2019][1697209.271405] Call Trace: [Wed Feb 6 09:40:35 2019][1697209.274035] [] schedule+0x29/0x70 [Wed Feb 6 09:40:35 2019][1697209.279216] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.286100] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:35 2019][1697209.291886] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.298363] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.305940] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697209.310998] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.317277] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:35 2019][1697209.323895] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.330169] ldlm_cn01_022 S ffffa07b76b48000 0 23725 2 0x00000080 [Wed Feb 6 09:40:35 2019][1697209.337444] Call Trace: [Wed Feb 6 09:40:35 2019][1697209.340077] [] schedule+0x29/0x70 [Wed Feb 6 09:40:35 2019][1697209.345257] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.352139] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:35 2019][1697209.357928] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.364405] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.371980] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697209.377039] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.383316] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:35 2019][1697209.389934] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.396209] ldlm_cn02_021 S ffffa07b76b49040 0 23726 2 0x00000080 [Wed Feb 6 09:40:35 2019][1697209.403479] Call Trace: [Wed Feb 6 09:40:35 2019][1697209.406108] [] schedule+0x29/0x70 [Wed Feb 6 09:40:35 2019][1697209.411288] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.418163] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:40:35 2019][1697209.425133] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:35 2019][1697209.430926] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.437403] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.444980] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697209.450041] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.456313] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:35 2019][1697209.462938] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.469233] ldlm_cn01_023 S ffffa08b760b8000 0 23727 2 0x00000080 [Wed Feb 6 09:40:35 2019][1697209.476512] Call Trace: [Wed Feb 6 09:40:35 2019][1697209.479145] [] schedule+0x29/0x70 [Wed Feb 6 09:40:35 2019][1697209.484324] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.491201] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:35 2019][1697209.496986] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.503463] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.511039] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697209.516121] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.522392] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:35 2019][1697209.529011] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.535286] ldlm_cn02_022 S ffffa07c295ab0c0 0 23728 2 0x00000080 [Wed Feb 6 09:40:35 2019][1697209.542569] Call Trace: [Wed Feb 6 09:40:35 2019][1697209.545203] [] schedule+0x29/0x70 [Wed Feb 6 09:40:35 2019][1697209.550381] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.557256] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:35 2019][1697209.563044] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.569520] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.577095] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697209.582158] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.588432] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:35 2019][1697209.595052] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.601327] ldlm_cn01_024 S ffffa07c295a9040 0 23729 2 0x00000080 [Wed Feb 6 09:40:35 2019][1697209.608602] Call Trace: [Wed Feb 6 09:40:35 2019][1697209.611233] [] schedule+0x29/0x70 [Wed Feb 6 09:40:35 2019][1697209.616415] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.623323] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:35 2019][1697209.629110] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.635591] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.643184] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697209.648241] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.654516] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:35 2019][1697209.661138] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.667412] ldlm_cn02_023 S ffffa08b76ccd140 0 23730 2 0x00000080 [Wed Feb 6 09:40:35 2019][1697209.674688] Call Trace: [Wed Feb 6 09:40:35 2019][1697209.677318] [] schedule+0x29/0x70 [Wed Feb 6 09:40:35 2019][1697209.682495] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.689372] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:35 2019][1697209.695156] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.701638] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.709212] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697209.714272] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.720551] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:35 2019][1697209.727168] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.733443] ldlm_cn01_025 S ffffa08a5f638000 0 23731 2 0x00000080 [Wed Feb 6 09:40:35 2019][1697209.740720] Call Trace: [Wed Feb 6 09:40:35 2019][1697209.743351] [] schedule+0x29/0x70 [Wed Feb 6 09:40:35 2019][1697209.748532] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.755415] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:35 2019][1697209.761201] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.767683] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.775252] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697209.780313] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.786591] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:35 2019][1697209.793210] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.799502] ldlm_cn02_024 S ffffa089e1f830c0 0 23732 2 0x00000080 [Wed Feb 6 09:40:35 2019][1697209.806787] Call Trace: [Wed Feb 6 09:40:35 2019][1697209.809417] [] schedule+0x29/0x70 [Wed Feb 6 09:40:35 2019][1697209.814596] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.821472] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:35 2019][1697209.827258] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.833738] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.841312] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697209.846373] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.852650] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:35 2019][1697209.859268] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.865544] ldlm_cn01_026 S ffffa089e1f81040 0 23733 2 0x00000080 [Wed Feb 6 09:40:35 2019][1697209.872818] Call Trace: [Wed Feb 6 09:40:35 2019][1697209.875450] [] schedule+0x29/0x70 [Wed Feb 6 09:40:35 2019][1697209.880629] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.887508] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:35 2019][1697209.893289] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.899768] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.907344] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697209.912406] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.918681] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:35 2019][1697209.925300] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.931575] ldlm_cn02_025 S ffffa06b03cb2080 0 23734 2 0x00000080 [Wed Feb 6 09:40:35 2019][1697209.938850] Call Trace: [Wed Feb 6 09:40:35 2019][1697209.941484] [] schedule+0x29/0x70 [Wed Feb 6 09:40:35 2019][1697209.946662] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.953537] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:35 2019][1697209.959323] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.965805] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697209.973376] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697209.978434] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.984713] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:35 2019][1697209.991332] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697209.997607] ldlm_cn01_027 S ffffa06b7807b0c0 0 23735 2 0x00000080 [Wed Feb 6 09:40:35 2019][1697210.004882] Call Trace: [Wed Feb 6 09:40:35 2019][1697210.007515] [] schedule+0x29/0x70 [Wed Feb 6 09:40:35 2019][1697210.012698] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697210.019584] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:35 2019][1697210.025367] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697210.031846] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697210.039417] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697210.044478] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697210.050776] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:35 2019][1697210.057399] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697210.063673] ldlm_cn01_028 S ffffa08a53b68000 0 23736 2 0x00000080 [Wed Feb 6 09:40:35 2019][1697210.070950] Call Trace: [Wed Feb 6 09:40:35 2019][1697210.073581] [] schedule+0x29/0x70 [Wed Feb 6 09:40:35 2019][1697210.078761] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697210.085637] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:35 2019][1697210.091422] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697210.097900] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697210.105474] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697210.110535] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697210.116811] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:35 2019][1697210.123432] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:35 2019][1697210.129706] ldlm_cn01_029 S ffffa08a53b69040 0 23737 2 0x00000080 [Wed Feb 6 09:40:35 2019][1697210.136983] Call Trace: [Wed Feb 6 09:40:35 2019][1697210.139614] [] schedule+0x29/0x70 [Wed Feb 6 09:40:35 2019][1697210.144791] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697210.151670] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:35 2019][1697210.157455] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697210.163938] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:35 2019][1697210.171507] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:35 2019][1697210.176567] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.182844] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697210.189463] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.195738] ldlm_cn01_030 S ffffa08a53b6a080 0 23738 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697210.203024] Call Trace: [Wed Feb 6 09:40:36 2019][1697210.205653] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697210.210833] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.217711] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:36 2019][1697210.223496] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.229971] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.237551] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697210.242608] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.248884] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697210.255504] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.261779] ldlm_cn02_026 S ffffa08a53b6b0c0 0 23739 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697210.269055] Call Trace: [Wed Feb 6 09:40:36 2019][1697210.271687] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697210.276866] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.283744] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:36 2019][1697210.289526] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.296006] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.303579] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697210.308641] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.314917] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697210.321535] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.327812] ldlm_cn01_031 S ffffa08a53b6c100 0 23740 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697210.335087] Call Trace: [Wed Feb 6 09:40:36 2019][1697210.337716] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697210.342898] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.349775] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:36 2019][1697210.355560] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.362063] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.369638] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697210.374699] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.380976] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697210.387594] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.393869] ldlm_cn02_027 S ffffa08a53b6d140 0 23741 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697210.401146] Call Trace: [Wed Feb 6 09:40:36 2019][1697210.403799] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697210.408973] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.415852] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:36 2019][1697210.421635] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.428113] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.435688] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697210.440748] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.447026] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697210.453667] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.459936] ldlm_cn02_028 S ffffa08a53b6e180 0 23742 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697210.467213] Call Trace: [Wed Feb 6 09:40:36 2019][1697210.469843] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697210.475024] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.481931] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:36 2019][1697210.487721] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.494198] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.501771] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697210.506855] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.513125] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697210.519748] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.526021] ldlm_cn02_029 S ffffa08b62fd8000 0 23743 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697210.533323] Call Trace: [Wed Feb 6 09:40:36 2019][1697210.535957] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697210.541133] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.548035] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:36 2019][1697210.553821] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.560303] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.567875] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697210.572932] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.579212] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697210.585831] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.592127] ldlm_cn02_030 S ffffa08b62fd9040 0 23747 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697210.599398] Call Trace: [Wed Feb 6 09:40:36 2019][1697210.602029] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697210.607209] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.614084] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:36 2019][1697210.619888] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.626369] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.633941] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697210.639001] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.645278] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697210.651895] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.658173] ldlm_cn02_031 S ffffa08b62fda080 0 23748 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697210.665448] Call Trace: [Wed Feb 6 09:40:36 2019][1697210.668077] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697210.673257] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.680136] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:36 2019][1697210.685919] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.692402] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:36 2019][1697210.699973] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697210.705033] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.711309] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697210.717929] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.724203] kworker/3:1 S ffffa08b62fdc100 0 23777 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697210.731485] Call Trace: [Wed Feb 6 09:40:36 2019][1697210.734120] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:36 2019][1697210.740309] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697210.745453] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:36 2019][1697210.751213] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:36 2019][1697210.757917] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697210.762980] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.769255] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697210.775877] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.782149] kworker/32:0 S ffffa0557c4230c0 0 23806 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697210.789429] Call Trace: [Wed Feb 6 09:40:36 2019][1697210.792064] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:36 2019][1697210.798255] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697210.803401] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:36 2019][1697210.809160] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:36 2019][1697210.815862] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697210.820927] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.827202] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697210.833824] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.840095] kworker/18:1 S ffffa07b79229040 0 23807 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697210.847376] Call Trace: [Wed Feb 6 09:40:36 2019][1697210.850037] [] ? ib_cq_completion_workqueue+0x1b/0x20 [ib_core] [Wed Feb 6 09:40:36 2019][1697210.857786] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:36 2019][1697210.863973] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697210.869122] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:36 2019][1697210.874875] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:36 2019][1697210.881584] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697210.886645] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.892921] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697210.899540] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.905816] kworker/35:2 S ffffa08b79e7a080 0 24048 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697210.913096] Call Trace: [Wed Feb 6 09:40:36 2019][1697210.915731] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697210.920882] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:36 2019][1697210.926634] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:36 2019][1697210.933343] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697210.938405] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.944681] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697210.951300] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.957574] kworker/7:2 S ffffa08b62fdb0c0 0 24062 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697210.964855] Call Trace: [Wed Feb 6 09:40:36 2019][1697210.967491] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697210.972640] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:36 2019][1697210.978392] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:36 2019][1697210.985101] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697210.990164] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697210.996439] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697211.003057] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697211.009332] kworker/47:0 S ffffa08b62fdd140 0 24086 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697211.016620] Call Trace: [Wed Feb 6 09:40:36 2019][1697211.019256] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:36 2019][1697211.025446] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697211.030590] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:36 2019][1697211.036348] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:36 2019][1697211.043055] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697211.048117] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697211.054391] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697211.061014] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697211.067287] kworker/21:0 S ffffa06b786b8000 0 24099 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697211.074566] Call Trace: [Wed Feb 6 09:40:36 2019][1697211.077200] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:36 2019][1697211.083392] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697211.088539] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:36 2019][1697211.094296] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:36 2019][1697211.101000] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697211.106063] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697211.112337] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697211.118961] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697211.125233] kworker/37:1 S ffffa05c298d30c0 0 24125 2 0x00000080 [Wed Feb 6 09:40:36 2019][1697211.132522] Call Trace: [Wed Feb 6 09:40:36 2019][1697211.135160] [] schedule+0x29/0x70 [Wed Feb 6 09:40:36 2019][1697211.140307] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:36 2019][1697211.146058] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:36 2019][1697211.152768] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:36 2019][1697211.157829] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697211.164104] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:36 2019][1697211.170725] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:36 2019][1697211.176998] pickup S ffffa06b787cd140 0 24156 18601 0x00000080 [Wed Feb 6 09:40:37 2019][1697211.184274] Call Trace: [Wed Feb 6 09:40:37 2019][1697211.186907] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:40:37 2019][1697211.193616] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697211.198764] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:40:37 2019][1697211.205905] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:40:37 2019][1697211.211830] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:40:37 2019][1697211.219148] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:37 2019][1697211.225683] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:40:37 2019][1697211.230917] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:37 2019][1697211.236670] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:40:37 2019][1697211.242427] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:37 2019][1697211.248614] ldlm_cn03_017 S ffffa08a73e2c100 0 24531 2 0x00000080 [Wed Feb 6 09:40:37 2019][1697211.255888] Call Trace: [Wed Feb 6 09:40:37 2019][1697211.258518] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697211.263705] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.270582] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:37 2019][1697211.276373] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.282855] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.290430] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:37 2019][1697211.295495] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.301766] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:37 2019][1697211.308391] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.314662] ldlm_cn03_018 S ffffa08b76674100 0 24532 2 0x00000080 [Wed Feb 6 09:40:37 2019][1697211.321945] Call Trace: [Wed Feb 6 09:40:37 2019][1697211.324575] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697211.329753] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.336634] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:37 2019][1697211.342419] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.348896] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.356471] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:37 2019][1697211.361533] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.367808] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:37 2019][1697211.374429] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.380707] ldlm_cn03_019 S ffffa08b77eec100 0 24582 2 0x00000080 [Wed Feb 6 09:40:37 2019][1697211.387980] Call Trace: [Wed Feb 6 09:40:37 2019][1697211.390609] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697211.395792] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.402674] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:37 2019][1697211.408462] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.414947] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.422523] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:37 2019][1697211.427582] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.433861] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:37 2019][1697211.440477] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.446754] ldlm_cn03_020 S ffffa08b77eed140 0 24583 2 0x00000080 [Wed Feb 6 09:40:37 2019][1697211.454028] Call Trace: [Wed Feb 6 09:40:37 2019][1697211.456660] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697211.461841] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.468748] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:37 2019][1697211.474536] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.481012] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.488588] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:37 2019][1697211.493673] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.499942] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:37 2019][1697211.506562] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.512837] ldlm_cn03_021 S ffffa08b77ee9040 0 24584 2 0x00000080 [Wed Feb 6 09:40:37 2019][1697211.520113] Call Trace: [Wed Feb 6 09:40:37 2019][1697211.522745] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697211.527925] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.534808] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:37 2019][1697211.540593] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.547071] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.554646] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:37 2019][1697211.559709] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.565984] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:37 2019][1697211.572604] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.578879] ldlm_cn03_022 S ffffa06b787cb0c0 0 24585 2 0x00000080 [Wed Feb 6 09:40:37 2019][1697211.586163] Call Trace: [Wed Feb 6 09:40:37 2019][1697211.588792] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697211.593973] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.600849] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:37 2019][1697211.606636] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.613116] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.620688] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:37 2019][1697211.625748] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.632024] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:37 2019][1697211.638645] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.644919] ldlm_cn03_023 S ffffa06b787cc100 0 24586 2 0x00000080 [Wed Feb 6 09:40:37 2019][1697211.652201] Call Trace: [Wed Feb 6 09:40:37 2019][1697211.654833] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697211.660014] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.666890] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:37 2019][1697211.672676] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.679152] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.686729] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:37 2019][1697211.691789] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.698065] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:37 2019][1697211.704687] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.710959] ldlm_cn03_024 S ffffa06b787c8000 0 24587 2 0x00000080 [Wed Feb 6 09:40:37 2019][1697211.718245] Call Trace: [Wed Feb 6 09:40:37 2019][1697211.720874] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697211.726057] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.732939] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:37 2019][1697211.738728] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.745204] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.752777] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:37 2019][1697211.757839] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.764115] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:37 2019][1697211.770756] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.777028] ldlm_cn03_025 S ffffa07b7a124100 0 24588 2 0x00000080 [Wed Feb 6 09:40:37 2019][1697211.784310] Call Trace: [Wed Feb 6 09:40:37 2019][1697211.786940] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697211.792122] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.798996] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:37 2019][1697211.804785] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.811275] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.818845] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:37 2019][1697211.823905] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.830181] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:37 2019][1697211.836801] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.843076] kworker/18:2 S ffffa07b7a121040 0 24592 2 0x00000080 [Wed Feb 6 09:40:37 2019][1697211.850365] Call Trace: [Wed Feb 6 09:40:37 2019][1697211.853000] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:37 2019][1697211.859190] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697211.864340] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:37 2019][1697211.870093] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:37 2019][1697211.876798] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:37 2019][1697211.881862] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.888135] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:37 2019][1697211.894759] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.901030] ldlm_cn03_026 S ffffa0749fb84100 0 24595 2 0x00000080 [Wed Feb 6 09:40:37 2019][1697211.908314] Call Trace: [Wed Feb 6 09:40:37 2019][1697211.910945] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697211.916128] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.923011] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:37 2019][1697211.928796] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.935273] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.942849] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:37 2019][1697211.947910] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.954187] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:37 2019][1697211.960807] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697211.967080] ldlm_cn03_027 S ffffa07b7c885140 0 24596 2 0x00000080 [Wed Feb 6 09:40:37 2019][1697211.974355] Call Trace: [Wed Feb 6 09:40:37 2019][1697211.976986] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697211.982167] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697211.989044] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:37 2019][1697211.994829] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697212.001306] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697212.008882] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:37 2019][1697212.013941] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697212.020220] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:37 2019][1697212.026839] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697212.033112] ldlm_cn03_028 S ffffa07b75d99040 0 24610 2 0x00000080 [Wed Feb 6 09:40:37 2019][1697212.040387] Call Trace: [Wed Feb 6 09:40:37 2019][1697212.043018] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697212.048200] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697212.055084] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:37 2019][1697212.060868] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697212.067347] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697212.074922] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:37 2019][1697212.079981] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697212.086260] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:37 2019][1697212.092879] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697212.099153] ldlm_cn03_029 S ffffa07b75d9c100 0 24611 2 0x00000080 [Wed Feb 6 09:40:37 2019][1697212.106430] Call Trace: [Wed Feb 6 09:40:37 2019][1697212.109059] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697212.114239] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697212.121116] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:37 2019][1697212.126901] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697212.133385] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:37 2019][1697212.140956] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:37 2019][1697212.146013] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697212.152292] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:37 2019][1697212.158909] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:37 2019][1697212.165187] ldlm_cn03_030 S ffffa07b75d9a080 0 24612 2 0x00000080 [Wed Feb 6 09:40:37 2019][1697212.172470] Call Trace: [Wed Feb 6 09:40:37 2019][1697212.175098] [] schedule+0x29/0x70 [Wed Feb 6 09:40:37 2019][1697212.180280] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:38 2019][1697212.187157] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:38 2019][1697212.192942] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:38 2019][1697212.199419] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:38 2019][1697212.206994] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697212.212057] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.218332] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697212.224951] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.231228] ldlm_cn03_031 S ffffa07b75d9d140 0 24613 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697212.238503] Call Trace: [Wed Feb 6 09:40:38 2019][1697212.241132] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697212.246312] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:38 2019][1697212.253190] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:38 2019][1697212.258977] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:38 2019][1697212.265461] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:38 2019][1697212.273035] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697212.278095] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.284372] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697212.290992] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.297268] kworker/u260:2 S ffffa08b70216180 0 24624 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697212.304547] Call Trace: [Wed Feb 6 09:40:38 2019][1697212.307181] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:38 2019][1697212.313374] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697212.318520] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:38 2019][1697212.324274] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:38 2019][1697212.330981] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697212.336043] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.342319] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697212.348940] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.355213] kworker/40:2 S ffffa05b7c10e180 0 24636 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697212.362493] Call Trace: [Wed Feb 6 09:40:38 2019][1697212.365131] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:38 2019][1697212.371319] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697212.376466] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:38 2019][1697212.382224] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:38 2019][1697212.388928] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697212.393990] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.400262] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697212.406887] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.413159] kworker/u259:2 S ffffa07b74eea080 0 24644 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697212.420440] Call Trace: [Wed Feb 6 09:40:38 2019][1697212.423074] [] ? __call_usermodehelper+0x4e/0x90 [Wed Feb 6 09:40:38 2019][1697212.429524] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:38 2019][1697212.435710] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697212.440856] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:38 2019][1697212.446603] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:38 2019][1697212.453334] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697212.458390] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.464668] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697212.471288] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.477562] kworker/43:1 S ffffa08b75828000 0 24661 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697212.484869] Call Trace: [Wed Feb 6 09:40:38 2019][1697212.487503] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:38 2019][1697212.493694] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697212.498838] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:38 2019][1697212.504594] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:38 2019][1697212.511302] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697212.516365] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.522638] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697212.529262] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.535533] kworker/14:2 S ffffa07b77ec8000 0 24676 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697212.542815] Call Trace: [Wed Feb 6 09:40:38 2019][1697212.545449] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:38 2019][1697212.551637] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697212.556787] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:38 2019][1697212.562541] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:38 2019][1697212.569249] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697212.574312] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.580584] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697212.587206] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.593503] kworker/46:0 S ffffa07b77ecc100 0 24693 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697212.600778] Call Trace: [Wed Feb 6 09:40:38 2019][1697212.603413] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:38 2019][1697212.609601] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697212.614748] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:38 2019][1697212.620506] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:38 2019][1697212.627212] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697212.632273] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.638547] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697212.645172] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.651444] kworker/39:2 S ffffa086d5c5e180 0 24707 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697212.658724] Call Trace: [Wed Feb 6 09:40:38 2019][1697212.661357] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697212.666507] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:38 2019][1697212.672262] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:38 2019][1697212.678972] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697212.684031] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.690306] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697212.696926] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.703201] kworker/23:0 S ffffa086d5c58000 0 24710 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697212.710482] Call Trace: [Wed Feb 6 09:40:38 2019][1697212.713118] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697212.718267] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:38 2019][1697212.724021] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:38 2019][1697212.730730] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697212.735790] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.742065] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697212.748685] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.754961] kworker/8:1 S ffffa04ca9b61040 0 24778 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697212.762239] Call Trace: [Wed Feb 6 09:40:38 2019][1697212.764875] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:38 2019][1697212.771065] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697212.776211] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:38 2019][1697212.781967] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:38 2019][1697212.788675] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697212.793737] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.800011] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697212.806631] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.812905] kworker/36:2 S ffffa059a67ae180 0 24836 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697212.820185] Call Trace: [Wed Feb 6 09:40:38 2019][1697212.822820] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697212.827971] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:38 2019][1697212.833724] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:38 2019][1697212.840433] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697212.845492] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.851768] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697212.858390] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.864664] kworker/11:1 S ffffa07c2957b0c0 0 24841 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697212.871952] Call Trace: [Wed Feb 6 09:40:38 2019][1697212.874589] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:38 2019][1697212.880777] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697212.885924] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:38 2019][1697212.891679] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:38 2019][1697212.898386] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697212.903450] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.909725] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697212.916345] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.922620] kworker/u258:2 S ffffa06b7c0aa080 0 24880 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697212.929898] Call Trace: [Wed Feb 6 09:40:38 2019][1697212.932535] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:38 2019][1697212.938727] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697212.943870] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:38 2019][1697212.949627] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:38 2019][1697212.956332] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697212.961415] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.967686] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697212.974309] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697212.980581] kworker/31:0 S ffffa08b60d70000 0 24881 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697212.987861] Call Trace: [Wed Feb 6 09:40:38 2019][1697212.990496] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697212.995647] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:38 2019][1697213.001402] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:38 2019][1697213.008110] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697213.013168] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697213.019446] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697213.026065] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697213.032338] kworker/42:2 S ffffa07b7a122080 0 24882 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697213.039611] Call Trace: [Wed Feb 6 09:40:38 2019][1697213.042247] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:38 2019][1697213.048437] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697213.053582] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:38 2019][1697213.059338] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:38 2019][1697213.066045] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697213.071108] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697213.077381] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697213.084005] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697213.090279] mdt_rdpg02_006 S ffffa07c29530000 0 24892 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697213.097554] Call Trace: [Wed Feb 6 09:40:38 2019][1697213.100185] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697213.105365] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:38 2019][1697213.112249] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:38 2019][1697213.118034] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:38 2019][1697213.124517] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:38 2019][1697213.132087] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:38 2019][1697213.137146] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697213.143423] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:38 2019][1697213.150044] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:38 2019][1697213.156318] kworker/u256:3 S ffffa08a30211040 0 24893 2 0x00000080 [Wed Feb 6 09:40:38 2019][1697213.163598] Call Trace: [Wed Feb 6 09:40:38 2019][1697213.166234] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:38 2019][1697213.172424] [] schedule+0x29/0x70 [Wed Feb 6 09:40:38 2019][1697213.177570] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:38 2019][1697213.183326] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:39 2019][1697213.190031] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697213.195096] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.201367] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697213.207992] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.214266] mdt_rdpg00_007 S ffffa08a30212080 0 24894 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697213.221547] Call Trace: [Wed Feb 6 09:40:39 2019][1697213.224179] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697213.229363] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:39 2019][1697213.236244] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:39 2019][1697213.242030] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:39 2019][1697213.248513] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:39 2019][1697213.256083] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697213.261142] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.267420] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697213.274041] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.280312] mdt_rdpg00_008 S ffffa08a30216180 0 24897 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697213.287589] Call Trace: [Wed Feb 6 09:40:39 2019][1697213.290221] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697213.295400] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:39 2019][1697213.302279] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:39 2019][1697213.308064] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:39 2019][1697213.314547] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:39 2019][1697213.322123] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697213.327186] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.333461] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697213.340079] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.346356] mdt_rdpg00_009 S ffffa08a30214100 0 24902 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697213.353629] Call Trace: [Wed Feb 6 09:40:39 2019][1697213.356260] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697213.361441] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:39 2019][1697213.368316] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:39 2019][1697213.374104] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:39 2019][1697213.380580] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:39 2019][1697213.388157] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697213.393216] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.399495] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697213.406112] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.412387] kworker/22:0 S ffffa06c299eb0c0 0 24913 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697213.419668] Call Trace: [Wed Feb 6 09:40:39 2019][1697213.422303] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:39 2019][1697213.428492] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697213.433641] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:39 2019][1697213.439395] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:39 2019][1697213.446100] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697213.451164] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.457438] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697213.464081] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.470348] kworker/2:0 S ffffa07b7a1230c0 0 24915 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697213.477630] Call Trace: [Wed Feb 6 09:40:39 2019][1697213.480267] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697213.485417] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:39 2019][1697213.491167] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:39 2019][1697213.497898] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697213.502955] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.509232] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697213.515875] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.522150] kworker/34:0 S ffffa07b7a126180 0 24916 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697213.529431] Call Trace: [Wed Feb 6 09:40:39 2019][1697213.532067] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:39 2019][1697213.538254] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697213.543401] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:39 2019][1697213.549160] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:39 2019][1697213.555865] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697213.560929] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.567203] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697213.573825] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.580098] kworker/26:0 S ffffa07529900000 0 24918 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697213.587386] Call Trace: [Wed Feb 6 09:40:39 2019][1697213.590021] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697213.595192] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:39 2019][1697213.600940] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:39 2019][1697213.607651] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697213.612712] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.618987] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697213.625606] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.631883] kworker/10:0 S ffffa07b7c882080 0 24930 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697213.639160] Call Trace: [Wed Feb 6 09:40:39 2019][1697213.641787] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697213.646938] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:39 2019][1697213.652692] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:39 2019][1697213.659401] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697213.664460] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.670737] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697213.677359] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.683632] kworker/20:0 S ffffa05b797330c0 0 24961 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697213.690931] Call Trace: [Wed Feb 6 09:40:39 2019][1697213.693565] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:39 2019][1697213.699753] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697213.704899] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:39 2019][1697213.710656] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:39 2019][1697213.717362] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697213.722424] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.728700] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697213.735322] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.741618] kworker/17:0 S ffffa05c29968000 0 24962 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697213.748893] Call Trace: [Wed Feb 6 09:40:39 2019][1697213.751527] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697213.756676] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:39 2019][1697213.762430] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:39 2019][1697213.769139] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697213.774199] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.780477] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697213.787095] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.793369] kworker/1:1 S ffffa05c2996b0c0 0 24973 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697213.800652] Call Trace: [Wed Feb 6 09:40:39 2019][1697213.803287] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:39 2019][1697213.809473] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697213.814620] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:39 2019][1697213.820379] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:39 2019][1697213.827082] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697213.832146] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.838419] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697213.845063] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.851331] kworker/19:2 S ffffa086d5c5a080 0 24987 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697213.858614] Call Trace: [Wed Feb 6 09:40:39 2019][1697213.861248] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:39 2019][1697213.867456] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697213.872603] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:39 2019][1697213.878359] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:39 2019][1697213.885063] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697213.890128] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.896401] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697213.903024] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.909297] kworker/38:2 S ffffa06c298db0c0 0 24998 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697213.916593] Call Trace: [Wed Feb 6 09:40:39 2019][1697213.919227] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:39 2019][1697213.925420] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697213.930566] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:39 2019][1697213.936321] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:39 2019][1697213.943029] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697213.948093] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.954365] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697213.960986] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697213.967260] kworker/u256:0 S ffffa07b42bd8000 0 25012 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697213.974566] Call Trace: [Wed Feb 6 09:40:39 2019][1697213.977203] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:39 2019][1697213.983391] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697213.988538] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:39 2019][1697213.994292] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:39 2019][1697214.001001] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697214.006085] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697214.012353] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697214.018977] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697214.025277] kworker/15:0 S ffffa08b75829040 0 25013 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697214.032563] Call Trace: [Wed Feb 6 09:40:39 2019][1697214.035198] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:39 2019][1697214.041389] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697214.046535] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:39 2019][1697214.052291] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:39 2019][1697214.058997] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697214.064060] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697214.070334] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697214.076957] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697214.083230] kworker/33:2 S ffffa06b786b9040 0 25047 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697214.090511] Call Trace: [Wed Feb 6 09:40:39 2019][1697214.093144] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:39 2019][1697214.099335] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697214.104483] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:39 2019][1697214.110239] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:39 2019][1697214.116945] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697214.122008] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697214.128304] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:39 2019][1697214.134920] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697214.141194] kworker/0:0 S ffffa05b69e86180 0 25065 2 0x00000080 [Wed Feb 6 09:40:39 2019][1697214.148474] Call Trace: [Wed Feb 6 09:40:39 2019][1697214.151109] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:39 2019][1697214.157319] [] schedule+0x29/0x70 [Wed Feb 6 09:40:39 2019][1697214.162461] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:39 2019][1697214.168219] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:39 2019][1697214.174924] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:39 2019][1697214.179988] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:39 2019][1697214.186261] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:40 2019][1697214.192884] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:40 2019][1697214.199158] kworker/45:1 S ffffa06b03cb0000 0 25099 2 0x00000080 [Wed Feb 6 09:40:40 2019][1697214.206438] Call Trace: [Wed Feb 6 09:40:40 2019][1697214.209074] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:40 2019][1697214.215261] [] schedule+0x29/0x70 [Wed Feb 6 09:40:40 2019][1697214.220408] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:40 2019][1697214.226163] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:40 2019][1697214.232872] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:40 2019][1697214.237934] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:40 2019][1697214.244206] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:40 2019][1697214.250830] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:40 2019][1697214.257103] kworker/11:0 S ffffa08b784c4100 0 25136 2 0x00000080 [Wed Feb 6 09:40:40 2019][1697214.264393] Call Trace: [Wed Feb 6 09:40:40 2019][1697214.267025] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:40 2019][1697214.273218] [] schedule+0x29/0x70 [Wed Feb 6 09:40:40 2019][1697214.278363] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:40 2019][1697214.284119] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:40 2019][1697214.290827] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:40 2019][1697214.295889] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:40 2019][1697214.302162] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:40 2019][1697214.308784] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:40 2019][1697214.315057] kworker/u256:1 S ffffa05b74eee180 0 25137 2 0x00000080 [Wed Feb 6 09:40:40 2019][1697214.322347] Call Trace: [Wed Feb 6 09:40:40 2019][1697214.324983] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:40 2019][1697214.331172] [] schedule+0x29/0x70 [Wed Feb 6 09:40:40 2019][1697214.336321] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:40 2019][1697214.342073] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:40 2019][1697214.348780] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:40 2019][1697214.353843] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:40 2019][1697214.360116] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:40 2019][1697214.366741] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:40 2019][1697214.373037] sshd S ffffa06b7a34d140 0 25138 18216 0x00000080 [Wed Feb 6 09:40:40 2019][1697214.380313] Call Trace: [Wed Feb 6 09:40:40 2019][1697214.382946] [] schedule+0x29/0x70 [Wed Feb 6 09:40:40 2019][1697214.388095] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:40:40 2019][1697214.395322] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:40:40 2019][1697214.401169] [] ? tty_write_room+0x20/0x30 [Wed Feb 6 09:40:40 2019][1697214.407004] [] ? n_tty_poll+0x1b7/0x1d0 [Wed Feb 6 09:40:40 2019][1697214.412674] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:40 2019][1697214.419208] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:40:40 2019][1697214.425483] [] do_select+0x6d1/0x7c0 [Wed Feb 6 09:40:40 2019][1697214.430891] [] ? ip_finish_output+0x284/0x7b0 [Wed Feb 6 09:40:40 2019][1697214.437078] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:40 2019][1697214.442573] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:40 2019][1697214.448069] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:40 2019][1697214.453561] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:40 2019][1697214.459060] [] ? check_preempt_wakeup+0x148/0x250 [Wed Feb 6 09:40:40 2019][1697214.465596] [] ? tracing_is_on+0x15/0x30 [Wed Feb 6 09:40:40 2019][1697214.471347] [] ? tracing_record_cmdline+0x1d/0x120 [Wed Feb 6 09:40:40 2019][1697214.477968] [] ? probe_sched_wakeup+0x2b/0xa0 [Wed Feb 6 09:40:40 2019][1697214.484157] [] ? tracing_is_on+0x15/0x30 [Wed Feb 6 09:40:40 2019][1697214.489913] [] ? tracing_record_cmdline+0x1d/0x120 [Wed Feb 6 09:40:40 2019][1697214.496533] [] ? probe_sched_wakeup+0x2b/0xa0 [Wed Feb 6 09:40:40 2019][1697214.502720] [] ? ttwu_do_wakeup+0xb5/0xe0 [Wed Feb 6 09:40:40 2019][1697214.508564] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:40:40 2019][1697214.514576] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:40 2019][1697214.521025] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:40 2019][1697214.527040] [] core_sys_select+0x1ff/0x340 [Wed Feb 6 09:40:40 2019][1697214.532970] [] ? __wake_up+0x44/0x50 [Wed Feb 6 09:40:40 2019][1697214.538374] [] ? tty_write_unlock+0x31/0x40 [Wed Feb 6 09:40:40 2019][1697214.544392] [] ? tty_ldisc_deref+0x19/0x20 [Wed Feb 6 09:40:40 2019][1697214.550316] [] ? tty_write+0x1eb/0x2c0 [Wed Feb 6 09:40:40 2019][1697214.555899] [] ? n_tty_ioctl+0xe0/0xe0 [Wed Feb 6 09:40:40 2019][1697214.561481] [] SyS_select+0xca/0x120 [Wed Feb 6 09:40:40 2019][1697214.566891] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:40 2019][1697214.573078] bash R running task 0 25140 25138 0x00000088 [Wed Feb 6 09:40:40 2019][1697214.580351] Call Trace: [Wed Feb 6 09:40:40 2019][1697214.582983] [] sched_show_task+0xa8/0x110 [Wed Feb 6 09:40:40 2019][1697214.588824] [] show_state_filter+0x82/0xd0 [Wed Feb 6 09:40:40 2019][1697214.594753] [] sysrq_handle_showstate+0x10/0x20 [Wed Feb 6 09:40:40 2019][1697214.601115] [] __handle_sysrq+0x10d/0x170 [Wed Feb 6 09:40:40 2019][1697214.606955] [] write_sysrq_trigger+0x28/0x40 [Wed Feb 6 09:40:40 2019][1697214.613060] [] proc_reg_write+0x40/0x80 [Wed Feb 6 09:40:40 2019][1697214.618723] [] vfs_write+0xc0/0x1f0 [Wed Feb 6 09:40:40 2019][1697214.624049] [] SyS_write+0x7f/0xf0 [Wed Feb 6 09:40:40 2019][1697214.629281] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:40 2019][1697214.635474] Sched Debug Version: v0.11, 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 09:40:40 2019][1697214.642609] ktime : 1697200109.747522 [Wed Feb 6 09:40:40 2019][1697214.649405] sched_clk : 1697214635.473047 [Wed Feb 6 09:40:40 2019][1697214.656198] cpu_clk : 1697214635.473077 [Wed Feb 6 09:40:40 2019][1697214.662995] jiffies : 5991867434 [Wed Feb 6 09:40:40 2019][1697214.669181] sched_clock_stable() : 1 [Wed Feb 6 09:40:40 2019][1697214.674588] [Wed Feb 6 09:40:40 2019][1697214.676260] sysctl_sched [Wed Feb 6 09:40:40 2019][1697214.678976] .sysctl_sched_latency : 24.000000 [Wed Feb 6 09:40:40 2019][1697214.685338] .sysctl_sched_min_granularity : 10.000000 [Wed Feb 6 09:40:40 2019][1697214.691697] .sysctl_sched_wakeup_granularity : 4.000000 [Wed Feb 6 09:40:40 2019][1697214.697973] .sysctl_sched_child_runs_first : 0 [Wed Feb 6 09:40:40 2019][1697214.703641] .sysctl_sched_features : 56955 [Wed Feb 6 09:40:40 2019][1697214.709654] .sysctl_sched_tunable_scaling : 1 (logaritmic) [Wed Feb 6 09:40:40 2019][1697214.716441] [Wed Feb 6 09:40:40 2019][1697214.718125] cpu#0, 1996.233 MHz [Wed Feb 6 09:40:40 2019][1697214.721450] .nr_running : 0 [Wed Feb 6 09:40:40 2019][1697214.726252] .load : 1024 [Wed Feb 6 09:40:40 2019][1697214.731315] .nr_switches : 1219912214 [Wed Feb 6 09:40:40 2019][1697214.736896] .nr_load_updates : 266307622 [Wed Feb 6 09:40:40 2019][1697214.742389] .nr_uninterruptible : 117864 [Wed Feb 6 09:40:40 2019][1697214.747624] .next_balance : 5991.867519 [Wed Feb 6 09:40:40 2019][1697214.753294] .curr->pid : 0 [Wed Feb 6 09:40:40 2019][1697214.758092] .clock : 1697214758.088303 [Wed Feb 6 09:40:40 2019][1697214.764272] .cpu_load[0] : 0 [Wed Feb 6 09:40:40 2019][1697214.769065] .cpu_load[1] : 0 [Wed Feb 6 09:40:40 2019][1697214.773858] .cpu_load[2] : 0 [Wed Feb 6 09:40:40 2019][1697214.778650] .cpu_load[3] : 0 [Wed Feb 6 09:40:40 2019][1697214.783443] .cpu_load[4] : 0 [Wed Feb 6 09:40:40 2019][1697214.788235] .avg_idle : 4447726 [Wed Feb 6 09:40:40 2019][1697214.793550] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:40 2019][1697214.798870] [Wed Feb 6 09:40:40 2019][1697214.800540] cfs_rq[0]:/ [Wed Feb 6 09:40:40 2019][1697214.803164] .exec_clock : 0.000000 [Wed Feb 6 09:40:40 2019][1697214.808564] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:40 2019][1697214.813964] .min_vruntime : 8843278.919244 [Wed Feb 6 09:40:40 2019][1697214.819882] .max_vruntime : 0.000001 [Wed Feb 6 09:40:40 2019][1697214.825282] .spread : 0.000000 [Wed Feb 6 09:40:40 2019][1697214.830682] .spread0 : 0.000000 [Wed Feb 6 09:40:40 2019][1697214.836080] .nr_spread_over : 0 [Wed Feb 6 09:40:40 2019][1697214.840873] .nr_running : 0 [Wed Feb 6 09:40:40 2019][1697214.845666] .load : 0 [Wed Feb 6 09:40:40 2019][1697214.850460] .runnable_load_avg : 0 [Wed Feb 6 09:40:40 2019][1697214.855253] .blocked_load_avg : 0 [Wed Feb 6 09:40:40 2019][1697214.860043] .tg_load_avg : 0 [Wed Feb 6 09:40:40 2019][1697214.864837] .tg_load_contrib : 0 [Wed Feb 6 09:40:40 2019][1697214.869629] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:40 2019][1697214.874422] .tg->runnable_avg : 0 [Wed Feb 6 09:40:40 2019][1697214.879215] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:40 2019][1697214.884008] .throttled : 0 [Wed Feb 6 09:40:40 2019][1697214.888800] .throttle_count : 0 [Wed Feb 6 09:40:40 2019][1697214.893594] .avg->runnable_avg_sum : 80 [Wed Feb 6 09:40:40 2019][1697214.898474] .avg->runnable_avg_period : 47236 [Wed Feb 6 09:40:40 2019][1697214.903619] [Wed Feb 6 09:40:40 2019][1697214.905294] rt_rq[0]:/ [Wed Feb 6 09:40:40 2019][1697214.907833] .rt_nr_running : 0 [Wed Feb 6 09:40:40 2019][1697214.912626] .rt_throttled : 0 [Wed Feb 6 09:40:40 2019][1697214.917417] .rt_time : 0.007745 [Wed Feb 6 09:40:40 2019][1697214.922816] .rt_runtime : 979.166667 [Wed Feb 6 09:40:40 2019][1697214.928390] [Wed Feb 6 09:40:40 2019][1697214.930064] runnable tasks: [Wed Feb 6 09:40:40 2019][1697214.933035] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:40 2019][1697214.943982] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:40 2019][1697214.954842] ksoftirqd/0 3 8840287.685415 3341797 120 0.000000 39966.433883 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697214.966177] kworker/0:0H 5 5754.153321 8 100 0.000000 0.041238 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697214.977513] migration/0 8 0.000000 1708 0 0.000000 209712.960658 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697214.988850] rcu_bh 9 513.816375 2 120 0.000000 0.001613 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.000187] lru-add-drain 11 517.818656 2 100 0.000000 0.002686 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.011522] watchdog/0 12 0.000000 424297 0 0.000000 70465.709498 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.022892] irq/26-AMD-Vi 270 0.000000 3 49 0.000000 0.041027 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.034230] kswapd0 278 3056.471186 4 120 0.000000 0.052900 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.045566] kworker/0:2 299 8843263.311904 329919 120 0.000000 19421.798371 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.056908] kworker/u262:0 7437 4931.366578 4 100 0.000000 0.015459 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.068251] kworker/0:1H 7833 8839940.371820 2767133 100 0.000000 13592.342321 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.079582] systemd-journal 7941 8843278.919244 161644 120 0.000000 38531.556019 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.090948] ptlrpc_hr00_000 126379 8843265.167817 15287033 120 0.000000 124816.059877 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.102374] ptlrpcd_00_02 126443 8843266.922501 4279899 120 0.000000 69104.055895 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.113817] mdt_seqm_0001 126662 8127448.495136 1062 120 0.000000 9.562087 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.125240] mdt_io00_002 126667 8677714.789768 1365129 120 0.000000 52560.205597 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.136680] osp-syn-26-1 126907 8843266.394745 198494 120 0.000000 4689.867555 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.148106] osp-syn-44-1 126931 8843266.391778 197019 120 0.000000 4885.072916 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.159534] ldlm_cn00_003 127202 8843089.522789 4065174 120 0.000000 203121.730999 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.170957] mdt_io00_003 127220 6369842.999775 959461 120 0.000000 34470.904910 0.000000 0 / [Wed Feb 6 09:40:40 2019][1697215.182387] mdt_io00_012 127656 8841850.061930 2439441 120 0.000000 109459.239722 0.000000 0 / [Wed Feb 6 09:40:41 2019][1697215.193812] mdt_io00_013 127700 5476604.164956 2157343 120 0.000000 85640.146118 0.000000 0 / [Wed Feb 6 09:40:41 2019][1697215.205237] ldlm_cb00_002 128539 6722143.749596 8930 120 0.000000 202.897285 0.000000 0 / [Wed Feb 6 09:40:41 2019][1697215.216673] ldlm_cn00_029 130343 8843082.855964 3004089 120 0.000000 185557.644846 0.000000 0 / [Wed Feb 6 09:40:41 2019][1697215.228122] mdt00_036 17900 8843255.831085 197403 120 0.000000 165334.066618 0.000000 0 / [Wed Feb 6 09:40:41 2019][1697215.239463] mdt00_052 17971 8721120.827676 71140 120 0.000000 12745.464356 0.000000 0 / [Wed Feb 6 09:40:41 2019][1697215.250802] kworker/u257:2 22262 7820594.304104 2 120 0.000000 0.011962 0.000000 0 / [Wed Feb 6 09:40:41 2019][1697215.262152] kworker/0:0 25065 8793201.326862 8 120 0.000000 0.066797 0.000000 0 / [Wed Feb 6 09:40:41 2019][1697215.273494] [Wed Feb 6 09:40:41 2019][1697215.275169] cpu#1, 1996.233 MHz [Wed Feb 6 09:40:41 2019][1697215.278490] .nr_running : 0 [Wed Feb 6 09:40:41 2019][1697215.283291] .load : 0 [Wed Feb 6 09:40:41 2019][1697215.288094] .nr_switches : 751676680 [Wed Feb 6 09:40:41 2019][1697215.293587] .nr_load_updates : 169799673 [Wed Feb 6 09:40:41 2019][1697215.299075] .nr_uninterruptible : -44594 [Wed Feb 6 09:40:41 2019][1697215.304310] .next_balance : 5991.868075 [Wed Feb 6 09:40:41 2019][1697215.309977] .curr->pid : 0 [Wed Feb 6 09:40:41 2019][1697215.314780] .clock : 1697215314.732853 [Wed Feb 6 09:40:41 2019][1697215.320967] .cpu_load[0] : 0 [Wed Feb 6 09:40:41 2019][1697215.325769] .cpu_load[1] : 0 [Wed Feb 6 09:40:41 2019][1697215.330568] .cpu_load[2] : 0 [Wed Feb 6 09:40:41 2019][1697215.335361] .cpu_load[3] : 0 [Wed Feb 6 09:40:41 2019][1697215.340153] .cpu_load[4] : 0 [Wed Feb 6 09:40:41 2019][1697215.344948] .avg_idle : 358852 [Wed Feb 6 09:40:41 2019][1697215.350183] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:41 2019][1697215.355504] [Wed Feb 6 09:40:41 2019][1697215.357180] cfs_rq[1]:/ [Wed Feb 6 09:40:41 2019][1697215.359807] .exec_clock : 0.000000 [Wed Feb 6 09:40:41 2019][1697215.365204] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:41 2019][1697215.370604] .min_vruntime : 8863489.977807 [Wed Feb 6 09:40:41 2019][1697215.376524] .max_vruntime : 0.000001 [Wed Feb 6 09:40:41 2019][1697215.381924] .spread : 0.000000 [Wed Feb 6 09:40:41 2019][1697215.387322] .spread0 : 20208.967135 [Wed Feb 6 09:40:41 2019][1697215.393069] .nr_spread_over : 0 [Wed Feb 6 09:40:41 2019][1697215.397862] .nr_running : 0 [Wed Feb 6 09:40:41 2019][1697215.402654] .load : 0 [Wed Feb 6 09:40:41 2019][1697215.407447] .runnable_load_avg : 0 [Wed Feb 6 09:40:41 2019][1697215.412239] .blocked_load_avg : 16 [Wed Feb 6 09:40:41 2019][1697215.417119] .tg_load_avg : 0 [Wed Feb 6 09:40:41 2019][1697215.421912] .tg_load_contrib : 0 [Wed Feb 6 09:40:41 2019][1697215.426704] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:41 2019][1697215.431498] .tg->runnable_avg : 0 [Wed Feb 6 09:40:41 2019][1697215.436288] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:41 2019][1697215.441081] .throttled : 0 [Wed Feb 6 09:40:41 2019][1697215.445876] .throttle_count : 0 [Wed Feb 6 09:40:41 2019][1697215.450668] .avg->runnable_avg_sum : 893 [Wed Feb 6 09:40:41 2019][1697215.455633] .avg->runnable_avg_period : 46855 [Wed Feb 6 09:40:41 2019][1697215.460776] [Wed Feb 6 09:40:41 2019][1697215.462447] rt_rq[1]:/ [Wed Feb 6 09:40:41 2019][1697215.464984] .rt_nr_running : 0 [Wed Feb 6 09:40:41 2019][1697215.469777] .rt_throttled : 0 [Wed Feb 6 09:40:41 2019][1697215.474571] .rt_time : 0.007955 [Wed Feb 6 09:40:41 2019][1697215.479969] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:41 2019][1697215.485629] [Wed Feb 6 09:40:41 2019][1697215.487302] runnable tasks: [Wed Feb 6 09:40:41 2019][1697215.490276] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:41 2019][1697215.501221] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:41 2019][1697215.512083] watchdog/1 13 -2.961978 424289 0 0.000000 58778.198033 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.523417] migration/1 14 0.000000 2155 0 0.000000 270497.241475 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.534752] ksoftirqd/1 15 8863466.547432 5280373 120 0.000000 53538.107692 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.546088] kworker/1:0H 17 6633.635180 13 100 0.000000 0.046005 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.557460] kworker/u263:0 7438 4571.588710 4 100 0.000000 0.016571 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.568801] multipathd 11653 0.000000 202 0 0.000000 19.286054 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.580134] ccp-1-q2 9977 5135.136368 2 120 0.000000 0.010229 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.591484] kworker/1:1H 18109 8863466.452668 3963737 100 0.000000 21354.972619 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.602826] agetty 18466 6668.278950 15 120 0.000000 17.680943 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.614170] kiblnd_sd_01_00 126425 8863494.243778 816229022 120 0.000000 5793661.889353 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.625593] ptlrpcd_01_00 126453 8863466.640622 4004086 120 0.000000 62614.697504 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.637013] ptlrpcd_01_05 126458 8863466.600335 3991258 120 0.000000 59170.423423 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.648435] ptlrpcd_01_09 126462 8863466.606898 3977933 120 0.000000 56567.671375 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.659862] ldlm_cb01_000 126588 44933.031088 3 120 0.000000 0.050425 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.671281] ldlm_cb01_001 126589 44933.047058 3 120 0.000000 0.026089 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.682708] mdt_rdpg01_001 126638 8863314.329691 3720380 120 0.000000 201716.440613 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.694125] mdt_attr01_000 126645 44933.368472 3 120 0.000000 0.069351 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.705550] mdt_attr01_001 126646 44933.390273 3 120 0.000000 0.031700 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.716974] osp-syn-13-3 126706 8863298.857173 126534 120 0.000000 3550.181697215.728395] osp-syn-11-3 126714 8863416.802367 125730 120 0.000000 4196.657420 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.739817] osp-syn-31-3 126720 8863379.016700 128630 120 0.000000 3590.928193 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.751247] osp-pre-0-1 126889 8863445.834355 205489 120 0.000000 4925.535591 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.762675] osp-pre-9-1 126914 8863440.973857 213806 120 0.000000 7281.741697 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.774097] osp-pre-40-1 126952 8863444.745020 213935 120 0.000000 9018.051996 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.785528] mdt_rdpg01_003 127229 8843600.728089 8404413 120 0.000000 327384.654228 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.796951] mdt_io01_012 127363 3834246.139286 3214376 120 0.000000 110133.169576 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.808374] mdt_io01_014 127691 8863484.181645 2459301 120 0.000000 92332.206006 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.819803] mdt_io01_021 129240 8863262.863665 2438558 120 0.000000 121380.994259 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.831232] mdt_io01_024 129283 3834246.137862 2668641 120 0.000000 115898.332940 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.842662] ldlm_cn01_014 4337 8863485.893581 5727809 120 0.000000 298212.260607 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.853993] mdt_rdpg01_008 18321 8863485.965954 129887 120 0.000000 9792.360799 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.865329] mdt_rdpg01_009 18322 8843600.724733 2550086 120 0.000000 111603.802407 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.876671] kworker/1:0 4055 8863475.554742 65991 120 0.000000 44727.122813 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.888029] mdt01_062 19136 8863496.504909 25386 120 0.000000 12119.360073 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.899376] kworker/1:1 24973 8828680.435827 13 120 0.000000 0.509258 0.000000 1 / [Wed Feb 6 09:40:41 2019][1697215.910723] [Wed Feb 6 09:40:41 2019][1697215.912404] cpu#2, 1996.233 MHz [Wed Feb 6 09:40:41 2019][1697215.915734] .nr_running : 0 [Wed Feb 6 09:40:41 2019][1697215.920534] .load : 1024 [Wed Feb 6 09:40:41 2019][1697215.925598] .nr_switches : 561355005 [Wed Feb 6 09:40:41 2019][1697215.931090] .nr_load_updates : 173805045 [Wed Feb 6 09:40:41 2019][1697215.936585] .nr_uninterruptible : -26320 [Wed Feb 6 09:40:41 2019][1697215.941821] .next_balance : 5991.868715 [Wed Feb 6 09:40:41 2019][1697215.947487] .curr->pid : 0 [Wed Feb 6 09:40:41 2019][1697215.952281] .clock : 1697215952.213798 [Wed Feb 6 09:40:41 2019][1697215.958469] .cpu_load[0] : 0 [Wed Feb 6 09:40:41 2019][1697215.963268] .cpu_load[1] : 0 [Wed Feb 6 09:40:41 2019][1697215.968063] .cpu_load[2] : 0 [Wed Feb 6 09:40:41 2019][1697215.972864] .cpu_load[3] : 0 [Wed Feb 6 09:40:41 2019][1697215.977667] .cpu_load[4] : 0 [Wed Feb 6 09:40:41 2019][1697215.982467] .avg_idle : 817933 [Wed Feb 6 09:40:41 2019][1697215.987701] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:41 2019][1697215.993023] [Wed Feb 6 09:40:41 2019][1697215.994699] cfs_rq[2]:/ [Wed Feb 6 09:40:41 2019][1697215.997324] .exec_clock : 0.000000 [Wed Feb 6 09:40:41 2019][1697216.002724] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:41 2019][1697216.008124] .min_vruntime : 4887648.309060 [Wed Feb 6 09:40:41 2019][1697216.014042] .max_vruntime : 0.000001 [Wed Feb 6 09:40:41 2019][1697216.019444] .spread : 0.000000 [Wed Feb 6 09:40:41 2019][1697216.024841] .spread0 : -3955635.037061 [Wed Feb 6 09:40:41 2019][1697216.030849] .nr_spread_over : 0 [Wed Feb 6 09:40:41 2019][1697216.035642] .nr_running : 0 [Wed Feb 6 09:40:41 2019][1697216.040433] .load : 0 [Wed Feb 6 09:40:41 2019][1697216.045227] .runnable_load_avg : 0 [Wed Feb 6 09:40:41 2019][1697216.050017] .blocked_load_avg : 20 [Wed Feb 6 09:40:41 2019][1697216.054897] .tg_load_avg : 0 [Wed Feb 6 09:40:41 2019][1697216.059690] .tg_load_contrib : 0 [Wed Feb 6 09:40:41 2019][1697216.064485] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:41 2019][1697216.069278] .tg->runnable_avg : 0 [Wed Feb 6 09:40:41 2019][1697216.074069] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:41 2019][1697216.078862] .throttled : 0 [Wed Feb 6 09:40:41 2019][1697216.083654] .throttle_count : 0 [Wed Feb 6 09:40:41 2019][1697216.088447] .avg->runnable_avg_sum : 748 [Wed Feb 6 09:40:41 2019][1697216.093414] .avg->runnable_avg_period : 48048 [Wed Feb 6 09:40:41 2019][1697216.098556] [Wed Feb 6 09:40:41 2019][1697216.100226] rt_rq[2]:/ [Wed Feb 6 09:40:41 2019][1697216.102765] .rt_nr_running : 0 [Wed Feb 6 09:40:41 2019][1697216.107556] .rt_throttled : 0 [Wed Feb 6 09:40:41 2019][1697216.112350] .rt_time : 0.000000 [Wed Feb 6 09:40:41 2019][1697216.117748] .rt_runtime : 775.271077 [Wed Feb 6 09:40:41 2019][1697216.123323] [Wed Feb 6 09:40:41 2019][1697216.124994] runnable tasks: [Wed Feb 6 09:40:41 2019][1697216.127967] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:41 2019][1697216.138914] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:41 2019][1697216.149777] watchdog/2 19 -5.968491 424276 0 0.000000 99573.146582 0.000000 2 / [Wed Feb 6 09:40:41 2019][1697216.161109] migration/2 20 0.000000 1609 0 0.000000 252652.079485 0.000000 2 / [Wed Feb 6 09:40:41 2019][1697216.172445] ksoftirqd/2 21 4887634.732183 2483634 120 0.000000 42435.830484 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.183781] kworker/2:0H 23 3640.958900 10 100 0.000000 0.035658 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.195147] irq/28-AMD-Vi 272 0.000000 3 49 0.000000 0.029926 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.206481] kswapd2 280 307.764935 4 120 0.000000 0.038923 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.217821] scsi_eh_2 7309 2105.193138 22 120 0.000000 5.623542 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.229172] writer#3 18229 4887624.328718 25041703 120 0.000000 228794.618314 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.240505] kworker/2:1H 18373 4887638.663203 2110190 100 0.000000 8832.936000 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.251844] slapd 18608 2854918.786047 3039 120 0.000000 9448.629051 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.263185] ptlrpc_hr02_002 126394 4887634.713047 15299413 120 0.000000 124980.840278 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.274619] ptlrpcd_02_01 126466 4887633.454351 6827949 120 0.000000 82483.699439 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.286040] ptlrpcd_02_03 126468 4887634.254886 6779718 120 0.000000 82124.424734 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.297463] ptlrpcd_02_11 126476 4887633.449573 6786566 120 0.000000 81938.710399 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.308891] ldlm_cb02_000 126590 4848882.635345 33060878 120 0.000000 477977.939336 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.320314] mdt_rdpg02_001 126640 4881677.869798 6654241 120 0.000000 282959.745104 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.331733] mdt_attr02_000 126647 200329.998913 3 120 0.000000 0.066586 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.343156] mdt_attr02_001 126648 200330.020824 3 120 0.000000 0.031789 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.354578] mdt_out02_000 126655 4827981.143250 15884473 120 0.000000 250214.404633 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.366003] mdt_io02_001 126672 4826467.495373 1126478 120 0.000000 52628.230234 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.377425] osp-pre-43-3 126729 4887624.353555 214719 120 0.000000 7513.064338 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.388856] osp-syn-46-1 126899 4887633.449632 196922 120 0.000000 5606.352022 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.400281] osp-pre-41-1 126940 4887625.371698 213962 120 0.000000 4032.403802 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.411699] osp-pre-4-1 126942 4887624.340940 213949 120 0.000000 8028.850568 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.423123] osp-syn-23-1 126943 4887633.456845 198622 120 0.000000 4826.802198 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.434548] osp-pre-1-1 126968 4887634.269102 213791 120 0.000000 7327.654454 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.445973] mdt02_003 127133 4828427.581580 25576666 120 0.000000 1114832.354132 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.457393] mdt02_005 127198 4882187.437459 26396663 120 0.000000 1720809.810511 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.468821] mdt02_008 127248 4887624.356210 26220976 120 0.000000 1170686.315295 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.480246] mdt_rdpg02_003 127249 4887651.543848 6237178 120 0.000000 282787.995866 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.491670] mdt_io02_007 127264 4887507.855040 2581597 120 0.000000 133593.777475 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.503095] mdt_io02_016 127693 4825575.743039 2310939 120 0.000000 103970.148048 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.514515] mdt_io02_017 127699 4825575.705778 2223728 120 0.000000 102256.546349 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.525939] ldlm_cn02_006 127802 4887480.335517 5778732 120 0.000000 321703.225686 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.537363] mdt_rdpg02_004 128524 4882000.407276 8770029 120 0.000000 391941.130930 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.548785] mdt02_013 129246 4887499.096850 24141876 120 0.000000 1042588.906250 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.560206] mdt02_015 129248 4887624.345129 27076044 120 0.000000 1251128.303142 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.571632] mdt_io02_022 129309 4887640.000899 1490958 120 0.000000 74501.776807 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.583062] kworker/2:1 16820 4887638.595628 159413 120 0.000000 15674.680121 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.594412] mdt02_032 17849 4882187.441175 174732 120 0.000000 165901.371975 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.605749] mdt02_033 17855 4887650.279044 99025 120 0.000000 163826.717871 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.617094] mdt02_040 17993 4828427.581800 194469 120 0.000000 192261.458858 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.628436] mdt02_060 18678 4887624.373462 208750 120 0.000000 320012.437167 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.639768] kworker/u259:1 20339 4887628.102487 875 120 0.000000 2680.645633 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.651104] ldlm_cn02_017 23719 4887537.810292 19213 120 0.000000 95199.943875 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.662442] ldlm_cn02_026 23739 4887490.581487 25247 120 0.000000 81317.823440 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.673782] kworker/2:0 24915 4819931.927379 4 120 0.000000 0.030298 0.000000 2 / [Wed Feb 6 09:40:42 2019][1697216.685164] [Wed Feb 6 09:40:42 2019][1697216.686840] cpu#3, 1996.233 MHz [Wed Feb 6 09:40:42 2019][1697216.690163] .nr_running : 0 [Wed Feb 6 09:40:42 2019][1697216.694965] .load : 0 [Wed Feb 6 09:40:42 2019][1697216.699764] .nr_switches : 135926038 [Wed Feb 6 09:40:42 2019][1697216.705259] .nr_load_updates : 71810389 [Wed Feb 6 09:40:42 2019][1697216.710668] .nr_uninterruptible : -820 [Wed Feb 6 09:40:42 2019][1697216.715730] .next_balance : 5991.869489 [Wed Feb 6 09:40:42 2019][1697216.721398] .curr->pid : 0 [Wed Feb 6 09:40:42 2019][1697216.726199] .clock : 1697216725.749944 [Wed Feb 6 09:40:42 2019][1697216.732387] .cpu_load[0] : 0 [Wed Feb 6 09:40:42 2019][1697216.737187] .cpu_load[1] : 0 [Wed Feb 6 09:40:42 2019][1697216.741980] .cpu_load[2] : 0 [Wed Feb 6 09:40:42 2019][1697216.746782] .cpu_load[3] : 0 [Wed Feb 6 09:40:42 2019][1697216.751585] .cpu_load[4] : 0 [Wed Feb 6 09:40:42 2019][1697216.756384] .avg_idle : 1148193 [Wed Feb 6 09:40:42 2019][1697216.761705] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:42 2019][1697216.767028] [Wed Feb 6 09:40:42 2019][1697216.768704] cfs_rq[3]:/ [Wed Feb 6 09:40:42 2019][1697216.771329] .exec_clock : 0.000000 [Wed Feb 6 09:40:42 2019][1697216.776729] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:42 2019][1697216.782127] .min_vruntime : 3340860.417219 [Wed Feb 6 09:40:42 2019][1697216.788047] .max_vruntime : 0.000001 [Wed Feb 6 09:40:42 2019][1697216.793447] .spread : 0.000000 [Wed Feb 6 09:40:42 2019][1697216.798846] .spread0 : -5502425.524831 [Wed Feb 6 09:40:42 2019][1697216.804852] .nr_spread_over : 0 [Wed Feb 6 09:40:42 2019][1697216.809643] .nr_running : 0 [Wed Feb 6 09:40:42 2019][1697216.814437] .load : 0 [Wed Feb 6 09:40:42 2019][1697216.819230] .runnable_load_avg : 0 [Wed Feb 6 09:40:42 2019][1697216.824023] .blocked_load_avg : 2 [Wed Feb 6 09:40:42 2019][1697216.828816] .tg_load_avg : 0 [Wed Feb 6 09:40:42 2019][1697216.833608] .tg_load_contrib : 0 [Wed Feb 6 09:40:42 2019][1697216.838400] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:42 2019][1697216.843194] .tg->runnable_avg : 0 [Wed Feb 6 09:40:42 2019][1697216.847986] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:42 2019][1697216.852778] .throttled : 0 [Wed Feb 6 09:40:42 2019][1697216.857573] .throttle_count : 0 [Wed Feb 6 09:40:42 2019][1697216.862366] .avg->runnable_avg_sum : 197 [Wed Feb 6 09:40:42 2019][1697216.867332] .avg->runnable_avg_period : 46818 [Wed Feb 6 09:40:42 2019][1697216.872474] [Wed Feb 6 09:40:42 2019][1697216.874142] rt_rq[3]:/ [Wed Feb 6 09:40:42 2019][1697216.876682] .rt_nr_running : 0 [Wed Feb 6 09:40:42 2019][1697216.881474] .rt_throttled : 0 [Wed Feb 6 09:40:42 2019][1697216.886267] .rt_time : 0.000000 [Wed Feb 6 09:40:42 2019][1697216.891669] .rt_runtime : 774.865899 [Wed Feb 6 09:40:42 2019][1697216.897240] [Wed Feb 6 09:40:42 2019][1697216.898912] runnable tasks: [Wed Feb 6 09:40:42 2019][1697216.901885] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:42 2019][1697216.912830] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:42 2019][1697216.923693] watchdog/3 25 -5.962098 424304 0 0.000000 45083.281847 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697216.935027] migration/3 26 0.000000 1344 0 0.000000 158428.626676 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697216.946364] ksoftirqd/3 27 3340817.197101 840263 120 0.000000 8835.125761 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697216.957700] kworker/3:0H 29 4542.815613 10 100 0.000000 0.039654 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697216.969060] irq/29-AMD-Vi 273 0.000000 3 49 0.000000 0.035055 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697216.980398] kswapd3 281 292.468483 4 120 0.000000 0.041828 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697216.991735] scsi_eh_0 2262 573.074836 2 120 0.000000 0.011391 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.003087] irqbalance 16783 3340838.251739 209725 120 0.000000 1316630.266078 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.014431] crond 18411 3340758.524641 59910 120 0.000000 6996.428052 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.025771] kworker/3:1H 19395 3340817.186161 1086334 100 0.000000 4692.177427 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.037107] cfs_rh_03 126361 3340814.989128 909 120 0.000000 880.221262 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.048529] ptlrpc_hr03_005 126404 3340847.820500 1678749 120 0.000000 14388.051977 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.059953] kiblnd_sd_03_02 126435 3340861.747978 87883423 120 0.000000 652518.733530 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.071377] ptlrpcd_03_02 126479 3340845.738027 5636739 120 0.000000 87694.820311 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.082799] ldlm_cb03_000 126593 34389.541535 3 120 0.000000 0.044624 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.094217] ldlm_cb03_001 126594 34389.554279 3 120 0.000000 0.024166 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.105645] mdt_attr03_000 126649 34389.849855 3 120 0.000000 0.060905 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.117065] mdt_attr03_001 126650 34389.875403 3 120 0.000000 0.035888 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.128488] mdt_io03_002 126676 3340846.382019 171218 120 0.000000 6306.492556 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.139911] osp-pre-39-3 126709 3340834.169442 214713 120 0.000000 4912.160967 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.151337] osp-pre-4-3 126753 3340834.087687 214630 120 0.000000 6471.389970 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.162755] osp-pre-23-3 126755 3340834.463186 214648 120 0.000000 5556.741027 0.000000 3 / [Wed Feb 6 09:40:42 2019][1697217.174188] osp-pre-16-1 126938 3340834.093668 213930 120 0.000000 7851.052115 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.185612] osp-pre-15-1 126964 3340848.350164 213945 120 0.000000 7978.610166 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.197035] lquota_wb_fir-M 127048 3340738.089123 17159 120 0.000000 86.509612 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.208457] mdt03_005 127195 3340831.031552 8014967 120 0.000000 328814.470985 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.219885] mdt03_009 127327 3340807.866197 7848355 120 0.000000 329632.245937 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.231332] mdt_out03_002 1423 3340845.710986 8811 120 0.000000 3056.695611 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.242667] kworker/3:0 33438 3340844.699378 98818 120 0.000000 7402.790255 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.254001] ldlm_cn03_008 51825 3340774.266464 1926425 120 0.000000 163845.757113 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.265342] mdt_io03_010 12793 3127217.581070 1180 120 0.000000 28.439574 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.276680] ldlm_cn03_009 12924 3340783.481840 1441832 120 0.000000 128610.999888 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.288017] ldlm_cn03_011 12936 3340776.675291 1243207 120 0.000000 106028.187082 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.299355] mdt03_021 17827 3266758.294443 7091 120 0.000000 424.991337 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.310701] mdt03_046 18250 3266758.295266 13043 120 0.000000 152032.284394 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.322043] kworker/3:1 23777 2857749.692779 8 120 0.000000 0.041838 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.333378] ldlm_cn03_018 24532 3272476.346935 4711 120 0.000000 43378.061579 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.344713] ldlm_cn03_019 24582 3340774.267286 3229 120 0.000000 33578.992423 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.356049] ldlm_cn03_021 24584 3340708.510192 2696 120 0.000000 28445.527222 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.367386] ldlm_cn03_022 24585 3271079.611622 4966 120 0.000000 44447.306838 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.378721] ldlm_cn03_023 24586 3340786.267334 11827 120 0.000000 70669.325807 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.390059] ldlm_cn03_024 24587 3340708.512056 5997 120 0.000000 80131.348218 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.401395] ldlm_cn03_025 24588 3340780.442185 4942 120 0.000000 30508.534179 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.412733] ldlm_cn03_030 24612 3340708.512326 3704 120 0.000000 41714.999188 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.424067] ldlm_cn03_031 24613 3340708.512276 4935 120 0.000000 52469.543704 0.000000 3 / [Wed Feb 6 09:40:43 2019][1697217.435421] [Wed Feb 6 09:40:43 2019][1697217.437098] cpu#4, 1996.233 MHz [Wed Feb 6 09:40:43 2019][1697217.440419] .nr_running : 0 [Wed Feb 6 09:40:43 2019][1697217.445219] .load : 0 [Wed Feb 6 09:40:43 2019][1697217.450012] .nr_switches : 1799144229 [Wed Feb 6 09:40:43 2019][1697217.455595] .nr_load_updates : 337800686 [Wed Feb 6 09:40:43 2019][1697217.461088] .nr_uninterruptible : 63722 [Wed Feb 6 09:40:43 2019][1697217.466236] .next_balance : 5991.870239 [Wed Feb 6 09:40:43 2019][1697217.471904] .curr->pid : 0 [Wed Feb 6 09:40:43 2019][1697217.476708] .clock : 1697217475.750991 [Wed Feb 6 09:40:43 2019][1697217.482894] .cpu_load[0] : 0 [Wed Feb 6 09:40:43 2019][1697217.487695] .cpu_load[1] : 0 [Wed Feb 6 09:40:43 2019][1697217.492497] .cpu_load[2] : 0 [Wed Feb 6 09:40:43 2019][1697217.497298] .cpu_load[3] : 0 [Wed Feb 6 09:40:43 2019][1697217.502099] .cpu_load[4] : 2 [Wed Feb 6 09:40:43 2019][1697217.506903] .avg_idle : 3016347 [Wed Feb 6 09:40:43 2019][1697217.512221] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:43 2019][1697217.517534] [Wed Feb 6 09:40:43 2019][1697217.519203] cfs_rq[4]:/ [Wed Feb 6 09:40:43 2019][1697217.521829] .exec_clock : 0.000000 [Wed Feb 6 09:40:43 2019][1697217.527227] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:43 2019][1697217.532628] .min_vruntime : 10779194.089162 [Wed Feb 6 09:40:43 2019][1697217.538634] .max_vruntime : 0.000001 [Wed Feb 6 09:40:43 2019][1697217.544032] .spread : 0.000000 [Wed Feb 6 09:40:43 2019][1697217.549432] .spread0 : 1935905.618948 [Wed Feb 6 09:40:43 2019][1697217.555351] .nr_spread_over : 0 [Wed Feb 6 09:40:43 2019][1697217.560143] .nr_running : 0 [Wed Feb 6 09:40:43 2019][1697217.564937] .load : 0 [Wed Feb 6 09:40:43 2019][1697217.569730] .runnable_load_avg : 0 [Wed Feb 6 09:40:43 2019][1697217.574523] .blocked_load_avg : 5 [Wed Feb 6 09:40:43 2019][1697217.579314] .tg_load_avg : 0 [Wed Feb 6 09:40:43 2019][1697217.584108] .tg_load_contrib : 0 [Wed Feb 6 09:40:43 2019][1697217.588900] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:43 2019][1697217.593693] .tg->runnable_avg : 0 [Wed Feb 6 09:40:43 2019][1697217.598486] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:43 2019][1697217.603280] .throttled : 0 [Wed Feb 6 09:40:43 2019][1697217.608072] .throttle_count : 0 [Wed Feb 6 09:40:43 2019][1697217.612865] .avg->runnable_avg_sum : 371 [Wed Feb 6 09:40:43 2019][1697217.617831] .avg->runnable_avg_period : 46246 [Wed Feb 6 09:40:43 2019][1697217.622970] [Wed Feb 6 09:40:43 2019][1697217.624642] rt_rq[4]:/ [Wed Feb 6 09:40:43 2019][1697217.627182] .rt_nr_running : 0 [Wed Feb 6 09:40:43 2019][1697217.631973] .rt_throttled : 0 [Wed Feb 6 09:40:43 2019][1697217.636769] .rt_time : 0.187422 [Wed Feb 6 09:40:43 2019][1697217.642166] .rt_runtime : 979.166667 [Wed Feb 6 09:40:43 2019][1697217.647739] [Wed Feb 6 09:40:43 2019][1697217.649412] runnable tasks: [Wed Feb 6 09:40:43 2019][1697217.652383] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:43 2019][1697217.663330] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:43 2019][1697217.674193] watchdog/4 31 -8.964073 424294 0 0.000000 65878.954201 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.685525] migration/4 32 0.000000 1677 0 0.000000 251532.082177 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.696862] ksoftirqd/4 33 10779154.595742 2203921 120 0.000000 28924.044410 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.708198] kworker/4:0 34 10779180.276467 401533 120 0.000000 18596.279405 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.719534] kworker/4:0H 35 3658.392034 10 100 0.000000 0.037102 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.730905] multipathd 11652 0.000000 3393125 0 0.000000 297805.331909 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.742261] kworker/4:1H 19440 10779175.560336 2349036 100 0.000000 14681.486463 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.753600] ptlrpc_hr00_002 126381 10779180.994430 15287051 120 0.000000 124548.778471 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.765022] kiblnd_sd_00_03 126424 10779194.894027 1556428156 120 0.000000 9507156.061117 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.776531] ptlrpcd_00_11 126452 10779182.877986 4270046 120 0.000000 66716.832047 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.787961] ldlm_cn00_000 126595 10766135.080968 3379120 120 0.000000 182952.416135 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.799387] mdt_out00_001 126652 10766307.109094 19381 120 0.000000 635.109692 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.810813] osp-syn-16-3 126748 10779117.294361 128742 120 0.000000 3516.099070 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.822242] osp-syn-10-1 126901 10779180.422466 198559 120 0.000000 4889.837912 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.833671] osp-syn-4-1 126941 10779160.252881 198348 120 0.000000 4838.436414 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.845090] osp-pre-7-1 126950 10779116.578084 213813 120 0.000000 6363.302554 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.856514] osp-syn-38-1 126961 10779160.219970 198156 120 0.000000 5500.902720 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.867942] mdt_out00_002 127200 10766477.630430 16021 120 0.000000 1194.696176 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.879363] ldlm_cn00_004 127232 10766135.076159 3347491 120 0.000000 200208.917111 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.890787] ldlm_cn00_007 128033 10778950.244691 2760686 120 0.000000 199736.159885 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.902209] mdt00_016 129257 10730326.970625 70530790 120 0.000000 2221320.327866 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.913626] mdt_io00_018 129263 7300174.315691 1505605 120 0.000000 65654.421745 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.925057] ldlm_cn00_015 130323 10778945.015151 2724609 120 0.000000 187998.076385 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.936481] ldlm_cn00_026 130336 10778945.014739 2108415 120 0.000000 185259.102837 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.947925] mdt00_031 17885 10779142.969125 312627 120 0.000000 1235329.030528 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.959260] mdt00_047 17940 10752396.732207 130827 120 0.000000 163382.355021 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.970601] kworker/4:2 23597 10332010.722442 17 120 0.000000 0.094408 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.981948] kworker/u256:3 24893 10779183.222415 4397 120 0.000000 5166.520402 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697217.993285] kworker/u256:0 25012 10778363.332352 1728 120 0.000000 19.172193 0.000000 0 / [Wed Feb 6 09:40:43 2019][1697218.004657] [Wed Feb 6 09:40:43 2019][1697218.006332] cpu#5, 1996.233 MHz [Wed Feb 6 09:40:43 2019][1697218.009654] .nr_running : 0 [Wed Feb 6 09:40:43 2019][1697218.014454] .load : 0 [Wed Feb 6 09:40:43 2019][1697218.019254] .nr_switches : 947727955 [Wed Feb 6 09:40:43 2019][1697218.024749] .nr_load_updates : 172600398 [Wed Feb 6 09:40:43 2019][1697218.030236] .nr_uninterruptible : -19202 [Wed Feb 6 09:40:43 2019][1697218.035470] .next_balance : 5991.870808 [Wed Feb 6 09:40:43 2019][1697218.041138] .curr->pid : 0 [Wed Feb 6 09:40:43 2019][1697218.045941] .clock : 1697218045.755684 [Wed Feb 6 09:40:43 2019][1697218.052129] .cpu_load[0] : 0 [Wed Feb 6 09:40:43 2019][1697218.056930] .cpu_load[1] : 0 [Wed Feb 6 09:40:43 2019][1697218.061731] .cpu_load[2] : 0 [Wed Feb 6 09:40:43 2019][1697218.066533] .cpu_load[3] : 0 [Wed Feb 6 09:40:43 2019][1697218.071333] .cpu_load[4] : 0 [Wed Feb 6 09:40:43 2019][1697218.076137] .avg_idle : 2597499 [Wed Feb 6 09:40:43 2019][1697218.081456] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:43 2019][1697218.086777] [Wed Feb 6 09:40:43 2019][1697218.088454] cfs_rq[5]:/ [Wed Feb 6 09:40:43 2019][1697218.091081] .exec_clock : 0.000000 [Wed Feb 6 09:40:43 2019][1697218.096478] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:43 2019][1697218.101879] .min_vruntime : 7687568.043404 [Wed Feb 6 09:40:43 2019][1697218.107797] .max_vruntime : 0.000001 [Wed Feb 6 09:40:43 2019][1697218.113198] .spread : 0.000000 [Wed Feb 6 09:40:43 2019][1697218.118597] .spread0 : -1155722.550142 [Wed Feb 6 09:40:43 2019][1697218.124603] .nr_spread_over : 0 [Wed Feb 6 09:40:43 2019][1697218.129396] .nr_running : 0 [Wed Feb 6 09:40:43 2019][1697218.134189] .load : 0 [Wed Feb 6 09:40:43 2019][1697218.138980] .runnable_load_avg : 0 [Wed Feb 6 09:40:43 2019][1697218.143775] .blocked_load_avg : 261 [Wed Feb 6 09:40:43 2019][1697218.148739] .tg_load_avg : 0 [Wed Feb 6 09:40:43 2019][1697218.153533] .tg_load_contrib : 0 [Wed Feb 6 09:40:43 2019][1697218.158326] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:43 2019][1697218.163118] .tg->runnable_avg : 0 [Wed Feb 6 09:40:43 2019][1697218.167911] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:43 2019][1697218.172704] .throttled : 0 [Wed Feb 6 09:40:43 2019][1697218.177496] .throttle_count : 0 [Wed Feb 6 09:40:43 2019][1697218.182290] .avg->runnable_avg_sum : 10467 [Wed Feb 6 09:40:44 2019][1697218.187429] .avg->runnable_avg_period : 46947 [Wed Feb 6 09:40:44 2019][1697218.192571] [Wed Feb 6 09:40:44 2019][1697218.194241] rt_rq[5]:/ [Wed Feb 6 09:40:44 2019][1697218.196780] .rt_nr_running : 0 [Wed Feb 6 09:40:44 2019][1697218.201572] .rt_throttled : 0 [Wed Feb 6 09:40:44 2019][1697218.206366] .rt_time : 0.000000 [Wed Feb 6 09:40:44 2019][1697218.211765] .rt_runtime : 979.166667 [Wed Feb 6 09:40:44 2019][1697218.217336] [Wed Feb 6 09:40:44 2019][1697218.219011] runnable tasks: [Wed Feb 6 09:40:44 2019][1697218.221983] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:44 2019][1697218.232928] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:44 2019][1697218.243793] watchdog/5 36 -8.959093 424290 0 0.000000 106296.982662 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.255125] migration/5 37 0.000000 1940 0 0.000000 240660.448547 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.266460] ksoftirqd/5 38 7687289.127970 4128019 120 0.000000 41347.450406 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.277797] kworker/5:0H 40 4044.610198 9 100 0.000000 0.037309 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.289178] kworker/5:1H 16026 7687289.068067 3150896 100 0.000000 14266.584469 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.300518] polkitd 16785 7687198.605335 452572 120 0.000000 16139.285985 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.311862] kdmwork-253:3 17730 7436810.889241 4918 120 0.000000 44.326019 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.323206] ptlrpc_hr01_003 126389 7687554.250695 28985395 120 0.000000 223396.395776 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.334625] ptlrpc_hr01_004 126390 7687561.301644 28985428 120 0.000000 221096.577960 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.346050] kiblnd_sd_01_00 126425 7687596.703119 816234883 120 0.000000 5793705.067720 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.357472] kiblnd_sd_01_03 126428 7687598.427353 816301273 120 0.000000 5811247.543309 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.368899] ptlrpcd_01_01 126454 7687555.052157 4005160 120 0.000000 56304.630375 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.380324] ptlrpcd_01_03 126456 7687554.336637 3996563 120 0.000000 62202.534693 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.391749] ptlrpcd_01_06 126459 7687554.251336 4022343 120 0.000000 59686.382945 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.403180] ldlm_cn01_000 126598 7687596.036915 12237279 120 0.000000 453023.985801 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.414624] osp-pre-0-1 126889 7687555.068748 205490 120 0.000000 4925.554847 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.426044] osp-pre-14-1 126906 7687554.271203 213940 120 0.000000 7796.659801 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.437474] osp-syn-37-1 126975 7687496.699374 196850 120 0.000000 4699.370866 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.448902] mdt_io01_004 127221 7687289.568917 2989298 120 0.000000 118584.598645 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.460328] mdt_io01_007 127228 7512631.538409 3641964 120 0.000000 147086.741461 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.471763] mdt_rdpg01_005 128517 7687602.086208 11796414 120 0.000000 505129.532774 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.483186] mdt01_021 129249 7571326.128004 36499369 120 0.000000 2321683.065623 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.494613] mdt_io01_031 130109 7687289.085511 3056837 120 0.000000 129240.557237 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.506051] kworker/5:2 95447 7687562.514817 43795 120 0.000000 15623.137521 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.517412] mdt01_047 17978 7687423.466395 108693 120 0.000000 164299.816070 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.528750] kworker/5:1 18809 5699832.661108 2 120 0.000000 0.009258 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.540089] mdt01_063 19141 7571326.129457 157772 120 0.000000 619801.404610 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.551428] ldlm_cn01_027 23735 7687607.584028 49338 120 0.000000 150918.919289 0.000000 1 / [Wed Feb 6 09:40:44 2019][1697218.562839] [Wed Feb 6 09:40:44 2019][1697218.564516] cpu#6, 1996.233 MHz [Wed Feb 6 09:40:44 2019][1697218.567847] .nr_running : 1 [Wed Feb 6 09:40:44 2019][1697218.572647] .load : 0 [Wed Feb 6 09:40:44 2019][1697218.577449] .nr_switches : 1387772738 [Wed Feb 6 09:40:44 2019][1697218.583029] .nr_load_updates : 178473536 [Wed Feb 6 09:40:44 2019][1697218.588523] .nr_uninterruptible : -8158 [Wed Feb 6 09:40:44 2019][1697218.593673] .next_balance : 5991.871367 [Wed Feb 6 09:40:44 2019][1697218.599341] .curr->pid : 23726 [Wed Feb 6 09:40:44 2019][1697218.604488] .clock : 1697218604.454303 [Wed Feb 6 09:40:44 2019][1697218.610677] .cpu_load[0] : 0 [Wed Feb 6 09:40:44 2019][1697218.615479] .cpu_load[1] : 192 [Wed Feb 6 09:40:44 2019][1697218.620474] .cpu_load[2] : 60 [Wed Feb 6 09:40:44 2019][1697218.625360] .cpu_load[3] : 133 [Wed Feb 6 09:40:44 2019][1697218.630334] .cpu_load[4] : 99 [Wed Feb 6 09:40:44 2019][1697218.635221] .avg_idle : 188642 [Wed Feb 6 09:40:44 2019][1697218.640455] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:44 2019][1697218.645776] [Wed Feb 6 09:40:44 2019][1697218.647443] cfs_rq[6]:/ [Wed Feb 6 09:40:44 2019][1697218.650071] .exec_clock : 0.000000 [Wed Feb 6 09:40:44 2019][1697218.655470] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:44 2019][1697218.660869] .min_vruntime : 7791096.723235 [Wed Feb 6 09:40:44 2019][1697218.666788] .max_vruntime : 0.000001 [Wed Feb 6 09:40:44 2019][1697218.672189] .spread : 0.000000 [Wed Feb 6 09:40:44 2019][1697218.677587] .spread0 : -1052223.361699 [Wed Feb 6 09:40:44 2019][1697218.683594] .nr_spread_over : 0 [Wed Feb 6 09:40:44 2019][1697218.688385] .nr_running : 0 [Wed Feb 6 09:40:44 2019][1697218.693178] .load : 0 [Wed Feb 6 09:40:44 2019][1697218.697972] .runnable_load_avg : 0 [Wed Feb 6 09:40:44 2019][1697218.702763] .blocked_load_avg : 118 [Wed Feb 6 09:40:44 2019][1697218.707730] .tg_load_avg : 0 [Wed Feb 6 09:40:44 2019][1697218.712522] .tg_load_contrib : 0 [Wed Feb 6 09:40:44 2019][1697218.717315] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:44 2019][1697218.722108] .tg->runnable_avg : 0 [Wed Feb 6 09:40:44 2019][1697218.726902] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:44 2019][1697218.731694] .throttled : 0 [Wed Feb 6 09:40:44 2019][1697218.736487] .throttle_count : 0 [Wed Feb 6 09:40:44 2019][1697218.741279] .avg->runnable_avg_sum : 2162 [Wed Feb 6 09:40:44 2019][1697218.746331] .avg->runnable_avg_period : 47437 [Wed Feb 6 09:40:44 2019][1697218.751476] [Wed Feb 6 09:40:44 2019][1697218.753145] rt_rq[6]:/ [Wed Feb 6 09:40:44 2019][1697218.755683] .rt_nr_running : 0 [Wed Feb 6 09:40:44 2019][1697218.760476] .rt_throttled : 0 [Wed Feb 6 09:40:44 2019][1697218.765267] .rt_time : 0.000000 [Wed Feb 6 09:40:44 2019][1697218.770669] .rt_runtime : 671.389566 [Wed Feb 6 09:40:44 2019][1697218.776241] [Wed Feb 6 09:40:44 2019][1697218.777912] runnable tasks: [Wed Feb 6 09:40:44 2019][1697218.780887] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:44 2019][1697218.791832] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:44 2019][1697218.802698] watchdog/6 41 -8.967900 424299 0 0.000000 67455.189984 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.814037] migration/6 42 0.000000 1596 0 0.000000 223290.131659 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.825370] ksoftirqd/6 43 7790978.241656 3358987 120 0.000000 57611.129329 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.836708] kworker/6:0H 45 4076.851765 10 100 0.000000 0.033033 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.848073] kauditd 440 7790769.170985 222618 120 0.000000 3194.865291 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.859431] chronyd 16811 7745140.541499 5030 120 0.000000 207.836982 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.870768] sssd_nss 16832 7791070.313010 441615 120 0.000000 14977.171887 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.882109] writer#0 18226 7790999.520300 24961833 120 0.000000 231478.299035 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.893448] kworker/6:1H 19274 7790978.224133 1865055 100 0.000000 7585.964874 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.904790] kworker/6:2 72368 7790997.001915 276048 120 0.000000 12757.817765 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.916130] lnet_discovery 126439 7791005.864196 1031250 120 0.000000 13415.891137 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.927557] ptlrpcd_02_00 126465 7790999.161496 6845024 120 0.000000 84921.863339 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.938978] ptlrpcd_02_09 126474 7791087.881162 6832730 120 0.000000 85096.799566 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.950403] kmmpd-dm-2 126576 7791087.871494 411751 120 0.000000 13114.720403 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.961827] ldlm_cn02_001 126602 7791061.770575 6426094 120 0.000000 278969.796342 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.973261] osp-syn-20-3 126758 7790978.169721 128174 120 0.000000 4324.347896 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.984695] osp-pre-0-1 126928 7790999.123724 213962 120 0.000000 9073.716602 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697218.996117] osp-syn-32-1 126933 7790985.772440 198500 120 0.000000 4752.448693 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.007539] osp-pre-32-1 126934 7790999.112434 213943 120 0.000000 7257.955803 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.018967] osp-syn-3-1 126971 7790985.806063 197572 120 0.000000 4752.201177 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.030386] osp-syn-34-1 126987 7790985.788200 197830 120 0.000000 4665.039237 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.041814] ldlm_cn02_004 127239 7791099.734947 5196610 120 0.000000 309662.657814 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.053241] mdt02_010 127297 7645133.380409 27171286 120 0.000000 1433178.683699 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.064664] mdt_io02_011 127515 3695886.755829 1853414 120 0.000000 67562.742342 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.076085] mdt_io02_014 127690 2021028.172001 597446 120 0.000000 22017.059034 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.087508] mdt_io02_015 127692 3691611.562739 576262 120 0.000000 21087.227353 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.098937] mdt_io02_018 129239 7633753.963028 1576800 120 0.000000 58955.453430 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.110371] mdt_io02_021 129293 7790853.345135 1874883 120 0.000000 85566.548995 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.121796] mdt_io02_025 130204 4049438.659077 1500601 120 0.000000 58230.379911 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.133232] kworker/6:1 10262 4040693.077406 2 120 0.000000 0.009118 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.144571] kworker/u264:2 10936 4074929.542843 37 100 0.000000 0.269726 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.155926] mdt02_057 18647 7701308.932805 5862 120 0.000000 8784.337902 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.167269] ldlm_cn02_029 23743 7791066.822518 33440 120 0.000000 109823.108135 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.178604] ldlm_cn02_031 23748 7791070.018006 31993 120 0.000000 96900.127490 0.000000 2 / [Wed Feb 6 09:40:44 2019][1697219.189994] [Wed Feb 6 09:40:44 2019][1697219.191668] cpu#7, 1996.233 MHz [Wed Feb 6 09:40:45 2019][1697219.194990] .nr_running : 1 [Wed Feb 6 09:40:45 2019][1697219.199791] .load : 1024 [Wed Feb 6 09:40:45 2019][1697219.204852] .nr_switches : 144834193 [Wed Feb 6 09:40:45 2019][1697219.210338] .nr_load_updates : 79528646 [Wed Feb 6 09:40:45 2019][1697219.215739] .nr_uninterruptible : -1246 [Wed Feb 6 09:40:45 2019][1697219.220886] .next_balance : 5991.872047 [Wed Feb 6 09:40:45 2019][1697219.226545] .curr->pid : 10650 [Wed Feb 6 09:40:45 2019][1697219.231685] .clock : 1697219230.768466 [Wed Feb 6 09:40:45 2019][1697219.237862] .cpu_load[0] : 1024 [Wed Feb 6 09:40:45 2019][1697219.242915] .cpu_load[1] : 1024 [Wed Feb 6 09:40:45 2019][1697219.247971] .cpu_load[2] : 1024 [Wed Feb 6 09:40:45 2019][1697219.253030] .cpu_load[3] : 1024 [Wed Feb 6 09:40:45 2019][1697219.258083] .cpu_load[4] : 1024 [Wed Feb 6 09:40:45 2019][1697219.263136] .avg_idle : 10000000 [Wed Feb 6 09:40:45 2019][1697219.268536] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:45 2019][1697219.273847] [Wed Feb 6 09:40:45 2019][1697219.275517] cfs_rq[7]:/ [Wed Feb 6 09:40:45 2019][1697219.278142] .exec_clock : 0.000000 [Wed Feb 6 09:40:45 2019][1697219.283543] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:45 2019][1697219.288941] .min_vruntime : 3183348.753766 [Wed Feb 6 09:40:45 2019][1697219.294860] .max_vruntime : 0.000001 [Wed Feb 6 09:40:45 2019][1697219.300260] .spread : 0.000000 [Wed Feb 6 09:40:45 2019][1697219.305659] .spread0 : -5659973.580791 [Wed Feb 6 09:40:45 2019][1697219.311667] .nr_spread_over : 0 [Wed Feb 6 09:40:45 2019][1697219.316457] .nr_running : 1 [Wed Feb 6 09:40:45 2019][1697219.321250] .load : 1024 [Wed Feb 6 09:40:45 2019][1697219.326304] .runnable_load_avg : 1023 [Wed Feb 6 09:40:45 2019][1697219.331356] .blocked_load_avg : 0 [Wed Feb 6 09:40:45 2019][1697219.336148] .tg_load_avg : 0 [Wed Feb 6 09:40:45 2019][1697219.340941] .tg_load_contrib : 0 [Wed Feb 6 09:40:45 2019][1697219.345734] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:45 2019][1697219.350528] .tg->runnable_avg : 0 [Wed Feb 6 09:40:45 2019][1697219.355320] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:45 2019][1697219.360113] .throttled : 0 [Wed Feb 6 09:40:45 2019][1697219.364906] .throttle_count : 0 [Wed Feb 6 09:40:45 2019][1697219.369697] .avg->runnable_avg_sum : 46229 [Wed Feb 6 09:40:45 2019][1697219.374838] .avg->runnable_avg_period : 47049 [Wed Feb 6 09:40:45 2019][1697219.379982] [Wed Feb 6 09:40:45 2019][1697219.381649] rt_rq[7]:/ [Wed Feb 6 09:40:45 2019][1697219.384188] .rt_nr_running : 0 [Wed Feb 6 09:40:45 2019][1697219.388982] .rt_throttled : 0 [Wed Feb 6 09:40:45 2019][1697219.393773] .rt_time : 4.958689 [Wed Feb 6 09:40:45 2019][1697219.399174] .rt_runtime : 961.593189 [Wed Feb 6 09:40:45 2019][1697219.404748] [Wed Feb 6 09:40:45 2019][1697219.406418] runnable tasks: [Wed Feb 6 09:40:45 2019][1697219.409392] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:45 2019][1697219.420338] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:45 2019][1697219.431204] watchdog/7 46 -8.960446 424280 0 0.000000 42118.012431 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.442542] migration/7 47 0.000000 1271 0 0.000000 220746.253006 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.453878] ksoftirqd/7 48 3182676.601203 707858 120 0.000000 7981.118134 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.465213] kworker/7:0H 50 20375.457665 10 100 0.000000 0.031672 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.476578] khugepaged 283 3182678.453295 171145 139 0.000000 247601.624182 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.487916] kworker/7:1 7267 3182697.334730 134711 120 0.000000 11761.461222 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.499267] sssd 16762 3182635.808601 56596 120 0.000000 1799.973868 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.510613] kworker/7:1H 44035 3182676.585450 1142787 100 0.000000 3882.304871 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.521976] osp-pre-13-3 126707 3182697.332656 214649 120 0.000000 8305.461991 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.533396] osp-pre-46-3 126711 3182697.334269 214651 120 0.000000 10159.622425 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.544822] osp-pre-0-3 126739 3182697.337956 214645 120 0.000000 4845.828373 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.556242] osp-pre-44-3 126743 3182695.184664 214591 120 0.000000 7378.554563 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.567667] osp-pre-40-3 126763 3182697.331575 214623 120 0.000000 6673.576642 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.579088] osp-syn-3-3 126782 3182642.132973 126207 120 0.000000 4692.539359 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.590511] osp-syn-34-3 126798 3182589.734738 126605 120 0.000000 3493.800064 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.601932] lquota_wb_fir-M 126870 3181795.413150 17159 120 0.000000 193.026060 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.613360] osp-syn-14-1 126905 3182681.701163 198577 120 0.000000 4682.486527 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.624781] osp-syn-16-1 126937 3182681.604031 199528 120 0.000000 4734.913152 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.636206] mdt03_003 127130 3161319.170844 6634409 120 0.000000 277269.879712 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.647634] ldlm_cn03_003 127455 3182586.088183 4007009 120 0.000000 173821.427841 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.659088] mdt_io03_006 40452 3181696.645231 105027 120 0.000000 3410.458175 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.670432] R ldlm_bl_09 10650 3183737.757685 1324024 120 0.000000 1449615.883145 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.681768] ldlm_cn03_016 14200 3182510.109875 322040 120 0.000000 99188.815077 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.693107] mdt03_019 17811 3182666.814565 201531 120 0.000000 158766.186475 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.704453] mdt03_038 17969 3182666.812552 87669 120 0.000000 160715.660699 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.715803] kworker/7:2 24062 2521130.776847 2 120 0.000000 0.006573 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.727134] ldlm_cn03_020 24583 3182653.709094 12487 120 0.000000 92029.442864 0.000000 3 / [Wed Feb 6 09:40:45 2019][1697219.738510] [Wed Feb 6 09:40:45 2019][1697219.740190] cpu#8, 1996.233 MHz [Wed Feb 6 09:40:45 2019][1697219.743510] .nr_running : 1 [Wed Feb 6 09:40:45 2019][1697219.748301] .load : 1024 [Wed Feb 6 09:40:45 2019][1697219.753353] .nr_switches : 1232208781 [Wed Feb 6 09:40:45 2019][1697219.758926] .nr_load_updates : 270179625 [Wed Feb 6 09:40:45 2019][1697219.764414] .nr_uninterruptible : 111439 [Wed Feb 6 09:40:45 2019][1697219.769664] .next_balance : 5991.872543 [Wed Feb 6 09:40:45 2019][1697219.775324] .curr->pid : 7941 [Wed Feb 6 09:40:45 2019][1697219.780377] .clock : 1697219780.374208 [Wed Feb 6 09:40:45 2019][1697219.786557] .cpu_load[0] : 0 [Wed Feb 6 09:40:45 2019][1697219.791350] .cpu_load[1] : 0 [Wed Feb 6 09:40:45 2019][1697219.796144] .cpu_load[2] : 0 [Wed Feb 6 09:40:45 2019][1697219.800944] .cpu_load[3] : 0 [Wed Feb 6 09:40:45 2019][1697219.805737] .cpu_load[4] : 0 [Wed Feb 6 09:40:45 2019][1697219.810527] .avg_idle : 4060926 [Wed Feb 6 09:40:45 2019][1697219.815844] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:45 2019][1697219.821165] [Wed Feb 6 09:40:45 2019][1697219.822840] cfs_rq[8]:/ [Wed Feb 6 09:40:45 2019][1697219.825467] .exec_clock : 0.000000 [Wed Feb 6 09:40:45 2019][1697219.830867] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:45 2019][1697219.836266] .min_vruntime : 8159788.764270 [Wed Feb 6 09:40:45 2019][1697219.842184] .max_vruntime : 0.000001 [Wed Feb 6 09:40:45 2019][1697219.847585] .spread : 0.000000 [Wed Feb 6 09:40:45 2019][1697219.852983] .spread0 : -683534.081040 [Wed Feb 6 09:40:45 2019][1697219.858904] .nr_spread_over : 0 [Wed Feb 6 09:40:45 2019][1697219.863695] .nr_running : 0 [Wed Feb 6 09:40:45 2019][1697219.868488] .load : 0 [Wed Feb 6 09:40:45 2019][1697219.873282] .runnable_load_avg : 0 [Wed Feb 6 09:40:45 2019][1697219.878074] .blocked_load_avg : 0 [Wed Feb 6 09:40:45 2019][1697219.882867] .tg_load_avg : 0 [Wed Feb 6 09:40:45 2019][1697219.887660] .tg_load_contrib : 0 [Wed Feb 6 09:40:45 2019][1697219.892453] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:45 2019][1697219.897246] .tg->runnable_avg : 0 [Wed Feb 6 09:40:45 2019][1697219.902038] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:45 2019][1697219.906829] .throttled : 0 [Wed Feb 6 09:40:45 2019][1697219.911623] .throttle_count : 0 [Wed Feb 6 09:40:45 2019][1697219.916417] .avg->runnable_avg_sum : 80 [Wed Feb 6 09:40:45 2019][1697219.921296] .avg->runnable_avg_period : 47287 [Wed Feb 6 09:40:45 2019][1697219.926436] [Wed Feb 6 09:40:45 2019][1697219.928107] rt_rq[8]:/ [Wed Feb 6 09:40:45 2019][1697219.930646] .rt_nr_running : 0 [Wed Feb 6 09:40:45 2019][1697219.935440] .rt_throttled : 0 [Wed Feb 6 09:40:45 2019][1697219.940233] .rt_time : 0.000000 [Wed Feb 6 09:40:45 2019][1697219.945633] .rt_runtime : 958.816639 [Wed Feb 6 09:40:45 2019][1697219.951205] [Wed Feb 6 09:40:45 2019][1697219.952877] runnable tasks: [Wed Feb 6 09:40:45 2019][1697219.955850] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:45 2019][1697219.966795] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:45 2019][1697219.977661] watchdog/8 51 -11.972858 424280 0 0.000000 68083.639256 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697219.988999] migration/8 52 0.000000 1688 0 0.000000 312044.665492 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.000336] ksoftirqd/8 53 8159666.989379 3020919 120 0.000000 34803.764818 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.011672] kworker/8:0H 55 2296.066421 10 100 0.000000 0.033543 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.023043] jbd2/sdk2-8 7868 8159760.295276 239802 120 0.000000 15369.338541 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.034380] systemd-journal 7941 8159780.679545 161780 120 0.000000 38549.425039 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.045717] multipathd 9718 0.000000 339420 0 0.000000 56869.821652 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.057057] kvm-irqfd-clean 14538 2040.133942 2 100 0.000000 0.176062 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.068396] kworker/8:1H 17097 8159704.313566 2771566 100 0.000000 13259.462941 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.079742] cfs_rh_02 126360 8159745.751501 903 120 0.000000 956.840337 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.091173] ptlrpcd_00_00 126441 8159776.848068 4263260 120 0.000000 69002.593992 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.102603] ldlm_cb00_000 126585 6480400.119114 7900 120 0.000000 158.802812 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.114030] mdt00_002 126625 8097173.020583 81156234 120 0.000000 2353576.295099 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.125454] mdt_fld_0000 126663 8159314.207392 1975 120 0.000000 26.573101 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.136880] osp-pre-45-3 126765 8159733.309330 214713 120 0.000000 6220.330725 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.148306] osp-pre-47-3 126771 8159733.112246 214724 120 0.000000 9258.374857 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.159732] dist_txn-3 126872 6861744.659076 19 120 0.000000 0.655003 0.000000 0 / [Wed Feb 6 09:40:45 2019][1697220.171150] osp_up0-1 126888 7989477.709147 279 120 0.000000 8.588288 0.000000 0 / [Wed Feb 6 09:40:46 2019][1697220.182576] osp-syn-31-1 126909 8159776.795919 199178 120 0.000000 5037.256293 0.000000 0 / [Wed Feb 6 09:40:46 2019][1697220.193998] osp-pre-5-1 126930 8159776.776543 213775 120 0.000000 7314.898411 0.000000 0 / [Wed Feb 6 09:40:46 2019][1697220.205421] osp-pre-47-1 126960 8159771.755925 213941 120 0.000000 4809.128134 0.000000 0 / [Wed Feb 6 09:40:46 2019][1697220.216841] osp-syn-29-1 126965 8159776.827890 199150 120 0.000000 4772.974625 0.000000 0 / [Wed Feb 6 09:40:46 2019][1697220.228278] mdt_io00_010 127364 8097079.697006 2005130 120 0.000000 89682.868869 0.000000 0 / [Wed Feb 6 09:40:46 2019][1697220.239698] mdt_io00_015 127702 6336806.924085 2083394 120 0.000000 81934.389972 0.000000 0 / [Wed Feb 6 09:40:46 2019][1697220.251130] mdt00_018 129259 8105147.460106 74366005 120 0.000000 2937392.747150 0.000000 0 / [Wed Feb 6 09:40:46 2019][1697220.262551] mdt00_022 129280 8105065.020813 85299943 120 0.000000 2472029.392988 0.000000 0 / [Wed Feb 6 09:40:46 2019][1697220.273978] ldlm_cn00_013 130321 8159751.883512 5440898 120 0.000000 258432.918688 0.000000 0 / [Wed Feb 6 09:40:46 2019][1697220.285404] ldlm_cn00_014 130322 8159746.337255 2528128 120 0.000000 196349.432331 0.000000 0 / [Wed Feb 6 09:40:46 2019][1697220.296830] mdt_rdpg00_005 1270 8159777.002923 953087 120 0.000000 50876.119703 0.000000 0 / [Wed Feb 6 09:40:46 2019][1697220.308190] mdt00_044 17932 8159724.195886 45253 120 0.000000 163302.363869 0.000000 0 / [Wed Feb 6 09:40:46 2019][1697220.319535] kworker/8:2 19466 8159776.790320 33528 120 0.000000 35798.253099 0.000000 0 / [Wed Feb 6 09:40:46 2019][1697220.330881] kworker/8:1 24778 8025551.747293 3 120 0.000000 0.352113 0.000000 0 / [Wed Feb 6 09:40:46 2019][1697220.342247] [Wed Feb 6 09:40:46 2019][1697220.343925] cpu#9, 1996.233 MHz [Wed Feb 6 09:40:46 2019][1697220.347245] .nr_running : 0 [Wed Feb 6 09:40:46 2019][1697220.352038] .load : 0 [Wed Feb 6 09:40:46 2019][1697220.356831] .nr_switches : 768735620 [Wed Feb 6 09:40:46 2019][1697220.362317] .nr_load_updates : 174236794 [Wed Feb 6 09:40:46 2019][1697220.367804] .nr_uninterruptible : -45804 [Wed Feb 6 09:40:46 2019][1697220.373029] .next_balance : 5991.873146 [Wed Feb 6 09:40:46 2019][1697220.378688] .curr->pid : 0 [Wed Feb 6 09:40:46 2019][1697220.383479] .clock : 1697220382.778096 [Wed Feb 6 09:40:46 2019][1697220.389661] .cpu_load[0] : 0 [Wed Feb 6 09:40:46 2019][1697220.394454] .cpu_load[1] : 0 [Wed Feb 6 09:40:46 2019][1697220.399246] .cpu_load[2] : 0 [Wed Feb 6 09:40:46 2019][1697220.404040] .cpu_load[3] : 0 [Wed Feb 6 09:40:46 2019][1697220.408830] .cpu_load[4] : 0 [Wed Feb 6 09:40:46 2019][1697220.413624] .avg_idle : 6755941 [Wed Feb 6 09:40:46 2019][1697220.418937] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:46 2019][1697220.424249] [Wed Feb 6 09:40:46 2019][1697220.425920] cfs_rq[9]:/ [Wed Feb 6 09:40:46 2019][1697220.428544] .exec_clock : 0.000000 [Wed Feb 6 09:40:46 2019][1697220.433945] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:46 2019][1697220.439342] .min_vruntime : 6655668.156514 [Wed Feb 6 09:40:46 2019][1697220.445262] .max_vruntime : 0.000001 [Wed Feb 6 09:40:46 2019][1697220.450662] .spread : 0.000000 [Wed Feb 6 09:40:46 2019][1697220.456061] .spread0 : -2187655.593741 [Wed Feb 6 09:40:46 2019][1697220.462066] .nr_spread_over : 0 [Wed Feb 6 09:40:46 2019][1697220.466861] .nr_running : 0 [Wed Feb 6 09:40:46 2019][1697220.471653] .load : 0 [Wed Feb 6 09:40:46 2019][1697220.476446] .runnable_load_avg : 0 [Wed Feb 6 09:40:46 2019][1697220.481237] .blocked_load_avg : 0 [Wed Feb 6 09:40:46 2019][1697220.486031] .tg_load_avg : 0 [Wed Feb 6 09:40:46 2019][1697220.490824] .tg_load_contrib : 0 [Wed Feb 6 09:40:46 2019][1697220.495617] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:46 2019][1697220.500409] .tg->runnable_avg : 0 [Wed Feb 6 09:40:46 2019][1697220.505203] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:46 2019][1697220.509995] .throttled : 0 [Wed Feb 6 09:40:46 2019][1697220.514786] .throttle_count : 0 [Wed Feb 6 09:40:46 2019][1697220.519580] .avg->runnable_avg_sum : 516 [Wed Feb 6 09:40:46 2019][1697220.524548] .avg->runnable_avg_period : 47191 [Wed Feb 6 09:40:46 2019][1697220.529688] [Wed Feb 6 09:40:46 2019][1697220.531359] rt_rq[9]:/ [Wed Feb 6 09:40:46 2019][1697220.533898] .rt_nr_running : 0 [Wed Feb 6 09:40:46 2019][1697220.538689] .rt_throttled : 0 [Wed Feb 6 09:40:46 2019][1697220.543482] .rt_time : 0.000000 [Wed Feb 6 09:40:46 2019][1697220.548882] .rt_runtime : 953.794264 [Wed Feb 6 09:40:46 2019][1697220.554454] [Wed Feb 6 09:40:46 2019][1697220.556126] runnable tasks: [Wed Feb 6 09:40:46 2019][1697220.559101] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:46 2019][1697220.570047] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:46 2019][1697220.580913] watchdog/9 56 -11.978148 424268 0 0.000000 79237.777860 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.592251] migration/9 57 0.000000 1810 0 0.000000 349317.579871 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.603588] ksoftirqd/9 58 6655654.387611 5179950 120 0.000000 52627.266187 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.614922] kworker/9:0H 60 2878.855836 9 100 0.000000 0.040526 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.626295] multipathd 9649 -6.758610 119 0 0.000000 28.275240 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.637637] kworker/9:1H 16051 6655654.190197 4107008 100 0.000000 19381.826282 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.648976] kdmflush 16262 2890.952437 2 100 0.000000 0.011551 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.660311] bioset 16277 2892.350162 2 100 0.000000 0.012142 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.671646] kdmflush 16353 2898.355618 2 100 0.000000 0.012894 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.682982] bioset 16355 2898.359581 2 100 0.000000 0.014457 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.694321] kdmflush 16477 2906.842690 2 100 0.000000 0.011743 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.705655] bioset 16479 2906.886873 2 100 0.000000 0.011010 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.716997] sssd_pam 16833 6655618.296346 339252 120 0.000000 9460.876017 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.728329] kdmflush 17726 3040.222350 2 100 0.000000 0.012704 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.739664] bioset 17729 3040.265924 2 100 0.000000 0.012663 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.751007] kdmflush 18670 3088.526498 2 100 0.000000 0.010400 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.762353] ptlrpc_hr01_002 126388 6655659.806655 28985416 120 0.000000 219816.709150 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.773777] ptlrpc_hr01_005 126391 6655656.219283 28985456 120 0.000000 219643.985542 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.785203] kiblnd_connd 126419 6655654.202735 1031813 120 0.000000 24971.659813 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.796629] ptlrpcd_01_00 126453 6655654.278326 4004094 120 0.000000 62614.884536 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.808055] ptlrpcd_01_07 126460 6655654.240806 3983185 120 0.000000 55626.187510 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.819485] ldlm_elt 126609 6654963.389815 632 120 0.000000 6557.530279 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.830908] mdt01_002 126628 6653304.205067 39613497 120 0.000000 1955941.830146 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.842335] mdt_out01_000 126653 3724896.671802 69 120 0.000000 2.549374 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.853755] mdt_io01_002 126670 4056401.537971 1974591 120 0.000000 80755.543305 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.865177] osp-syn-39-3 126708 6655462.968065 127401 120 0.000000 3574.596922 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.876616] osp-pre-40-1 126952 6655535.640771 213936 120 0.000000 9018.070341 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.888048] mdt_io01_008 127235 6544892.241918 2896420 120 0.000000 126354.660056 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.899476] mdt_io01_015 127694 6554388.189307 1601029 120 0.000000 60191.654228 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.910895] mdt_io01_016 127695 1580004.913210 1270087 120 0.000000 44118.756936 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.922322] mdt01_013 129227 6655672.342048 36481741 120 0.000000 1590608.937489 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.933766] kworker/9:0 7406 6655656.063572 42531 120 0.000000 17439.087647 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.945102] ldlm_bl_14 10655 6655654.198657 1327107 120 0.000000 1816015.559987 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.956442] mdt01_039 17831 6655445.623425 155578 120 0.000000 15331.591688 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.967794] mdt01_061 19115 6653304.203504 181180 120 0.000000 18283.949526 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.979129] kworker/u258:1 22145 6655654.881863 336 120 0.000000 4255.569790 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697220.990463] kworker/9:1 22207 5729928.690979 2 120 0.000000 0.009789 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697221.001803] ldlm_cn01_030 23738 6655604.411098 46449 120 0.000000 120534.502285 0.000000 1 / [Wed Feb 6 09:40:46 2019][1697221.013172] [Wed Feb 6 09:40:46 2019][1697221.014846] cpu#10, 1996.233 MHz [Wed Feb 6 09:40:46 2019][1697221.018253] .nr_running : 0 [Wed Feb 6 09:40:46 2019][1697221.023045] .load : 0 [Wed Feb 6 09:40:46 2019][1697221.027839] .nr_switches : 659413465 [Wed Feb 6 09:40:46 2019][1697221.033325] .nr_load_updates : 171736219 [Wed Feb 6 09:40:46 2019][1697221.038811] .nr_uninterruptible : -19639 [Wed Feb 6 09:40:46 2019][1697221.044038] .next_balance : 5991.873818 [Wed Feb 6 09:40:46 2019][1697221.049696] .curr->pid : 0 [Wed Feb 6 09:40:46 2019][1697221.054490] .clock : 1697221053.779126 [Wed Feb 6 09:40:46 2019][1697221.060670] .cpu_load[0] : 0 [Wed Feb 6 09:40:46 2019][1697221.065471] .cpu_load[1] : 0 [Wed Feb 6 09:40:46 2019][1697221.070261] .cpu_load[2] : 0 [Wed Feb 6 09:40:46 2019][1697221.075056] .cpu_load[3] : 112 [Wed Feb 6 09:40:46 2019][1697221.080023] .cpu_load[4] : 102 [Wed Feb 6 09:40:46 2019][1697221.084998] .avg_idle : 1190610 [Wed Feb 6 09:40:46 2019][1697221.090319] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:46 2019][1697221.095633] [Wed Feb 6 09:40:46 2019][1697221.097308] cfs_rq[10]:/ [Wed Feb 6 09:40:46 2019][1697221.100021] .exec_clock : 0.000000 [Wed Feb 6 09:40:46 2019][1697221.105420] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:46 2019][1697221.110820] .min_vruntime : 6526495.688590 [Wed Feb 6 09:40:46 2019][1697221.116740] .max_vruntime : 0.000001 [Wed Feb 6 09:40:46 2019][1697221.122136] .spread : 0.000000 [Wed Feb 6 09:40:46 2019][1697221.127537] .spread0 : -2316829.259966 [Wed Feb 6 09:40:46 2019][1697221.133544] .nr_spread_over : 0 [Wed Feb 6 09:40:46 2019][1697221.138337] .nr_running : 0 [Wed Feb 6 09:40:46 2019][1697221.143128] .load : 0 [Wed Feb 6 09:40:46 2019][1697221.147921] .runnable_load_avg : 0 [Wed Feb 6 09:40:46 2019][1697221.152716] .blocked_load_avg : 15 [Wed Feb 6 09:40:46 2019][1697221.157595] .tg_load_avg : 0 [Wed Feb 6 09:40:46 2019][1697221.162387] .tg_load_contrib : 0 [Wed Feb 6 09:40:46 2019][1697221.167178] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:46 2019][1697221.171972] .tg->runnable_avg : 0 [Wed Feb 6 09:40:46 2019][1697221.176765] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:47 2019][1697221.181558] .throttled : 0 [Wed Feb 6 09:40:47 2019][1697221.186351] .throttle_count : 0 [Wed Feb 6 09:40:47 2019][1697221.191142] .avg->runnable_avg_sum : 361 [Wed Feb 6 09:40:47 2019][1697221.196109] .avg->runnable_avg_period : 47229 [Wed Feb 6 09:40:47 2019][1697221.201252] [Wed Feb 6 09:40:47 2019][1697221.202921] rt_rq[10]:/ [Wed Feb 6 09:40:47 2019][1697221.205547] .rt_nr_running : 0 [Wed Feb 6 09:40:47 2019][1697221.210340] .rt_throttled : 0 [Wed Feb 6 09:40:47 2019][1697221.215132] .rt_time : 0.000000 [Wed Feb 6 09:40:47 2019][1697221.220532] .rt_runtime : 979.166667 [Wed Feb 6 09:40:47 2019][1697221.226103] [Wed Feb 6 09:40:47 2019][1697221.227775] runnable tasks: [Wed Feb 6 09:40:47 2019][1697221.230750] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:47 2019][1697221.241696] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:47 2019][1697221.252564] watchdog/10 61 -11.979271 424293 0 0.000000 102788.309005 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.263901] migration/10 62 0.000000 1530 0 0.000000 214437.159155 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.275235] ksoftirqd/10 63 6526484.686688 3132825 120 0.000000 37241.061961 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.286573] kworker/10:0H 65 3076.764213 10 100 0.000000 0.060085 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.297931] kdevtmpfs 255 105869.030004 327 120 0.000000 3.431263 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.309264] kworker/10:1 1089 6526484.187847 301552 120 0.000000 19458.920214 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.320600] poll_mpt3sas0_h 5528 6285380.008483 7 100 0.000000 271.762896 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.331957] sshd 18216 6525666.467758 324 120 0.000000 61.613233 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.343288] rsyslogd 18221 6480558.115627 2842 120 0.000000 48.414485 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.354623] kworker/10:1H 18372 6526476.133160 2322170 100 0.000000 10154.630833 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.365961] slapd 18607 4058067.670357 1772 120 0.000000 17.114158 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.377304] ptlrpc_hr02_000 126392 6526480.621141 15299432 120 0.000000 123210.921062 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.388732] kiblnd_sd_02_03 126432 3787892.733197 720348989 120 0.000000 4642791.100226 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.400156] ptlrpcd_02_02 126467 6526479.358758 6897150 120 0.000000 88205.511059 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.411575] ptlrpcd_02_05 126470 6526479.565382 6796714 120 0.000000 81861.501083 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.422995] ptlrpcd_02_06 126471 6526480.557774 6827465 120 0.000000 86544.941759 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.434419] ptlrpcd_02_11 126476 6526479.115045 6786577 120 0.000000 81938.847756 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.445849] ldlm_bl_02 126608 6526477.463119 16175794 120 0.000000 426836.701880 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.457277] mdt_out02_001 126656 6526466.169066 17686330 120 0.000000 286353.549876 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.468701] osp-syn-42-3 126722 6526387.862608 126196 120 0.000000 5593.722349 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.480120] osp-pre-17-3 126737 6526467.301615 214684 120 0.000000 5783.726034 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.491544] osp-pre-7-3 126761 6526462.245470 214632 120 0.000000 4014.527909 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.502965] osp-syn-38-3 126772 6526476.890763 126423 120 0.000000 3526.697280 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.514394] osp-pre-2-1 126916 6526465.646185 213949 120 0.000000 7496.059564 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.525826] osp-pre-30-1 126986 6526467.737722 213952 120 0.000000 6110.834136 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.537251] lquota_wb_fir-M 127047 6526179.046847 17159 120 0.000000 2055.379009 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.548676] ldlm_cn02_003 127204 6526476.305201 8716898 120 0.000000 316436.557754 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.560096] mdt02_007 127216 6328277.477080 27161360 120 0.000000 1291023.501914 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.571522] mdt_out02_004 127318 6526480.521044 19194100 120 0.000000 303493.954915 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.582951] ldlm_cn02_007 129080 6526487.194273 7905880 120 0.000000 342628.050581 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.594379] mdt02_019 129266 6281860.546116 20868275 120 0.000000 1004696.495036 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.605806] mdt02_021 129279 6328277.685631 25100667 120 0.000000 1108361.938912 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.617229] mdt02_022 129594 6328277.476579 23521934 120 0.000000 964254.786172 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.628651] mdt02_024 129793 6328277.476479 20528437 120 0.000000 987870.576107 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.640091] ldlm_bl_06 10647 6526476.929867 1322731 120 0.000000 107940.911081 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.651426] ldlm_bl_10 10651 6526462.129882 1323159 120 0.000000 233443.326746 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.662763] ldlm_bl_16 10657 6526468.680522 1324187 120 0.000000 249118.789992 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.674119] mdt02_051 18267 6328277.476138 85845 120 0.000000 10478.220245 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.685453] mdt02_058 18653 6281860.546217 212189 120 0.000000 161673.949376 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.696789] mdt02_061 18682 6526486.865346 26232 120 0.000000 8127.452405 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.708135] kworker/u259:2 24644 6330344.815052 223 120 0.000000 1873.109651 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.719475] kworker/10:0 24930 6288619.041436 2 120 0.000000 0.034135 0.000000 2 / [Wed Feb 6 09:40:47 2019][1697221.730832] [Wed Feb 6 09:40:47 2019][1697221.732508] cpu#11, 1996.233 MHz [Wed Feb 6 09:40:47 2019][1697221.735914] .nr_running : 0 [Wed Feb 6 09:40:47 2019][1697221.740707] .load : 0 [Wed Feb 6 09:40:47 2019][1697221.745499] .nr_switches : 136470017 [Wed Feb 6 09:40:47 2019][1697221.750986] .nr_load_updates : 77584271 [Wed Feb 6 09:40:47 2019][1697221.756388] .nr_uninterruptible : -946 [Wed Feb 6 09:40:47 2019][1697221.761447] .next_balance : 5991.874533 [Wed Feb 6 09:40:47 2019][1697221.767106] .curr->pid : 0 [Wed Feb 6 09:40:47 2019][1697221.771900] .clock : 1697221771.786273 [Wed Feb 6 09:40:47 2019][1697221.778078] .cpu_load[0] : 0 [Wed Feb 6 09:40:47 2019][1697221.782872] .cpu_load[1] : 0 [Wed Feb 6 09:40:47 2019][1697221.787662] .cpu_load[2] : 0 [Wed Feb 6 09:40:47 2019][1697221.792456] .cpu_load[3] : 0 [Wed Feb 6 09:40:47 2019][1697221.797248] .cpu_load[4] : 0 [Wed Feb 6 09:40:47 2019][1697221.802043] .avg_idle : 7738911 [Wed Feb 6 09:40:47 2019][1697221.807354] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:47 2019][1697221.812667] [Wed Feb 6 09:40:47 2019][1697221.814337] cfs_rq[11]:/ [Wed Feb 6 09:40:47 2019][1697221.817049] .exec_clock : 0.000000 [Wed Feb 6 09:40:47 2019][1697221.822450] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:47 2019][1697221.827849] .min_vruntime : 1966117.255014 [Wed Feb 6 09:40:47 2019][1697221.833767] .max_vruntime : 0.000001 [Wed Feb 6 09:40:47 2019][1697221.839167] .spread : 0.000000 [Wed Feb 6 09:40:47 2019][1697221.844565] .spread0 : -6877208.139812 [Wed Feb 6 09:40:47 2019][1697221.850571] .nr_spread_over : 0 [Wed Feb 6 09:40:47 2019][1697221.855365] .nr_running : 0 [Wed Feb 6 09:40:47 2019][1697221.860158] .load : 0 [Wed Feb 6 09:40:47 2019][1697221.864949] .runnable_load_avg : 0 [Wed Feb 6 09:40:47 2019][1697221.869742] .blocked_load_avg : 0 [Wed Feb 6 09:40:47 2019][1697221.874535] .tg_load_avg : 0 [Wed Feb 6 09:40:47 2019][1697221.879329] .tg_load_contrib : 0 [Wed Feb 6 09:40:47 2019][1697221.884122] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:47 2019][1697221.888915] .tg->runnable_avg : 0 [Wed Feb 6 09:40:47 2019][1697221.893707] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:47 2019][1697221.898499] .throttled : 0 [Wed Feb 6 09:40:47 2019][1697221.903292] .throttle_count : 0 [Wed Feb 6 09:40:47 2019][1697221.908086] .avg->runnable_avg_sum : 12 [Wed Feb 6 09:40:47 2019][1697221.912964] .avg->runnable_avg_period : 46568 [Wed Feb 6 09:40:47 2019][1697221.918107] [Wed Feb 6 09:40:47 2019][1697221.919777] rt_rq[11]:/ [Wed Feb 6 09:40:47 2019][1697221.922403] .rt_nr_running : 0 [Wed Feb 6 09:40:47 2019][1697221.927194] .rt_throttled : 0 [Wed Feb 6 09:40:47 2019][1697221.931987] .rt_time : 0.000000 [Wed Feb 6 09:40:47 2019][1697221.937388] .rt_runtime : 958.767362 [Wed Feb 6 09:40:47 2019][1697221.942961] [Wed Feb 6 09:40:47 2019][1697221.944633] runnable tasks: [Wed Feb 6 09:40:47 2019][1697221.947606] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:47 2019][1697221.958551] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:47 2019][1697221.969418] watchdog/11 66 -11.978850 424305 0 0.000000 38828.424676 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697221.980754] migration/11 67 0.000000 1268 0 0.000000 139312.525257 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697221.992091] ksoftirqd/11 68 1966105.267257 887339 120 0.000000 9947.029903 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.003428] kworker/11:0H 70 3733.010030 10 100 0.000000 0.035416 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.014818] kworker/11:1H 18821 1966105.163154 1157013 100 0.000000 3791.643893 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.026183] mdt_io03_001 126675 1966105.896500 67751 120 0.000000 5261.721128 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.037603] osp-pre-12-3 126705 1966105.178159 214660 120 0.000000 8203.784664 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.049023] osp-pre-14-3 126717 1966105.173290 214631 120 0.000000 6869.678339 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.060448] osp-syn-17-3 126736 1966054.212349 127072 120 0.000000 3622.326011 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.071885] osp-pre-37-1 126976 1966105.188699 213954 120 0.000000 6121.912671 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.083317] ldlm_bl_03 127223 1966105.172528 16236829 120 0.000000 291963.754025 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.094773] kworker/11:2 11106 1966105.354572 115548 120 0.000000 7363.550113 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.106115] mdt_io03_004 40442 1966105.166647 149943 120 0.000000 8949.410783 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.117449] ldlm_cn03_007 51807 1965704.122361 1316705 120 0.000000 161247.438269 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.128794] mdt03_019 17811 1966105.257399 201532 120 0.000000 158766.188860 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.140136] mdt03_032 17925 1925259.157509 32231 120 0.000000 3373.323649 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.151493] ldlm_cn03_029 24611 1966110.123341 10844 120 0.000000 80088.055144 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.162828] kworker/11:1 24841 1964888.078158 8 120 0.000000 0.077486 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.174167] kworker/11:0 25136 1965448.126710 3 120 0.000000 0.020479 0.000000 3 / [Wed Feb 6 09:40:47 2019][1697222.185524] [Wed Feb 6 09:40:48 2019][1697222.187201] cpu#12, 1996.233 MHz [Wed Feb 6 09:40:48 2019][1697222.190609] .nr_running : 0 [Wed Feb 6 09:40:48 2019][1697222.195402] .load : 0 [Wed Feb 6 09:40:48 2019][1697222.200195] .nr_switches : 1804221542 [Wed Feb 6 09:40:48 2019][1697222.205769] .nr_load_updates : 338433399 [Wed Feb 6 09:40:48 2019][1697222.211262] .nr_uninterruptible : 114970 [Wed Feb 6 09:40:48 2019][1697222.216488] .next_balance : 5991.874989 [Wed Feb 6 09:40:48 2019][1697222.222146] .curr->pid : 0 [Wed Feb 6 09:40:48 2019][1697222.226940] .clock : 1697222226.788620 [Wed Feb 6 09:40:48 2019][1697222.233120] .cpu_load[0] : 0 [Wed Feb 6 09:40:48 2019][1697222.237910] .cpu_load[1] : 0 [Wed Feb 6 09:40:48 2019][1697222.242704] .cpu_load[2] : 0 [Wed Feb 6 09:40:48 2019][1697222.247502] .cpu_load[3] : 0 [Wed Feb 6 09:40:48 2019][1697222.252299] .cpu_load[4] : 0 [Wed Feb 6 09:40:48 2019][1697222.257090] .avg_idle : 7737440 [Wed Feb 6 09:40:48 2019][1697222.262405] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:48 2019][1697222.267717] [Wed Feb 6 09:40:48 2019][1697222.269386] cfs_rq[12]:/ [Wed Feb 6 09:40:48 2019][1697222.272100] .exec_clock : 0.000000 [Wed Feb 6 09:40:48 2019][1697222.277497] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:48 2019][1697222.282896] .min_vruntime : 9090041.177351 [Wed Feb 6 09:40:48 2019][1697222.288817] .max_vruntime : 0.000001 [Wed Feb 6 09:40:48 2019][1697222.294217] .spread : 0.000000 [Wed Feb 6 09:40:48 2019][1697222.299614] .spread0 : 246715.179919 [Wed Feb 6 09:40:48 2019][1697222.305448] .nr_spread_over : 0 [Wed Feb 6 09:40:48 2019][1697222.310241] .nr_running : 0 [Wed Feb 6 09:40:48 2019][1697222.315032] .load : 0 [Wed Feb 6 09:40:48 2019][1697222.319825] .runnable_load_avg : 0 [Wed Feb 6 09:40:48 2019][1697222.324619] .blocked_load_avg : 0 [Wed Feb 6 09:40:48 2019][1697222.329411] .tg_load_avg : 0 [Wed Feb 6 09:40:48 2019][1697222.334203] .tg_load_contrib : 0 [Wed Feb 6 09:40:48 2019][1697222.338995] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:48 2019][1697222.343791] .tg->runnable_avg : 0 [Wed Feb 6 09:40:48 2019][1697222.348583] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:48 2019][1697222.353375] .throttled : 0 [Wed Feb 6 09:40:48 2019][1697222.358169] .throttle_count : 0 [Wed Feb 6 09:40:48 2019][1697222.362961] .avg->runnable_avg_sum : 29 [Wed Feb 6 09:40:48 2019][1697222.367840] .avg->runnable_avg_period : 47584 [Wed Feb 6 09:40:48 2019][1697222.372981] [Wed Feb 6 09:40:48 2019][1697222.374651] rt_rq[12]:/ [Wed Feb 6 09:40:48 2019][1697222.377279] .rt_nr_running : 0 [Wed Feb 6 09:40:48 2019][1697222.382070] .rt_throttled : 0 [Wed Feb 6 09:40:48 2019][1697222.386864] .rt_time : 0.000000 [Wed Feb 6 09:40:48 2019][1697222.392264] .rt_runtime : 667.918562 [Wed Feb 6 09:40:48 2019][1697222.397835] [Wed Feb 6 09:40:48 2019][1697222.399508] runnable tasks: [Wed Feb 6 09:40:48 2019][1697222.402482] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:48 2019][1697222.413428] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:48 2019][1697222.424295] watchdog/12 71 -11.981195 424285 0 0.000000 96497.455440 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.435630] migration/12 72 0.000000 1626 0 0.000000 208551.451559 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.446968] ksoftirqd/12 73 9090029.398498 1930043 120 0.000000 25422.846591 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.458303] kworker/12:0H 75 3226.488997 10 100 0.000000 0.050345 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.469672] multipathd 9715 0.000000 73 0 0.000000 4.396325 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.481025] kworker/12:1H 17833 9090029.177950 2102154 100 0.000000 11067.403179 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.492365] cfs_rh_01 126359 9089315.422842 902 120 0.000000 914.182899 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.503788] ptlrpc_hr00_003 126382 9090029.197429 15287053 120 0.000000 126682.010961 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.515218] ptlrpcd_00_07 126448 9090029.189555 4228236 120 0.000000 69176.988740 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.526649] jbd2/dm-2-8 126577 9089471.188076 8712144 120 0.000000 685658.826171 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.538088] osp-pre-41-3 126751 9089452.061673 214711 120 0.000000 7106.865975 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.549525] osp-syn-7-1 126949 9089470.318768 198912 120 0.000000 5087.753899 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.560945] osp-pre-7-1 126950 9089467.779633 213814 120 0.000000 6363.328092 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.572382] mdt_io00_011 127365 9090033.134621 2406770 120 0.000000 100653.327134 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.583827] ldlm_cn00_010 130318 9089440.651864 4232701 120 0.000000 200520.643832 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.595247] ldlm_cn00_016 130324 9089452.648067 1401123 120 0.000000 162734.886749 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.606673] mdt_io00_024 1012 5349190.506404 1543507 120 0.000000 70056.564836 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.618011] kworker/12:0 46894 9090029.197189 154782 120 0.000000 9990.581591 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.629362] mdt00_050 17967 8966801.826786 223622 120 0.000000 323258.403642 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.640696] mdt00_051 17970 8966801.825143 171669 120 0.000000 315186.990130 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.652040] kworker/12:1 23245 8460095.468604 2 120 0.000000 0.008385 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.663389] mdt_rdpg00_009 24902 9089469.349716 9432 120 0.000000 721.816064 0.000000 0 / [Wed Feb 6 09:40:48 2019][1697222.674750] [Wed Feb 6 09:40:48 2019][1697222.676424] cpu#13, 1996.233 MHz [Wed Feb 6 09:40:48 2019][1697222.679832] .nr_running : 0 [Wed Feb 6 09:40:48 2019][1697222.684624] .load : 0 [Wed Feb 6 09:40:48 2019][1697222.689415] .nr_switches : 948196157 [Wed Feb 6 09:40:48 2019][1697222.694902] .nr_load_updates : 175972966 [Wed Feb 6 09:40:48 2019][1697222.700389] .nr_uninterruptible : -21468 [Wed Feb 6 09:40:48 2019][1697222.705613] .next_balance : 5991.875480 [Wed Feb 6 09:40:48 2019][1697222.711274] .curr->pid : 0 [Wed Feb 6 09:40:48 2019][1697222.716068] .clock : 1697222715.792398 [Wed Feb 6 09:40:48 2019][1697222.722253] .cpu_load[0] : 0 [Wed Feb 6 09:40:48 2019][1697222.727047] .cpu_load[1] : 0 [Wed Feb 6 09:40:48 2019][1697222.731839] .cpu_load[2] : 0 [Wed Feb 6 09:40:48 2019][1697222.736635] .cpu_load[3] : 0 [Wed Feb 6 09:40:48 2019][1697222.741435] .cpu_load[4] : 0 [Wed Feb 6 09:40:48 2019][1697222.746226] .avg_idle : 1110468 [Wed Feb 6 09:40:48 2019][1697222.751540] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:48 2019][1697222.756852] [Wed Feb 6 09:40:48 2019][1697222.758522] cfs_rq[13]:/ [Wed Feb 6 09:40:48 2019][1697222.761233] .exec_clock : 0.000000 [Wed Feb 6 09:40:48 2019][1697222.766632] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:48 2019][1697222.772033] .min_vruntime : 6829508.764750 [Wed Feb 6 09:40:48 2019][1697222.777952] .max_vruntime : 0.000001 [Wed Feb 6 09:40:48 2019][1697222.783350] .spread : 0.000000 [Wed Feb 6 09:40:48 2019][1697222.788750] .spread0 : -2013817.864221 [Wed Feb 6 09:40:48 2019][1697222.794758] .nr_spread_over : 0 [Wed Feb 6 09:40:48 2019][1697222.799549] .nr_running : 0 [Wed Feb 6 09:40:48 2019][1697222.804342] .load : 0 [Wed Feb 6 09:40:48 2019][1697222.809135] .runnable_load_avg : 0 [Wed Feb 6 09:40:48 2019][1697222.813928] .blocked_load_avg : 4 [Wed Feb 6 09:40:48 2019][1697222.818722] .tg_load_avg : 0 [Wed Feb 6 09:40:48 2019][1697222.823513] .tg_load_contrib : 0 [Wed Feb 6 09:40:48 2019][1697222.828306] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:48 2019][1697222.833100] .tg->runnable_avg : 0 [Wed Feb 6 09:40:48 2019][1697222.837890] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:48 2019][1697222.842685] .throttled : 0 [Wed Feb 6 09:40:48 2019][1697222.847478] .throttle_count : 0 [Wed Feb 6 09:40:48 2019][1697222.852270] .avg->runnable_avg_sum : 267 [Wed Feb 6 09:40:48 2019][1697222.857236] .avg->runnable_avg_period : 46699 [Wed Feb 6 09:40:48 2019][1697222.862377] [Wed Feb 6 09:40:48 2019][1697222.864049] rt_rq[13]:/ [Wed Feb 6 09:40:48 2019][1697222.866673] .rt_nr_running : 0 [Wed Feb 6 09:40:48 2019][1697222.871467] .rt_throttled : 0 [Wed Feb 6 09:40:48 2019][1697222.876260] .rt_time : 0.007174 [Wed Feb 6 09:40:48 2019][1697222.881659] .rt_runtime : 793.009396 [Wed Feb 6 09:40:48 2019][1697222.887231] [Wed Feb 6 09:40:48 2019][1697222.888904] runnable tasks: [Wed Feb 6 09:40:48 2019][1697222.891876] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:48 2019][1697222.902822] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:48 2019][1697222.913690] watchdog/13 76 -11.979411 424291 0 0.000000 127999.490026 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697222.925025] migration/13 77 0.000000 1797 0 0.000000 237801.402280 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697222.936362] ksoftirqd/13 78 6829211.015083 4106520 120 0.000000 43912.806609 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697222.947698] kworker/13:0H 80 5286.026046 10 100 0.000000 0.035196 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697222.959072] dbus-daemon 16765 6829118.227692 1296590 120 0.000000 127726.173502 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697222.970414] kworker/13:1H 39650 6829210.982191 3293454 100 0.000000 14764.213674 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697222.981759] kiblnd_sd_01_00 126425 6829510.298565 816244096 120 0.000000 5793770.780557 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697222.993196] ldlm_cn01_002 126600 6829449.053976 9349453 120 0.000000 370137.049680 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.004616] mdt01_000 126626 6652617.350327 39954902 120 0.000000 2004665.638515 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.016040] mdt_io01_000 126668 6616065.330064 2844898 120 0.000000 125488.491708 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.027462] osp-pre-0-3 126700 6829494.424856 205473 120 0.000000 4780.484767 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.038889] osp-pre-0-1 126889 6829496.828923 205491 120 0.000000 4925.566710 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.050321] osp-syn-8-1 126979 6829479.859407 198804 120 0.000000 4839.405032 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.061746] osp-syn-28-1 126983 6829479.861521 199153 120 0.000000 4742.839953 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.073172] mdt_io01_005 127222 6829211.301360 2020789 120 0.000000 77580.632864 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.084592] ldlm_cn01_005 127252 6829433.735893 11200407 120 0.000000 417052.628628 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.096017] mdt_io01_017 127696 6630374.459882 1402397 120 0.000000 53666.385056 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.107441] mdt01_016 129230 6652617.350888 34961762 120 0.000000 1329033.152910 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.118861] mdt01_022 129252 6653254.729811 33865438 120 0.000000 1539423.718164 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.130287] mdt01_031 129373 6652645.116314 32674183 120 0.000000 1573291.252378 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.141710] ldlm_cn01_010 131071 6829433.734731 7629930 120 0.000000 323238.987565 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.153130] ldlm_cn01_013 1444 6829433.735272 8111670 120 0.000000 338983.576451 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.164465] mdt_rdpg01_008 18321 6829499.397730 130598 120 0.000000 9809.866502 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.175806] kworker/13:2 110381 6829481.341167 91147 120 0.000000 15389.610947 0.000000 1 / [Wed Feb 6 09:40:48 2019][1697223.187230] kworker/13:0 113688 4275282.370474 2 120 0.000000 0.009307 0.000000 1 / [Wed Feb 6 09:40:49 2019][1697223.198659] ldlm_cn01_017 14201 6829433.735182 599280 120 0.000000 134680.556342 0.000000 1 / [Wed Feb 6 09:40:49 2019][1697223.209992] mdt01_037 17815 6653247.673010 192214 120 0.000000 17294.662326 0.000000 1 / [Wed Feb 6 09:40:49 2019][1697223.221335] mdt01_048 17991 6829499.530532 330823 120 0.000000 317700.700352 0.000000 1 / [Wed Feb 6 09:40:49 2019][1697223.232672] mdt01_049 17992 6652645.117506 48405 120 0.000000 9015.512749 0.000000 1 / [Wed Feb 6 09:40:49 2019][1697223.244008] mdt01_050 17994 6579097.978010 244748 120 0.000000 555849.010246 0.000000 1 / [Wed Feb 6 09:40:49 2019][1697223.255344] mdt01_052 17998 6829301.101263 200745 120 0.000000 172636.257936 0.000000 1 / [Wed Feb 6 09:40:49 2019][1697223.266684] mdt01_055 19060 6829372.590848 302230 120 0.000000 472761.486447 0.000000 1 / [Wed Feb 6 09:40:49 2019][1697223.278018] ldlm_cn01_021 23723 6829433.735121 54573 120 0.000000 135566.945176 0.000000 1 / [Wed Feb 6 09:40:49 2019][1697223.289351] ldlm_cn01_023 23727 6829433.735031 48297 120 0.000000 116783.255453 0.000000 1 / [Wed Feb 6 09:40:49 2019][1697223.300724] [Wed Feb 6 09:40:49 2019][1697223.302398] cpu#14, 1996.233 MHz [Wed Feb 6 09:40:49 2019][1697223.305805] .nr_running : 0 [Wed Feb 6 09:40:49 2019][1697223.310600] .load : 0 [Wed Feb 6 09:40:49 2019][1697223.315399] .nr_switches : 981204511 [Wed Feb 6 09:40:49 2019][1697223.320886] .nr_load_updates : 178987620 [Wed Feb 6 09:40:49 2019][1697223.326372] .nr_uninterruptible : -25457 [Wed Feb 6 09:40:49 2019][1697223.331599] .next_balance : 5991.876102 [Wed Feb 6 09:40:49 2019][1697223.337258] .curr->pid : 0 [Wed Feb 6 09:40:49 2019][1697223.342052] .clock : 1697223341.801336 [Wed Feb 6 09:40:49 2019][1697223.348231] .cpu_load[0] : 0 [Wed Feb 6 09:40:49 2019][1697223.353023] .cpu_load[1] : 0 [Wed Feb 6 09:40:49 2019][1697223.357814] .cpu_load[2] : 0 [Wed Feb 6 09:40:49 2019][1697223.362609] .cpu_load[3] : 0 [Wed Feb 6 09:40:49 2019][1697223.367400] .cpu_load[4] : 0 [Wed Feb 6 09:40:49 2019][1697223.372194] .avg_idle : 7748816 [Wed Feb 6 09:40:49 2019][1697223.377507] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:49 2019][1697223.382820] [Wed Feb 6 09:40:49 2019][1697223.384487] cfs_rq[14]:/ [Wed Feb 6 09:40:49 2019][1697223.387201] .exec_clock : 0.000000 [Wed Feb 6 09:40:49 2019][1697223.392599] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:49 2019][1697223.398000] .min_vruntime : 4931091.412912 [Wed Feb 6 09:40:49 2019][1697223.403918] .max_vruntime : 0.000001 [Wed Feb 6 09:40:49 2019][1697223.409319] .spread : 0.000000 [Wed Feb 6 09:40:49 2019][1697223.414718] .spread0 : -3912236.055087 [Wed Feb 6 09:40:49 2019][1697223.420723] .nr_spread_over : 0 [Wed Feb 6 09:40:49 2019][1697223.425517] .nr_running : 0 [Wed Feb 6 09:40:49 2019][1697223.430309] .load : 0 [Wed Feb 6 09:40:49 2019][1697223.435101] .runnable_load_avg : 0 [Wed Feb 6 09:40:49 2019][1697223.439896] .blocked_load_avg : 5 [Wed Feb 6 09:40:49 2019][1697223.444688] .tg_load_avg : 0 [Wed Feb 6 09:40:49 2019][1697223.449479] .tg_load_contrib : 0 [Wed Feb 6 09:40:49 2019][1697223.454272] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:49 2019][1697223.459066] .tg->runnable_avg : 0 [Wed Feb 6 09:40:49 2019][1697223.463859] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:49 2019][1697223.468652] .throttled : 0 [Wed Feb 6 09:40:49 2019][1697223.473444] .throttle_count : 0 [Wed Feb 6 09:40:49 2019][1697223.478237] .avg->runnable_avg_sum : 14 [Wed Feb 6 09:40:49 2019][1697223.483117] .avg->runnable_avg_period : 46628 [Wed Feb 6 09:40:49 2019][1697223.488256] [Wed Feb 6 09:40:49 2019][1697223.489928] rt_rq[14]:/ [Wed Feb 6 09:40:49 2019][1697223.492553] .rt_nr_running : 0 [Wed Feb 6 09:40:49 2019][1697223.497348] .rt_throttled : 0 [Wed Feb 6 09:40:49 2019][1697223.502141] .rt_time : 0.006623 [Wed Feb 6 09:40:49 2019][1697223.507540] .rt_runtime : 958.767419 [Wed Feb 6 09:40:49 2019][1697223.513111] [Wed Feb 6 09:40:49 2019][1697223.514784] runnable tasks: [Wed Feb 6 09:40:49 2019][1697223.517757] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:49 2019][1697223.528704] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:49 2019][1697223.539568] watchdog/14 81 -11.980313 424289 0 0.000000 51269.037110 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.550906] migration/14 82 0.000000 1619 0 0.000000 284576.569801 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.562243] ksoftirqd/14 83 4931078.083229 3269917 120 0.000000 37208.935439 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.573579] kworker/14:0H 85 3594.323807 9 100 0.000000 0.035134 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.584938] kworker/14:1 7269 4931079.403596 302686 120 0.000000 13824.327442 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.596283] kworker/14:1H 16416 4930997.808977 2245158 100 0.000000 9273.031462 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.607616] rpciod 16731 4643736.178390 4 100 0.000000 848.545616 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.618955] writer#2 18228 4931078.310335 25115108 120 0.000000 228700.687018 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.630293] cfs_rh_00 126358 4931068.587483 903 120 0.000000 966.830159 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.641730] ptlrpcd_02_07 126472 4931079.221749 6847530 120 0.000000 86711.845615 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.653162] osp-syn-46-3 126710 4931028.023754 126550 120 0.000000 4871.381500 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.664584] osp-syn-44-3 126742 4931078.160475 126307 120 0.000000 3604.438300 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.676006] osp-syn-7-3 126760 4931056.303070 126345 120 0.000000 5780.675197 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.687433] osp-syn-42-1 126911 4931079.091143 196609 120 0.000000 4927.039923 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.698858] osp-syn-32-1 126933 4931079.207322 198502 120 0.000000 4752.516821 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.710284] osp-syn-3-1 126971 4931077.203787 197574 120 0.000000 4752.234049 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.721705] osp-pre-30-1 126986 4931079.190330 213953 120 0.000000 6110.856448 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.733132] ldlm_cb02_003 127234 4850161.527330 31781796 120 0.000000 453936.690425 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.744561] mdt_io02_012 127535 4773216.263100 1064973 120 0.000000 48100.333341 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.755998] mdt02_026 129806 4769498.779322 23936893 120 0.000000 1344604.841822 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.767438] ldlm_bl_11 10652 4931078.232361 1323591 120 0.000000 395439.670101 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.778775] ldlm_bl_13 10654 4931078.704211 1322615 120 0.000000 857058.271206 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.790146] kworker/14:2 24676 4649748.052371 4 120 0.000000 0.057870 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.801486] sshd 25138 4930732.003069 122 120 0.000000 16.985975 0.000000 2 / [Wed Feb 6 09:40:49 2019][1697223.812860] [Wed Feb 6 09:40:49 2019][1697223.814535] cpu#15, 1996.233 MHz [Wed Feb 6 09:40:49 2019][1697223.817944] .nr_running : 0 [Wed Feb 6 09:40:49 2019][1697223.822736] .load : 0 [Wed Feb 6 09:40:49 2019][1697223.827529] .nr_switches : 144051026 [Wed Feb 6 09:40:49 2019][1697223.833015] .nr_load_updates : 79913262 [Wed Feb 6 09:40:49 2019][1697223.838415] .nr_uninterruptible : -1199 [Wed Feb 6 09:40:49 2019][1697223.843555] .next_balance : 5991.876616 [Wed Feb 6 09:40:49 2019][1697223.849215] .curr->pid : 0 [Wed Feb 6 09:40:49 2019][1697223.854014] .clock : 1697223853.806948 [Wed Feb 6 09:40:49 2019][1697223.860195] .cpu_load[0] : 0 [Wed Feb 6 09:40:49 2019][1697223.864986] .cpu_load[1] : 0 [Wed Feb 6 09:40:49 2019][1697223.869779] .cpu_load[2] : 0 [Wed Feb 6 09:40:49 2019][1697223.874573] .cpu_load[3] : 1 [Wed Feb 6 09:40:49 2019][1697223.879366] .cpu_load[4] : 3 [Wed Feb 6 09:40:49 2019][1697223.884167] .avg_idle : 768973 [Wed Feb 6 09:40:49 2019][1697223.889391] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:49 2019][1697223.894706] [Wed Feb 6 09:40:49 2019][1697223.896374] cfs_rq[15]:/ [Wed Feb 6 09:40:49 2019][1697223.899088] .exec_clock : 0.000000 [Wed Feb 6 09:40:49 2019][1697223.904486] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:49 2019][1697223.909885] .min_vruntime : 1650435.767627 [Wed Feb 6 09:40:49 2019][1697223.915806] .max_vruntime : 0.000001 [Wed Feb 6 09:40:49 2019][1697223.921205] .spread : 0.000000 [Wed Feb 6 09:40:49 2019][1697223.926603] .spread0 : -7192892.336108 [Wed Feb 6 09:40:49 2019][1697223.932610] .nr_spread_over : 0 [Wed Feb 6 09:40:49 2019][1697223.937403] .nr_running : 0 [Wed Feb 6 09:40:49 2019][1697223.942195] .load : 0 [Wed Feb 6 09:40:49 2019][1697223.946987] .runnable_load_avg : 0 [Wed Feb 6 09:40:49 2019][1697223.951780] .blocked_load_avg : 29 [Wed Feb 6 09:40:49 2019][1697223.956661] .tg_load_avg : 0 [Wed Feb 6 09:40:49 2019][1697223.961453] .tg_load_contrib : 0 [Wed Feb 6 09:40:49 2019][1697223.966245] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:49 2019][1697223.971038] .tg->runnable_avg : 0 [Wed Feb 6 09:40:49 2019][1697223.975830] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:49 2019][1697223.980624] .throttled : 0 [Wed Feb 6 09:40:49 2019][1697223.985416] .throttle_count : 0 [Wed Feb 6 09:40:49 2019][1697223.990210] .avg->runnable_avg_sum : 1183 [Wed Feb 6 09:40:49 2019][1697223.995262] .avg->runnable_avg_period : 47805 [Wed Feb 6 09:40:49 2019][1697224.000405] [Wed Feb 6 09:40:49 2019][1697224.002075] rt_rq[15]:/ [Wed Feb 6 09:40:49 2019][1697224.004700] .rt_nr_running : 0 [Wed Feb 6 09:40:49 2019][1697224.009493] .rt_throttled : 0 [Wed Feb 6 09:40:49 2019][1697224.014286] .rt_time : 0.000000 [Wed Feb 6 09:40:49 2019][1697224.019684] .rt_runtime : 979.166667 [Wed Feb 6 09:40:49 2019][1697224.025256] [Wed Feb 6 09:40:49 2019][1697224.026930] runnable tasks: [Wed Feb 6 09:40:49 2019][1697224.029903] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:49 2019][1697224.040850] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:49 2019][1697224.051718] watchdog/15 86 -11.979220 424305 0 0.000000 45264.569967 0.000000 3 / [Wed Feb 6 09:40:49 2019][1697224.063053] migration/15 87 0.000000 1288 0 0.000000 156368.655840 0.000000 3 / [Wed Feb 6 09:40:49 2019][1697224.074390] ksoftirqd/15 88 1650398.949291 687866 120 0.000000 9125.867783 0.000000 3 / [Wed Feb 6 09:40:49 2019][1697224.085724] kworker/15:0H 90 7982.425146 10 100 0.000000 0.037490 0.000000 3 / [Wed Feb 6 09:40:49 2019][1697224.097081] writeback 258 1586873.426854 4 100 0.000000 0.631950 0.000000 3 / [Wed Feb 6 09:40:49 2019][1697224.108417] khugepaged 283 1650422.139329 171146 139 0.000000 247601.636185 0.000000 3 / [Wed Feb 6 09:40:49 2019][1697224.119753] kworker/15:1 7270 1650413.498622 145273 120 0.000000 12780.025708 0.000000 3 / [Wed Feb 6 09:40:49 2019][1697224.131110] reader#2 18233 1650394.255292 171097 120 0.000000 30538.204051 0.000000 3 / [Wed Feb 6 09:40:49 2019][1697224.142442] reader#3 18234 1650412.530111 186744 120 0.000000 15740.238457 0.000000 3 / [Wed Feb 6 09:40:49 2019][1697224.153778] master 18601 1650330.519644 97709 120 0.000000 3921.294739 0.000000 3 / [Wed Feb 6 09:40:49 2019][1697224.165117] kworker/15:1H 44173 1650398.941299 1132038 100 0.000000 3913.803130 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.176454] ptlrpc_hr03_000 126398 1650412.984382 1678749 120 0.000000 14665.776954 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.187876] ptlrpcd_rcv 126440 1650186.629617 1176 120 0.000000 10287.928665 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.199303] ptlrpcd_03_05 126482 1650420.681575 5615337 120 0.000000 82198.273471 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.210733] mdt03_000 126632 1605628.159046 6811106 120 0.000000 435689.961639 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.222158] mdt03_001 126633 1606170.525281 7730636 120 0.000000 471156.663658 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.233588] osp-pre-12-3 126705 1650412.996806 214661 120 0.000000 8203.805754 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.245012] osp-pre-13-3 126707 1650412.989713 214650 120 0.000000 8305.475988 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.256438] osp-syn-2-3 126726 1650248.070412 127872 120 0.000000 3615.668460 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.267858] osp-pre-18-3 126733 1650412.993459 214711 120 0.000000 7861.935843 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.279283] osp-pre-40-3 126763 1650412.995764 214624 120 0.000000 6673.596690 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.290704] osp-syn-45-3 126764 1650424.508773 127852 120 0.000000 4796.671724 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.302129] lquota_wb_fir-M 126871 1650186.624078 17159 120 0.000000 1623.443585 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.313551] osp-pre-11-1 126904 1650412.835734 213770 120 0.000000 8942.687991 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.324975] osp-pre-43-1 126918 1650412.986436 213920 120 0.000000 3052.303223 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.336398] osp-syn-15-1 126963 1650412.748541 198083 120 0.000000 4795.006187 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.347823] mdt03_006 127196 1605343.136283 7781398 120 0.000000 919786.104289 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.359246] mdt03_007 127254 1650328.286691 7093200 120 0.000000 296955.699558 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.370714] ldlm_bl_05 10646 1650412.988360 1326594 120 0.000000 153142.114151 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.382051] ldlm_bl_07 10648 1650439.782130 1322580 120 0.000000 93007.893570 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.393395] mdt03_022 17828 1606482.605053 39084 120 0.000000 155657.719224 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.404735] mdt03_025 17850 1650441.964747 209368 120 0.000000 313059.052136 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.416070] mdt03_028 17890 1650412.128807 44561 120 0.000000 5443.112193 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.427410] mdt03_036 17963 1605343.136955 25105 120 0.000000 1340.871981 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.438756] mdt03_047 18260 1605245.193999 40581 120 0.000000 943.444135 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.450100] mdt_seqm_0003 23717 1650282.406866 112 120 0.000000 2.699409 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.461436] ldlm_cn03_020 24583 1650450.495901 12556 120 0.000000 92048.331653 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.472782] kworker/15:0 25013 1624154.786594 17 120 0.000000 0.180882 0.000000 3 / [Wed Feb 6 09:40:50 2019][1697224.484166] [Wed Feb 6 09:40:50 2019][1697224.485839] cpu#16, 1996.233 MHz [Wed Feb 6 09:40:50 2019][1697224.489245] .nr_running : 0 [Wed Feb 6 09:40:50 2019][1697224.494038] .load : 0 [Wed Feb 6 09:40:50 2019][1697224.498831] .nr_switches : 1243973089 [Wed Feb 6 09:40:50 2019][1697224.504404] .nr_load_updates : 274466473 [Wed Feb 6 09:40:50 2019][1697224.509891] .nr_uninterruptible : 97863 [Wed Feb 6 09:40:50 2019][1697224.515029] .next_balance : 5991.877287 [Wed Feb 6 09:40:50 2019][1697224.520690] .curr->pid : 0 [Wed Feb 6 09:40:50 2019][1697224.525482] .clock : 1697224524.807658 [Wed Feb 6 09:40:50 2019][1697224.531663] .cpu_load[0] : 0 [Wed Feb 6 09:40:50 2019][1697224.536464] .cpu_load[1] : 0 [Wed Feb 6 09:40:50 2019][1697224.541257] .cpu_load[2] : 144 [Wed Feb 6 09:40:50 2019][1697224.546229] .cpu_load[3] : 49 [Wed Feb 6 09:40:50 2019][1697224.551110] .cpu_load[4] : 26 [Wed Feb 6 09:40:50 2019][1697224.555989] .avg_idle : 3603184 [Wed Feb 6 09:40:50 2019][1697224.561302] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:50 2019][1697224.566615] [Wed Feb 6 09:40:50 2019][1697224.568284] cfs_rq[16]:/ [Wed Feb 6 09:40:50 2019][1697224.570995] .exec_clock : 0.000000 [Wed Feb 6 09:40:50 2019][1697224.576396] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:50 2019][1697224.581796] .min_vruntime : 11646872.161499 [Wed Feb 6 09:40:50 2019][1697224.587801] .max_vruntime : 0.000001 [Wed Feb 6 09:40:50 2019][1697224.593200] .spread : 0.000000 [Wed Feb 6 09:40:50 2019][1697224.598599] .spread0 : 2803519.008486 [Wed Feb 6 09:40:50 2019][1697224.604520] .nr_spread_over : 0 [Wed Feb 6 09:40:50 2019][1697224.609311] .nr_running : 0 [Wed Feb 6 09:40:50 2019][1697224.614105] .load : 0 [Wed Feb 6 09:40:50 2019][1697224.618897] .runnable_load_avg : 0 [Wed Feb 6 09:40:50 2019][1697224.623690] .blocked_load_avg : 0 [Wed Feb 6 09:40:50 2019][1697224.628483] .tg_load_avg : 0 [Wed Feb 6 09:40:50 2019][1697224.633274] .tg_load_contrib : 0 [Wed Feb 6 09:40:50 2019][1697224.638067] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:50 2019][1697224.642862] .tg->runnable_avg : 0 [Wed Feb 6 09:40:50 2019][1697224.647654] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:50 2019][1697224.652446] .throttled : 0 [Wed Feb 6 09:40:50 2019][1697224.657240] .throttle_count : 0 [Wed Feb 6 09:40:50 2019][1697224.662032] .avg->runnable_avg_sum : 7876 [Wed Feb 6 09:40:50 2019][1697224.667085] .avg->runnable_avg_period : 46885 [Wed Feb 6 09:40:50 2019][1697224.672232] [Wed Feb 6 09:40:50 2019][1697224.673907] rt_rq[16]:/ [Wed Feb 6 09:40:50 2019][1697224.676531] .rt_nr_running : 0 [Wed Feb 6 09:40:50 2019][1697224.681325] .rt_throttled : 0 [Wed Feb 6 09:40:50 2019][1697224.686116] .rt_time : 0.000000 [Wed Feb 6 09:40:50 2019][1697224.691516] .rt_runtime : 961.936854 [Wed Feb 6 09:40:50 2019][1697224.697088] [Wed Feb 6 09:40:50 2019][1697224.698761] runnable tasks: [Wed Feb 6 09:40:50 2019][1697224.701735] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:50 2019][1697224.712681] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:50 2019][1697224.723551] watchdog/16 91 -11.983890 424297 0 0.000000 60675.248555 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.734885] migration/16 92 0.000000 2041 0 0.000000 234019.320087 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.746220] ksoftirqd/16 93 11642646.266986 3077740 120 0.000000 32847.690909 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.757557] kworker/16:0H 95 3682.435989 9 100 0.000000 0.033676 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.768925] jbd2/sdk2-8 7868 11646830.938887 239806 120 0.000000 15369.459609 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.780264] kmpath_handlerd 9276 2670.863854 2 100 0.000000 0.006613 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.791608] kworker/16:1H 16049 11646826.334650 2898432 100 0.000000 14715.161286 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.802958] rs:main Q:Reg 18242 11646862.413743 144985 120 0.000000 8811.949375 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.814304] kiblnd_sd_00_02 126423 11646881.514929 1556592703 120 0.000000 9507289.669953 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.825820] ptlrpcd_00_09 126450 11646868.572259 4237535 120 0.000000 63265.322575 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.837246] ldlm_cb00_001 126587 151664.880530 102 120 0.000000 2.368295 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.848672] ldlm_cn00_001 126596 11646860.201204 2474914 120 0.000000 235784.904027 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.860096] mdt00_000 126623 11646771.799529 79856699 120 0.000000 2157375.922991 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.871524] ll_evictor 126688 11635988.984418 99 120 0.000000 1130.258671 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.882949] osp_up0-3 126699 6979459.190229 15 120 0.000000 0.410975 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.894381] osp_up2-1 126886 11371979.945209 259 120 0.000000 7.103433 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.905806] osp-syn-17-1 126925 11646867.330987 198624 120 0.000000 6889.765425 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.917229] osp-pre-47-1 126960 11646830.834209 213942 120 0.000000 4809.144524 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.928647] osp-syn-29-1 126965 11646867.308153 199153 120 0.000000 4773.047423 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.940075] mdt00_005 127136 11520015.021923 66662146 120 0.000000 2119126.613370 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.951498] mdt_io00_004 127225 11510532.591193 1208214 120 0.000000 48607.115071 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.962918] ldlm_bl_04 127238 11646860.171027 15945027 120 0.000000 320814.469824 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.974348] ldlm_cn00_008 128040 11646860.209800 4751574 120 0.000000 245550.392192 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.985769] mdt00_011 128084 11509828.606548 77497536 120 0.000000 2300783.973804 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697224.997209] ldlm_cn00_010 130318 11646860.202336 4232703 120 0.000000 200520.649613 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697225.008632] ldlm_cn00_012 130320 11646869.025945 3601661 120 0.000000 228030.769971 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697225.020055] ldlm_cn00_014 130322 11646860.209619 2528130 120 0.000000 196349.448641 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697225.031478] ldlm_cn00_015 130323 11646860.236100 2725583 120 0.000000 188084.206362 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697225.042900] ldlm_cn00_016 130324 11646860.243814 1401131 120 0.000000 162740.232087 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697225.054325] ldlm_cn00_021 130329 11646860.265524 2960188 120 0.000000 240426.452994 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697225.065746] ldlm_cn00_025 130333 11646811.287301 2714804 120 0.000000 167084.968215 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697225.077170] ldlm_cn00_027 130337 11646811.966559 3412863 120 0.000000 213117.460577 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697225.088592] ldlm_cn00_031 130353 11646872.202336 2926650 120 0.000000 195407.553305 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697225.100017] mdt_io00_025 1416 11646648.623502 1679211 120 0.000000 75633.855623 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697225.111353] kworker/16:2 2843 11646844.939995 260593 120 0.000000 22367.821265 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697225.122700] ldlm_bl_17 10658 11646860.170185 1329286 120 0.000000 10054.898419 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697225.134034] mdt_fld_0003 11594 11646331.541841 1121 120 0.000000 13.733763 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697225.145377] mdt00_034 17894 11520681.490186 72709 120 0.000000 14013.255800 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697225.156717] mdt00_039 17911 11520034.711039 159507 120 0.000000 151702.926469 0.000000 0 / [Wed Feb 6 09:40:50 2019][1697225.168050] mdt00_044 17932 11646832.487410 45254 120 0.000000 163302.367345 0.000000 0 / [Wed Feb 6 09:40:51 2019][1697225.179387] mdt00_045 17935 11520035.542445 39954 120 0.000000 163261.954378 0.000000 0 / [Wed Feb 6 09:40:51 2019][1697225.190721] mdt00_046 17939 11520015.022905 158357 120 0.000000 312703.792841 0.000000 0 / [Wed Feb 6 09:40:51 2019][1697225.202058] mdt00_049 17953 11646771.759224 201515 120 0.000000 313149.069721 0.000000 0 / [Wed Feb 6 09:40:51 2019][1697225.213407] kworker/16:0 19775 10677592.647456 9 120 0.000000 0.051685 0.000000 0 / [Wed Feb 6 09:40:51 2019][1697225.224751] mdt_rdpg00_007 24894 11646880.992778 12273 120 0.000000 2628.318205 0.000000 0 / [Wed Feb 6 09:40:51 2019][1697225.236346] [Wed Feb 6 09:40:51 2019][1697225.238021] cpu#17, 1996.233 MHz [Wed Feb 6 09:40:51 2019][1697225.241428] .nr_running : 0 [Wed Feb 6 09:40:51 2019][1697225.246218] .load : 0 [Wed Feb 6 09:40:51 2019][1697225.251012] .nr_switches : 765479783 [Wed Feb 6 09:40:51 2019][1697225.256499] .nr_load_updates : 173711033 [Wed Feb 6 09:40:51 2019][1697225.261984] .nr_uninterruptible : -44928 [Wed Feb 6 09:40:51 2019][1697225.267211] .next_balance : 5991.878041 [Wed Feb 6 09:40:51 2019][1697225.272870] .curr->pid : 0 [Wed Feb 6 09:40:51 2019][1697225.277661] .clock : 1697225276.813561 [Wed Feb 6 09:40:51 2019][1697225.283840] .cpu_load[0] : 0 [Wed Feb 6 09:40:51 2019][1697225.288634] .cpu_load[1] : 0 [Wed Feb 6 09:40:51 2019][1697225.293426] .cpu_load[2] : 0 [Wed Feb 6 09:40:51 2019][1697225.298220] .cpu_load[3] : 0 [Wed Feb 6 09:40:51 2019][1697225.303012] .cpu_load[4] : 0 [Wed Feb 6 09:40:51 2019][1697225.307806] .avg_idle : 2316431 [Wed Feb 6 09:40:51 2019][1697225.313119] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:51 2019][1697225.318430] [Wed Feb 6 09:40:51 2019][1697225.320100] cfs_rq[17]:/ [Wed Feb 6 09:40:51 2019][1697225.322813] .exec_clock : 0.000000 [Wed Feb 6 09:40:51 2019][1697225.328212] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:51 2019][1697225.333612] .min_vruntime : 7958773.986519 [Wed Feb 6 09:40:51 2019][1697225.339531] .max_vruntime : 0.000001 [Wed Feb 6 09:40:51 2019][1697225.344931] .spread : 0.000000 [Wed Feb 6 09:40:51 2019][1697225.350329] .spread0 : -884590.953042 [Wed Feb 6 09:40:51 2019][1697225.356249] .nr_spread_over : 0 [Wed Feb 6 09:40:51 2019][1697225.361041] .nr_running : 0 [Wed Feb 6 09:40:51 2019][1697225.365836] .load : 0 [Wed Feb 6 09:40:51 2019][1697225.370627] .runnable_load_avg : 0 [Wed Feb 6 09:40:51 2019][1697225.375419] .blocked_load_avg : 0 [Wed Feb 6 09:40:51 2019][1697225.380214] .tg_load_avg : 0 [Wed Feb 6 09:40:51 2019][1697225.385005] .tg_load_contrib : 0 [Wed Feb 6 09:40:51 2019][1697225.389799] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:51 2019][1697225.394590] .tg->runnable_avg : 0 [Wed Feb 6 09:40:51 2019][1697225.399383] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:51 2019][1697225.404176] .throttled : 0 [Wed Feb 6 09:40:51 2019][1697225.408969] .throttle_count : 0 [Wed Feb 6 09:40:51 2019][1697225.413763] .avg->runnable_avg_sum : 53 [Wed Feb 6 09:40:51 2019][1697225.418641] .avg->runnable_avg_period : 47920 [Wed Feb 6 09:40:51 2019][1697225.423787] [Wed Feb 6 09:40:51 2019][1697225.425462] rt_rq[17]:/ [Wed Feb 6 09:40:51 2019][1697225.428088] .rt_nr_running : 0 [Wed Feb 6 09:40:51 2019][1697225.432881] .rt_throttled : 0 [Wed Feb 6 09:40:51 2019][1697225.437674] .rt_time : 0.000000 [Wed Feb 6 09:40:51 2019][1697225.443074] .rt_runtime : 958.767362 [Wed Feb 6 09:40:51 2019][1697225.448645] [Wed Feb 6 09:40:51 2019][1697225.450316] runnable tasks: [Wed Feb 6 09:40:51 2019][1697225.453290] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:51 2019][1697225.464236] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:51 2019][1697225.475107] watchdog/17 96 -11.979351 424280 0 0.000000 106040.075216 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.486442] migration/17 97 0.000000 3156 0 0.000000 252299.315665 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.497777] ksoftirqd/17 98 7958763.291995 5169191 120 0.000000 51911.984692 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.509114] kworker/17:0H 100 3407.088421 9 100 0.000000 0.038525 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.520480] multipathd 9693 3656.836591 40 120 0.000000 1.239968 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.531823] kworker/17:1H 16409 7958763.280682 3961813 100 0.000000 20556.108863 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.543168] bioset 18672 3601.636087 2 100 0.000000 0.013545 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.554511] kiblnd_connd 126419 7958759.412026 1031818 120 0.000000 24971.727180 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.565939] ptlrpcd_01_00 126453 7958759.389435 4004102 120 0.000000 62615.137144 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.577357] ptlrpcd_01_02 126455 7958759.354526 4015765 120 0.000000 61227.935180 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.588791] mdt01_001 126627 7853781.833928 37452326 120 0.000000 1585543.610369 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.600211] mdt_rdpg01_000 126637 7958716.328042 3546730 120 0.000000 161433.678804 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.611657] osp-syn-30-1 126985 7958759.273163 197480 120 0.000000 4654.306423 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.623090] mdt_io01_009 127261 5265317.119343 2977432 120 0.000000 121382.299516 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.634515] mdt_io01_011 127336 4713761.362557 1026738 120 0.000000 35108.844984 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.645938] mdt_io01_014 127691 7958763.344925 2459488 120 0.000000 92338.007229 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.657366] ldlm_cn01_008 129107 7958771.196970 7831258 120 0.000000 346860.004908 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.668790] mdt01_014 129228 7853781.834801 39536331 120 0.000000 1823319.134631 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.680215] mdt01_018 129242 7881682.544822 33626088 120 0.000000 1837613.601820 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.691635] mdt01_019 129243 7958778.140663 32648918 120 0.000000 1327757.251515 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.703060] mdt01_025 129255 7853781.893180 32482078 120 0.000000 1304347.883222 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.714484] mdt01_027 129265 7853781.834580 21495597 120 0.000000 794338.602573 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.725905] mdt_io01_022 129281 7932839.276374 2846945 120 0.000000 108022.492175 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.737327] mdt01_029 129371 7853781.896646 27043422 120 0.000000 996914.845064 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.748752] mdt01_035 129797 7853781.840601 31135006 120 0.000000 1282956.622769 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.760185] kworker/17:2 34465 7958761.845227 151310 120 0.000000 20721.555095 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.771541] mdt01_044 17973 7853781.834490 182006 120 0.000000 138137.238720 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.782888] mdt01_058 19087 7853781.834479 34149 120 0.000000 11755.163731 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.794224] mdt01_060 19113 7853781.892969 80403 120 0.000000 15711.388674 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.805576] kworker/17:0 24962 7861110.786009 2 120 0.000000 0.020709 0.000000 1 / [Wed Feb 6 09:40:51 2019][1697225.816940] [Wed Feb 6 09:40:51 2019][1697225.818616] cpu#18, 1996.233 MHz [Wed Feb 6 09:40:51 2019][1697225.822023] .nr_running : 0 [Wed Feb 6 09:40:51 2019][1697225.826815] .load : 1024 [Wed Feb 6 09:40:51 2019][1697225.831868] .nr_switches : 1332872961 [Wed Feb 6 09:40:51 2019][1697225.837441] .nr_load_updates : 173677192 [Wed Feb 6 09:40:51 2019][1697225.842927] .nr_uninterruptible : -31261 [Wed Feb 6 09:40:51 2019][1697225.848153] .next_balance : 5991.878622 [Wed Feb 6 09:40:51 2019][1697225.853811] .curr->pid : 0 [Wed Feb 6 09:40:51 2019][1697225.858604] .clock : 1697225857.818404 [Wed Feb 6 09:40:51 2019][1697225.864786] .cpu_load[0] : 0 [Wed Feb 6 09:40:51 2019][1697225.869578] .cpu_load[1] : 0 [Wed Feb 6 09:40:51 2019][1697225.874370] .cpu_load[2] : 0 [Wed Feb 6 09:40:51 2019][1697225.879162] .cpu_load[3] : 0 [Wed Feb 6 09:40:51 2019][1697225.883955] .cpu_load[4] : 0 [Wed Feb 6 09:40:51 2019][1697225.888748] .avg_idle : 2874959 [Wed Feb 6 09:40:51 2019][1697225.894059] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:51 2019][1697225.899374] [Wed Feb 6 09:40:51 2019][1697225.901043] cfs_rq[18]:/ [Wed Feb 6 09:40:51 2019][1697225.903757] .exec_clock : 0.000000 [Wed Feb 6 09:40:51 2019][1697225.909156] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:51 2019][1697225.914555] .min_vruntime : 3787955.273245 [Wed Feb 6 09:40:51 2019][1697225.920474] .max_vruntime : 0.000001 [Wed Feb 6 09:40:51 2019][1697225.925872] .spread : 0.000000 [Wed Feb 6 09:40:51 2019][1697225.931273] .spread0 : -5055409.666316 [Wed Feb 6 09:40:51 2019][1697225.937278] .nr_spread_over : 0 [Wed Feb 6 09:40:51 2019][1697225.942072] .nr_running : 0 [Wed Feb 6 09:40:51 2019][1697225.946864] .load : 0 [Wed Feb 6 09:40:51 2019][1697225.951658] .runnable_load_avg : 0 [Wed Feb 6 09:40:51 2019][1697225.956451] .blocked_load_avg : 8 [Wed Feb 6 09:40:51 2019][1697225.961243] .tg_load_avg : 0 [Wed Feb 6 09:40:51 2019][1697225.966035] .tg_load_contrib : 0 [Wed Feb 6 09:40:51 2019][1697225.970828] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:51 2019][1697225.975620] .tg->runnable_avg : 0 [Wed Feb 6 09:40:51 2019][1697225.980412] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:51 2019][1697225.985205] .throttled : 0 [Wed Feb 6 09:40:51 2019][1697225.989999] .throttle_count : 0 [Wed Feb 6 09:40:51 2019][1697225.994792] .avg->runnable_avg_sum : 68 [Wed Feb 6 09:40:51 2019][1697225.999671] .avg->runnable_avg_period : 46574 [Wed Feb 6 09:40:51 2019][1697226.004814] [Wed Feb 6 09:40:51 2019][1697226.006482] rt_rq[18]:/ [Wed Feb 6 09:40:51 2019][1697226.009110] .rt_nr_running : 0 [Wed Feb 6 09:40:51 2019][1697226.013902] .rt_throttled : 0 [Wed Feb 6 09:40:51 2019][1697226.018695] .rt_time : 0.000000 [Wed Feb 6 09:40:51 2019][1697226.024093] .rt_runtime : 900.084181 [Wed Feb 6 09:40:51 2019][1697226.029666] [Wed Feb 6 09:40:51 2019][1697226.031340] runnable tasks: [Wed Feb 6 09:40:51 2019][1697226.034311] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:51 2019][1697226.045257] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:51 2019][1697226.056131] watchdog/18 101 -11.980343 424292 0 0.000000 65785.057441 0.000000 2 / [Wed Feb 6 09:40:51 2019][1697226.067461] migration/18 102 0.000000 2104 0 0.000000 262345.974385 0.000000 2 / [Wed Feb 6 09:40:51 2019][1697226.078799] ksoftirqd/18 103 3787943.451312 1558972 120 0.000000 21942.082798 0.000000 2 / [Wed Feb 6 09:40:51 2019][1697226.090134] kworker/18:0H 105 3126.746188 9 100 0.000000 0.036590 0.000000 2 / [Wed Feb 6 09:40:51 2019][1697226.101510] kworker/18:1H 16554 3787943.461492 1594722 100 0.000000 7753.029432 0.000000 2 / [Wed Feb 6 09:40:51 2019][1697226.112847] reader#0 18231 3787817.419555 9222074 120 0.000000 659060.950779 0.000000 2 / [Wed Feb 6 09:40:51 2019][1697226.124197] ptlrpc_hr02_003 126395 3787943.193086 15299444 120 0.000000 122074.876647 0.000000 2 / [Wed Feb 6 09:40:51 2019][1697226.135623] kiblnd_sd_02_02 126431 6526582.559832 720339462 120 0.000000 4648048.662206 0.000000 2 / [Wed Feb 6 09:40:51 2019][1697226.147042] kiblnd_sd_02_03 126432 6526582.720880 720358179 120 0.000000 4642851.952696 0.000000 2 / [Wed Feb 6 09:40:51 2019][1697226.158469] ptlrpcd_02_05 126470 3787942.961900 6796726 120 0.000000 81861.817909 0.000000 2 / [Wed Feb 6 09:40:51 2019][1697226.169904] mdt02_001 126630 3731934.418604 27227348 120 0.000000 1548601.079313 0.000000 2 / [Wed Feb 6 09:40:51 2019][1697226.181332] osp_up2-3 126697 2227982.440076 23 120 0.000000 0.606063 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.192761] osp-pre-17-3 126737 3787881.970631 214685 120 0.000000 5783.744920 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.204182] osp-pre-7-3 126761 3787881.133590 214633 120 0.000000 4014.554089 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.215604] osp-pre-1-3 126779 3787942.375224 214591 120 0.000000 7803.646811 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.227034] osp-syn-19-1 126919 3787927.721849 198200 120 0.000000 4746.627366 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.238458] osp-pre-25-1 126924 3787882.192269 213944 120 0.000000 4823.681827 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.249895] mdt_rdpg02_003 127249 3787950.438663 6237929 120 0.000000 282818.310162 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.261320] mdt_io02_008 127265 3787758.294813 1531929 120 0.000000 70897.412977 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.272754] mdt_io02_019 129241 3787944.618097 1836783 120 0.000000 81608.075392 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.284179] mdt02_018 129264 3732661.221981 26189281 120 0.000000 1207115.241015 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.295620] kworker/u264:0 10636 3731077.524635 1044 100 0.000000 10059.779714 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.306952] ldlm_bl_12 10653 3787943.060247 1325418 120 0.000000 316021.366115 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.318300] mdt02_038 17930 3787893.073810 194156 120 0.000000 178315.230217 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.329636] mdt02_041 17997 3731903.205743 137712 120 0.000000 618333.873631 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.340968] mdt02_044 18178 3787926.892258 134429 120 0.000000 163825.785093 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.352304] mdt02_045 18223 3732587.268709 3631 120 0.000000 6563.015403 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.363643] mdt02_050 18266 3724437.089798 34830 120 0.000000 10915.727695 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.374987] kworker/18:1 23807 3604876.352389 16 120 0.000000 1239.110447 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.386324] kworker/18:2 24592 3787943.608515 9715 120 0.000000 16968.786043 0.000000 2 / [Wed Feb 6 09:40:52 2019][1697226.397672] [Wed Feb 6 09:40:52 2019][1697226.399349] cpu#19, 1996.233 MHz [Wed Feb 6 09:40:52 2019][1697226.402756] .nr_running : 0 [Wed Feb 6 09:40:52 2019][1697226.407549] .load : 0 [Wed Feb 6 09:40:52 2019][1697226.412345] .nr_switches : 136718073 [Wed Feb 6 09:40:52 2019][1697226.417830] .nr_load_updates : 75446240 [Wed Feb 6 09:40:52 2019][1697226.423229] .nr_uninterruptible : -841 [Wed Feb 6 09:40:52 2019][1697226.428281] .next_balance : 5991.879202 [Wed Feb 6 09:40:52 2019][1697226.433942] .curr->pid : 0 [Wed Feb 6 09:40:52 2019][1697226.438732] .clock : 1697226438.120122 [Wed Feb 6 09:40:52 2019][1697226.444913] .cpu_load[0] : 0 [Wed Feb 6 09:40:52 2019][1697226.449704] .cpu_load[1] : 0 [Wed Feb 6 09:40:52 2019][1697226.454498] .cpu_load[2] : 0 [Wed Feb 6 09:40:52 2019][1697226.459289] .cpu_load[3] : 0 [Wed Feb 6 09:40:52 2019][1697226.464082] .cpu_load[4] : 0 [Wed Feb 6 09:40:52 2019][1697226.468876] .avg_idle : 915077 [Wed Feb 6 09:40:52 2019][1697226.474102] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:52 2019][1697226.479415] [Wed Feb 6 09:40:52 2019][1697226.481085] cfs_rq[19]:/ [Wed Feb 6 09:40:52 2019][1697226.483797] .exec_clock : 0.000000 [Wed Feb 6 09:40:52 2019][1697226.489197] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:52 2019][1697226.494596] .min_vruntime : 4782727.125868 [Wed Feb 6 09:40:52 2019][1697226.500516] .max_vruntime : 0.000001 [Wed Feb 6 09:40:52 2019][1697226.505916] .spread : 0.000000 [Wed Feb 6 09:40:52 2019][1697226.511315] .spread0 : -4060637.813693 [Wed Feb 6 09:40:52 2019][1697226.517319] .nr_spread_over : 0 [Wed Feb 6 09:40:52 2019][1697226.522112] .nr_running : 0 [Wed Feb 6 09:40:52 2019][1697226.526907] .load : 0 [Wed Feb 6 09:40:52 2019][1697226.531697] .runnable_load_avg : 0 [Wed Feb 6 09:40:52 2019][1697226.536492] .blocked_load_avg : 5 [Wed Feb 6 09:40:52 2019][1697226.541283] .tg_load_avg : 0 [Wed Feb 6 09:40:52 2019][1697226.546076] .tg_load_contrib : 0 [Wed Feb 6 09:40:52 2019][1697226.550870] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:52 2019][1697226.555662] .tg->runnable_avg : 0 [Wed Feb 6 09:40:52 2019][1697226.560455] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:52 2019][1697226.565248] .throttled : 0 [Wed Feb 6 09:40:52 2019][1697226.570039] .throttle_count : 0 [Wed Feb 6 09:40:52 2019][1697226.574833] .avg->runnable_avg_sum : 322 [Wed Feb 6 09:40:52 2019][1697226.579799] .avg->runnable_avg_period : 46779 [Wed Feb 6 09:40:52 2019][1697226.584942] [Wed Feb 6 09:40:52 2019][1697226.586611] rt_rq[19]:/ [Wed Feb 6 09:40:52 2019][1697226.589238] .rt_nr_running : 0 [Wed Feb 6 09:40:52 2019][1697226.594030] .rt_throttled : 0 [Wed Feb 6 09:40:52 2019][1697226.598823] .rt_time : 0.000000 [Wed Feb 6 09:40:52 2019][1697226.604221] .rt_runtime : 963.992477 [Wed Feb 6 09:40:52 2019][1697226.609795] [Wed Feb 6 09:40:52 2019][1697226.611466] runnable tasks: [Wed Feb 6 09:40:52 2019][1697226.614439] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:52 2019][1697226.625385] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:52 2019][1697226.636259] watchdog/19 106 -11.979712 424306 0 0.000000 48273.554695 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.647591] migration/19 107 0.000000 1739 0 0.000000 126898.682214 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.658926] ksoftirqd/19 108 4782618.785532 807297 120 0.000000 9197.823847 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.670263] kworker/19:0H 110 3136.201410 10 100 0.000000 0.038342 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.681614] khungtaskd 257 4782536.315340 14150 120 0.000000 2644.546901 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.692957] kworker/19:1 7271 4782713.805966 149699 120 0.000000 10865.120945 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.704289] kworker/u261:0 7436 1729.350850 4 100 0.000000 0.016761 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.715639] auditd 16727 4782547.561112 242208 116 0.000000 7574.664215 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.726980] smartd 16782 4584179.322487 10545 120 0.000000 1426.550890 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.738318] kworker/19:1H 17852 4782668.527299 1040694 100 0.000000 5337.077218 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.749651] collectd 18219 4782634.171868 172984 120 0.000000 7024.072840 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.760987] in:imjournal 18241 4782727.125868 1998455 120 0.000000 35609.429040 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.772335] ptlrpc_hr03_003 126402 4782714.032824 1678753 120 0.000000 14280.994389 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.783755] ptlrpc_hr03_004 126403 4782672.083347 1678748 120 0.000000 14450.418986 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.795179] kiblnd_sd_03_00 126433 4782728.950655 87878571 120 0.000000 656638.397310 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.806608] ptlrpcd_03_02 126479 4782712.282650 5636777 120 0.000000 87695.568311 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.818032] ptlrpcd_03_07 126484 4782712.631743 5610617 120 0.000000 89382.822723 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.829454] ptlrpcd_03_09 126486 4782716.716452 5603057 120 0.000000 89405.208812 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.840876] ptlrpcd_03_10 126487 4782717.162105 5607191 120 0.000000 87346.584113 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.852310] mdt_io03_002 126676 4782714.560480 171322 120 0.000000 6309.550590 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.863734] osp-pre-25-3 126735 4782665.109282 214630 120 0.000000 3402.917276 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.875156] osp-syn-32-3 126744 4782624.629410 126631 120 0.000000 3534.812283 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.886578] osp-syn-23-3 126754 4782675.173146 126602 120 0.000000 3589.567951 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.898001] osp-syn-37-3 126786 4782601.169354 127611 120 0.000000 3964.583773 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.909424] osp-pre-35-3 126793 4782665.108270 214672 120 0.000000 8496.379002 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.920845] osp-pre-30-3 126797 4782668.373659 214701 120 0.000000 6716.061200 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.932268] osp-pre-34-3 126799 4782668.536946 214632 120 0.000000 5164.881335 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.943693] osp-pre-10-1 126902 4782665.106436 213962 120 0.000000 7551.282573 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.955116] osp-pre-21-1 126936 4782665.107037 213947 120 0.000000 5768.609454 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.966582] mdt_io03_004 40442 4782701.205237 149944 120 0.000000 8949.414851 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.977915] mdt_io03_009 41381 4782406.968929 91929 120 0.000000 5202.960348 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697226.989253] ldlm_cn03_006 51608 4782678.826142 2362993 120 0.000000 146940.968096 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697227.000595] ldlm_cn03_010 12925 4782713.093233 1935806 120 0.000000 169978.093092 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697227.011935] ldlm_cn03_016 14200 4782684.316212 322045 120 0.000000 99194.306480 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697227.023269] mdt03_014 17804 4782658.343703 65111 120 0.000000 4837.574399 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697227.034606] mdt03_016 17806 4782720.406007 160748 120 0.000000 8179.662043 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697227.045941] mdt03_017 17807 4707531.890869 71978 120 0.000000 6565.160388 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697227.057284] mdt03_030 17905 4782670.356535 77699 120 0.000000 5368.434574 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697227.068624] mdt03_033 17934 4707531.803235 26544 120 0.000000 154872.099141 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697227.079960] mdt03_034 17957 4707531.803174 7869 120 0.000000 1587.028401 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697227.091300] mdt03_049 18268 4782734.546865 126137 120 0.000000 314300.209588 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697227.102631] mdt03_051 18277 4782388.253394 143245 120 0.000000 162801.853829 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697227.113980] ldlm_cn03_023 24586 4782680.442737 11829 120 0.000000 70671.506203 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697227.125318] kworker/19:2 24987 4745716.881761 8 120 0.000000 0.044997 0.000000 3 / [Wed Feb 6 09:40:52 2019][1697227.136658] [Wed Feb 6 09:40:52 2019][1697227.138333] cpu#20, 1996.233 MHz [Wed Feb 6 09:40:52 2019][1697227.141753] .nr_running : 0 [Wed Feb 6 09:40:52 2019][1697227.146550] .load : 0 [Wed Feb 6 09:40:52 2019][1697227.151343] .nr_switches : 1770932723 [Wed Feb 6 09:40:52 2019][1697227.156912] .nr_load_updates : 333606824 [Wed Feb 6 09:40:52 2019][1697227.162399] .nr_uninterruptible : 135210 [Wed Feb 6 09:40:52 2019][1697227.167626] .next_balance : 5991.879942 [Wed Feb 6 09:40:52 2019][1697227.173284] .curr->pid : 0 [Wed Feb 6 09:40:52 2019][1697227.178080] .clock : 1697227178.042796 [Wed Feb 6 09:40:52 2019][1697227.184257] .cpu_load[0] : 0 [Wed Feb 6 09:40:52 2019][1697227.189050] .cpu_load[1] : 0 [Wed Feb 6 09:40:53 2019][1697227.193843] .cpu_load[2] : 0 [Wed Feb 6 09:40:53 2019][1697227.198636] .cpu_load[3] : 0 [Wed Feb 6 09:40:53 2019][1697227.203430] .cpu_load[4] : 0 [Wed Feb 6 09:40:53 2019][1697227.208222] .avg_idle : 441908 [Wed Feb 6 09:40:53 2019][1697227.213448] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:53 2019][1697227.218759] [Wed Feb 6 09:40:53 2019][1697227.220429] cfs_rq[20]:/ [Wed Feb 6 09:40:53 2019][1697227.223144] .exec_clock : 0.000000 [Wed Feb 6 09:40:53 2019][1697227.228542] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:53 2019][1697227.233942] .min_vruntime : 10320032.794928 [Wed Feb 6 09:40:53 2019][1697227.239947] .max_vruntime : 0.000001 [Wed Feb 6 09:40:53 2019][1697227.245347] .spread : 0.000000 [Wed Feb 6 09:40:53 2019][1697227.250747] .spread0 : 1476667.855367 [Wed Feb 6 09:40:53 2019][1697227.256666] .nr_spread_over : 0 [Wed Feb 6 09:40:53 2019][1697227.261457] .nr_running : 0 [Wed Feb 6 09:40:53 2019][1697227.266250] .load : 0 [Wed Feb 6 09:40:53 2019][1697227.271044] .runnable_load_avg : 0 [Wed Feb 6 09:40:53 2019][1697227.275838] .blocked_load_avg : 21 [Wed Feb 6 09:40:53 2019][1697227.280716] .tg_load_avg : 0 [Wed Feb 6 09:40:53 2019][1697227.285509] .tg_load_contrib : 0 [Wed Feb 6 09:40:53 2019][1697227.290302] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:53 2019][1697227.295095] .tg->runnable_avg : 0 [Wed Feb 6 09:40:53 2019][1697227.299886] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:53 2019][1697227.304679] .throttled : 0 [Wed Feb 6 09:40:53 2019][1697227.309472] .throttle_count : 0 [Wed Feb 6 09:40:53 2019][1697227.314266] .avg->runnable_avg_sum : 694 [Wed Feb 6 09:40:53 2019][1697227.319230] .avg->runnable_avg_period : 46497 [Wed Feb 6 09:40:53 2019][1697227.324374] [Wed Feb 6 09:40:53 2019][1697227.326044] rt_rq[20]:/ [Wed Feb 6 09:40:53 2019][1697227.328669] .rt_nr_running : 0 [Wed Feb 6 09:40:53 2019][1697227.333462] .rt_throttled : 0 [Wed Feb 6 09:40:53 2019][1697227.338256] .rt_time : 0.007033 [Wed Feb 6 09:40:53 2019][1697227.343655] .rt_runtime : 958.767362 [Wed Feb 6 09:40:53 2019][1697227.349226] [Wed Feb 6 09:40:53 2019][1697227.350898] runnable tasks: [Wed Feb 6 09:40:53 2019][1697227.353872] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:53 2019][1697227.364819] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:53 2019][1697227.375689] watchdog/20 111 -11.981906 424291 0 0.000000 63997.798651 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.387021] migration/20 112 0.000000 1676 0 0.000000 258027.186654 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.398358] ksoftirqd/20 113 10319899.696039 1817028 120 0.000000 24879.269309 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.409693] kworker/20:0H 115 3380.393944 10 100 0.000000 0.040476 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.421057] systemd-udevd 7984 156970.484314 2960 120 0.000000 748.060940 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.432402] kdmwork-253:2 16480 10320024.152290 33969041 120 0.000000 404395.041374 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.443744] kworker/20:1H 18110 10320020.149630 2006151 100 0.000000 11329.868428 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.455083] ptlrpc_hr00_001 126380 10320018.468484 15287031 120 0.000000 125301.798707 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.466505] ptlrpc_hr00_004 126383 10320016.224564 15287066 120 0.000000 125513.126628 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.477934] ptlrpcd_00_04 126445 10319987.777368 4268357 120 0.000000 73608.339857 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.489350] ptlrpcd_00_08 126449 10319985.624678 4295721 120 0.000000 70892.648766 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.500774] ptlrpcd_00_11 126452 10319999.944756 4270060 120 0.000000 66717.243572 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.512209] mdt00_001 126624 10116715.552966 75694748 120 0.000000 2076956.611391 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.523628] mdt_rdpg00_001 126636 10319776.034930 4331640 120 0.000000 228371.765640 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.535060] osp-pre-45-3 126765 10319915.042103 214714 120 0.000000 6220.347326 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.546489] osp-syn-10-1 126901 10320024.147421 198565 120 0.000000 4889.991713 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.557915] osp-syn-11-1 126903 10319985.603178 197822 120 0.000000 4798.213693 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.569341] osp-syn-43-1 126917 10320024.154954 197267 120 0.000000 5485.259872 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.580760] osp-syn-4-1 126941 10320024.166918 198355 120 0.000000 4838.579233 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.592185] osp-syn-45-1 126953 10319985.601946 197718 120 0.000000 4733.599631 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.603613] mdt00_007 127214 10320036.459177 81956724 120 0.000000 2339066.863623 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.615043] mdt_io00_011 127365 10320024.462763 2406851 120 0.000000 100657.203188 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.626475] mdt_io00_017 129238 10090053.046056 1341514 120 0.000000 52692.456122 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.637906] mdt00_024 129794 10116715.550411 77582480 120 0.000000 2091401.249293 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.649336] ldlm_cn00_019 130327 10320029.876303 3026722 120 0.000000 200095.643898 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.660767] kworker/20:1 125816 10320020.802523 52482 120 0.000000 12353.441698 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.672197] mdt00_027 17860 10117179.545415 94425 120 0.000000 465259.928254 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.683544] mdt00_056 18907 10116715.551151 31684 120 0.000000 8556.736087 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.694890] kworker/20:0 24961 10134871.329019 9 120 0.000000 0.378741 0.000000 0 / [Wed Feb 6 09:40:53 2019][1697227.706238] [Wed Feb 6 09:40:53 2019][1697227.707912] cpu#21, 1996.233 MHz [Wed Feb 6 09:40:53 2019][1697227.711320] .nr_running : 0 [Wed Feb 6 09:40:53 2019][1697227.716111] .load : 0 [Wed Feb 6 09:40:53 2019][1697227.720905] .nr_switches : 946477770 [Wed Feb 6 09:40:53 2019][1697227.726392] .nr_load_updates : 172282108 [Wed Feb 6 09:40:53 2019][1697227.731875] .nr_uninterruptible : -17929 [Wed Feb 6 09:40:53 2019][1697227.737103] .next_balance : 5991.880511 [Wed Feb 6 09:40:53 2019][1697227.742762] .curr->pid : 0 [Wed Feb 6 09:40:53 2019][1697227.747556] .clock : 1697227747.466386 [Wed Feb 6 09:40:53 2019][1697227.753733] .cpu_load[0] : 0 [Wed Feb 6 09:40:53 2019][1697227.758527] .cpu_load[1] : 0 [Wed Feb 6 09:40:53 2019][1697227.763320] .cpu_load[2] : 0 [Wed Feb 6 09:40:53 2019][1697227.768114] .cpu_load[3] : 0 [Wed Feb 6 09:40:53 2019][1697227.772905] .cpu_load[4] : 0 [Wed Feb 6 09:40:53 2019][1697227.777698] .avg_idle : 964182 [Wed Feb 6 09:40:53 2019][1697227.782924] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:53 2019][1697227.788236] [Wed Feb 6 09:40:53 2019][1697227.789907] cfs_rq[21]:/ [Wed Feb 6 09:40:53 2019][1697227.792618] .exec_clock : 0.000000 [Wed Feb 6 09:40:53 2019][1697227.798018] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:53 2019][1697227.803419] .min_vruntime : 6672065.750484 [Wed Feb 6 09:40:53 2019][1697227.809337] .max_vruntime : 0.000001 [Wed Feb 6 09:40:53 2019][1697227.814737] .spread : 0.000000 [Wed Feb 6 09:40:53 2019][1697227.820136] .spread0 : -2171299.189077 [Wed Feb 6 09:40:53 2019][1697227.826142] .nr_spread_over : 0 [Wed Feb 6 09:40:53 2019][1697227.830935] .nr_running : 0 [Wed Feb 6 09:40:53 2019][1697227.835728] .load : 0 [Wed Feb 6 09:40:53 2019][1697227.840521] .runnable_load_avg : 0 [Wed Feb 6 09:40:53 2019][1697227.845312] .blocked_load_avg : 12 [Wed Feb 6 09:40:53 2019][1697227.850192] .tg_load_avg : 0 [Wed Feb 6 09:40:53 2019][1697227.854986] .tg_load_contrib : 0 [Wed Feb 6 09:40:53 2019][1697227.859779] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:53 2019][1697227.864572] .tg->runnable_avg : 0 [Wed Feb 6 09:40:53 2019][1697227.869362] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:53 2019][1697227.874157] .throttled : 0 [Wed Feb 6 09:40:53 2019][1697227.878949] .throttle_count : 0 [Wed Feb 6 09:40:53 2019][1697227.883742] .avg->runnable_avg_sum : 432 [Wed Feb 6 09:40:53 2019][1697227.888707] .avg->runnable_avg_period : 46779 [Wed Feb 6 09:40:53 2019][1697227.893851] [Wed Feb 6 09:40:53 2019][1697227.895519] rt_rq[21]:/ [Wed Feb 6 09:40:53 2019][1697227.898144] .rt_nr_running : 0 [Wed Feb 6 09:40:53 2019][1697227.902940] .rt_throttled : 0 [Wed Feb 6 09:40:53 2019][1697227.907731] .rt_time : 0.000000 [Wed Feb 6 09:40:53 2019][1697227.913131] .rt_runtime : 958.767362 [Wed Feb 6 09:40:53 2019][1697227.918703] [Wed Feb 6 09:40:53 2019][1697227.920375] runnable tasks: [Wed Feb 6 09:40:53 2019][1697227.923347] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:53 2019][1697227.934295] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:53 2019][1697227.945168] watchdog/21 116 -11.979572 424282 0 0.000000 105219.399667 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697227.956499] migration/21 117 0.000000 1762 0 0.000000 245140.817893 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697227.967834] ksoftirqd/21 118 6635437.737502 4116188 120 0.000000 49300.798656 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697227.979170] kworker/21:0H 120 2911.048610 9 100 0.000000 0.030069 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697227.990544] kworker/21:1H 16411 6635437.669043 3136382 100 0.000000 13798.095959 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697228.001882] systemd-logind 16849 6635443.959373 1769995 120 0.000000 57833.564368 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697228.013232] kiblnd_sd_01_01 126426 6672067.715955 816306230 120 0.000000 5804907.422974 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697228.024660] ptlrpcd_01_04 126457 6672053.699519 4022299 120 0.000000 63480.394023 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697228.036085] ldlm_cn01_001 126599 6671957.626825 10109580 120 0.000000 402557.239935 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697228.047515] mdt_out01_001 126654 6436658.190036 156 120 0.000000 1.069828 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697228.058958] osp-pre-33-1 126946 6672041.837320 213956 120 0.000000 9817.561826 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697228.070384] mdt01_003 127066 6549864.670630 37301784 120 0.000000 1517299.185537 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697228.081804] mdt01_004 127131 6548028.984041 36260889 120 0.000000 1459952.120356 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697228.093248] mdt01_023 129253 6546559.415773 33739418 120 0.000000 1383252.867080 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697228.104675] mdt_io01_030 130108 4350411.004893 3505679 120 0.000000 141319.095428 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697228.116111] kworker/21:2 86721 6672044.151545 49711 120 0.000000 13821.843469 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697228.127463] mdt01_043 17972 6546559.416825 234659 120 0.000000 471123.327706 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697228.138797] mdt01_045 17974 6548028.991064 20139 120 0.000000 10894.128713 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697228.150136] mdt01_053 18044 6549140.386077 129665 120 0.000000 13847.823931 0.000000 1 / [Wed Feb 6 09:40:53 2019][1697228.161472] mdt01_055 19060 6672023.379815 302233 120 0.000000 472761.497257 0.000000 1 / [Wed Feb 6 09:40:54 2019][1697228.172805] mdt01_057 19086 6548028.982217 33134 120 0.000000 14577.519640 0.000000 1 / [Wed Feb 6 09:40:54 2019][1697228.184144] ldlm_cn01_019 23652 6672052.616449 45058 120 0.000000 110747.883362 0.000000 1 / [Wed Feb 6 09:40:54 2019][1697228.195483] kworker/21:0 24099 6311980.035530 8 120 0.000000 0.039655 0.000000 1 / [Wed Feb 6 09:40:54 2019][1697228.206861] [Wed Feb 6 09:40:54 2019][1697228.208532] cpu#22, 1996.233 MHz [Wed Feb 6 09:40:54 2019][1697228.211938] .nr_running : 0 [Wed Feb 6 09:40:54 2019][1697228.216733] .load : 0 [Wed Feb 6 09:40:54 2019][1697228.221533] .nr_switches : 671993977 [Wed Feb 6 09:40:54 2019][1697228.227018] .nr_load_updates : 174883371 [Wed Feb 6 09:40:54 2019][1697228.232506] .nr_uninterruptible : -13086 [Wed Feb 6 09:40:54 2019][1697228.237732] .next_balance : 5991.881009 [Wed Feb 6 09:40:54 2019][1697228.243390] .curr->pid : 0 [Wed Feb 6 09:40:54 2019][1697228.248183] .clock : 1697228247.837194 [Wed Feb 6 09:40:54 2019][1697228.254362] .cpu_load[0] : 0 [Wed Feb 6 09:40:54 2019][1697228.259155] .cpu_load[1] : 0 [Wed Feb 6 09:40:54 2019][1697228.263949] .cpu_load[2] : 0 [Wed Feb 6 09:40:54 2019][1697228.268740] .cpu_load[3] : 0 [Wed Feb 6 09:40:54 2019][1697228.273535] .cpu_load[4] : 0 [Wed Feb 6 09:40:54 2019][1697228.278326] .avg_idle : 10000000 [Wed Feb 6 09:40:54 2019][1697228.283724] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:54 2019][1697228.289037] [Wed Feb 6 09:40:54 2019][1697228.290707] cfs_rq[22]:/ [Wed Feb 6 09:40:54 2019][1697228.293421] .exec_clock : 0.000000 [Wed Feb 6 09:40:54 2019][1697228.298821] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:54 2019][1697228.304219] .min_vruntime : 5771244.964536 [Wed Feb 6 09:40:54 2019][1697228.310138] .max_vruntime : 0.000001 [Wed Feb 6 09:40:54 2019][1697228.315537] .spread : 0.000000 [Wed Feb 6 09:40:54 2019][1697228.320939] .spread0 : -3072119.975025 [Wed Feb 6 09:40:54 2019][1697228.326944] .nr_spread_over : 0 [Wed Feb 6 09:40:54 2019][1697228.331736] .nr_running : 0 [Wed Feb 6 09:40:54 2019][1697228.336528] .load : 0 [Wed Feb 6 09:40:54 2019][1697228.341322] .runnable_load_avg : 0 [Wed Feb 6 09:40:54 2019][1697228.346115] .blocked_load_avg : 0 [Wed Feb 6 09:40:54 2019][1697228.350906] .tg_load_avg : 0 [Wed Feb 6 09:40:54 2019][1697228.355699] .tg_load_contrib : 0 [Wed Feb 6 09:40:54 2019][1697228.360493] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:54 2019][1697228.365284] .tg->runnable_avg : 0 [Wed Feb 6 09:40:54 2019][1697228.370078] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:54 2019][1697228.374869] .throttled : 0 [Wed Feb 6 09:40:54 2019][1697228.379662] .throttle_count : 0 [Wed Feb 6 09:40:54 2019][1697228.384456] .avg->runnable_avg_sum : 55 [Wed Feb 6 09:40:54 2019][1697228.389335] .avg->runnable_avg_period : 47875 [Wed Feb 6 09:40:54 2019][1697228.394479] [Wed Feb 6 09:40:54 2019][1697228.396149] rt_rq[22]:/ [Wed Feb 6 09:40:54 2019][1697228.398773] .rt_nr_running : 0 [Wed Feb 6 09:40:54 2019][1697228.403566] .rt_throttled : 0 [Wed Feb 6 09:40:54 2019][1697228.408359] .rt_time : 0.000000 [Wed Feb 6 09:40:54 2019][1697228.413760] .rt_runtime : 958.767362 [Wed Feb 6 09:40:54 2019][1697228.419332] [Wed Feb 6 09:40:54 2019][1697228.421003] runnable tasks: [Wed Feb 6 09:40:54 2019][1697228.423977] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:54 2019][1697228.434923] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:54 2019][1697228.445797] watchdog/22 121 -11.979942 424300 0 0.000000 54722.689291 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.457135] migration/22 122 0.000000 1575 0 0.000000 226308.881851 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.468470] ksoftirqd/22 123 5771148.853074 3421313 120 0.000000 38564.538838 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.479807] kworker/22:0H 125 3293.542825 9 100 0.000000 0.038441 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.491177] kworker/22:1H 16193 5771232.591220 2516670 100 0.000000 11131.758965 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.502522] writer#3 18229 5771233.523582 25041809 120 0.000000 228796.378631 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.513871] ptlrpc_hr02_001 126393 5771232.795427 15299430 120 0.000000 123833.089106 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.525292] ptlrpc_hr02_005 126397 5771232.796018 15299409 120 0.000000 121657.956076 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.536724] ptlrpcd_02_00 126465 5771232.818379 6845044 120 0.000000 84922.183921 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.548151] ldlm_cb02_001 126591 2737988.420676 10943554 120 0.000000 157921.875926 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.559589] osp-syn-7-3 126760 5771232.632492 126349 120 0.000000 5780.731693 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.571010] osp-pre-24-3 126769 5771232.803532 214686 120 0.000000 2831.088964 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.582433] osp-pre-8-3 126791 5771232.671794 214660 120 0.000000 5869.073436 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.593859] osp-pre-2-1 126916 5771232.646046 213951 120 0.000000 7496.095762 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.605276] osp-pre-25-1 126924 5771232.803943 213945 120 0.000000 4823.697767 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.616712] mdt_io02_006 127233 5519339.483574 2302532 120 0.000000 100856.760640 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.628132] mdt02_008 127248 5771245.963300 26223065 120 0.000000 1170723.887234 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.639555] mdt_io02_009 127319 5520263.780557 2137565 120 0.000000 98093.956070 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.650995] mdt_io02_023 130117 5500106.228805 1867175 120 0.000000 92655.030783 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.662426] kworker/22:2 20891 5771232.947303 124778 120 0.000000 10825.033193 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.673788] mdt02_048 18262 5771156.273006 166129 120 0.000000 617368.945535 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.685125] mdt02_055 18335 5564924.174183 209604 120 0.000000 921706.077390 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.696468] ldlm_cn02_027 23741 5771245.490277 19388 120 0.000000 70629.141698 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.707811] kworker/22:0 24913 5514279.935051 4 120 0.000000 0.031128 0.000000 2 / [Wed Feb 6 09:40:54 2019][1697228.719192] [Wed Feb 6 09:40:54 2019][1697228.720869] cpu#23, 1996.233 MHz [Wed Feb 6 09:40:54 2019][1697228.724275] .nr_running : 0 [Wed Feb 6 09:40:54 2019][1697228.729069] .load : 0 [Wed Feb 6 09:40:54 2019][1697228.733861] .nr_switches : 144788098 [Wed Feb 6 09:40:54 2019][1697228.739348] .nr_load_updates : 79715337 [Wed Feb 6 09:40:54 2019][1697228.744747] .nr_uninterruptible : -1130 [Wed Feb 6 09:40:54 2019][1697228.749886] .next_balance : 5991.881524 [Wed Feb 6 09:40:54 2019][1697228.755545] .curr->pid : 0 [Wed Feb 6 09:40:54 2019][1697228.760338] .clock : 1697228759.924370 [Wed Feb 6 09:40:54 2019][1697228.766519] .cpu_load[0] : 0 [Wed Feb 6 09:40:54 2019][1697228.771308] .cpu_load[1] : 0 [Wed Feb 6 09:40:54 2019][1697228.776104] .cpu_load[2] : 0 [Wed Feb 6 09:40:54 2019][1697228.780896] .cpu_load[3] : 0 [Wed Feb 6 09:40:54 2019][1697228.785687] .cpu_load[4] : 0 [Wed Feb 6 09:40:54 2019][1697228.790482] .avg_idle : 6485540 [Wed Feb 6 09:40:54 2019][1697228.795793] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:54 2019][1697228.801106] [Wed Feb 6 09:40:54 2019][1697228.802776] cfs_rq[23]:/ [Wed Feb 6 09:40:54 2019][1697228.805488] .exec_clock : 0.000000 [Wed Feb 6 09:40:54 2019][1697228.810887] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:54 2019][1697228.816288] .min_vruntime : 1858720.732583 [Wed Feb 6 09:40:54 2019][1697228.822207] .max_vruntime : 0.000001 [Wed Feb 6 09:40:54 2019][1697228.827607] .spread : 0.000000 [Wed Feb 6 09:40:54 2019][1697228.833004] .spread0 : -6984644.206978 [Wed Feb 6 09:40:54 2019][1697228.839010] .nr_spread_over : 0 [Wed Feb 6 09:40:54 2019][1697228.843804] .nr_running : 0 [Wed Feb 6 09:40:54 2019][1697228.848596] .load : 0 [Wed Feb 6 09:40:54 2019][1697228.853389] .runnable_load_avg : 0 [Wed Feb 6 09:40:54 2019][1697228.858182] .blocked_load_avg : 0 [Wed Feb 6 09:40:54 2019][1697228.862976] .tg_load_avg : 0 [Wed Feb 6 09:40:54 2019][1697228.867768] .tg_load_contrib : 0 [Wed Feb 6 09:40:54 2019][1697228.872560] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:54 2019][1697228.877352] .tg->runnable_avg : 0 [Wed Feb 6 09:40:54 2019][1697228.882144] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:54 2019][1697228.886938] .throttled : 0 [Wed Feb 6 09:40:54 2019][1697228.891732] .throttle_count : 0 [Wed Feb 6 09:40:54 2019][1697228.896523] .avg->runnable_avg_sum : 8 [Wed Feb 6 09:40:54 2019][1697228.901316] .avg->runnable_avg_period : 46761 [Wed Feb 6 09:40:54 2019][1697228.906458] [Wed Feb 6 09:40:54 2019][1697228.908128] rt_rq[23]:/ [Wed Feb 6 09:40:54 2019][1697228.910755] .rt_nr_running : 0 [Wed Feb 6 09:40:54 2019][1697228.915548] .rt_throttled : 0 [Wed Feb 6 09:40:54 2019][1697228.920339] .rt_time : 0.000000 [Wed Feb 6 09:40:54 2019][1697228.925740] .rt_runtime : 979.166667 [Wed Feb 6 09:40:54 2019][1697228.931313] [Wed Feb 6 09:40:54 2019][1697228.932984] runnable tasks: [Wed Feb 6 09:40:54 2019][1697228.935957] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:54 2019][1697228.946903] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:54 2019][1697228.957777] watchdog/23 126 -11.980143 424305 0 0.000000 56872.284633 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697228.969109] migration/23 127 0.000000 1259 0 0.000000 128099.623514 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697228.980445] ksoftirqd/23 128 1858708.753132 702890 120 0.000000 9597.151605 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697228.991779] kworker/23:0H 130 28523.266939 10 100 0.000000 0.040456 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.003133] kworker/23:1 805 1858605.890622 136725 120 0.000000 10244.213505 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.014479] ccp-1-q3 9979 1872.817364 2 120 0.000000 0.012273 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.025824] rpcbind 16809 1858437.437693 56517 120 0.000000 2208.313936 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.037165] reader#1 18232 1858707.413884 186263 120 0.000000 15668.210217 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.048500] kworker/23:1H 43868 1858708.732743 1120649 100 0.000000 4110.441252 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.059835] ptlrpc_hr03_001 126400 1858708.738093 1678756 120 0.000000 14450.500500 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.071260] monitor_thread 126437 1858708.740718 1029267 120 0.000000 91872.634408 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.082685] jbd2/dm-1-8 126575 1858711.446094 8394636 120 0.000000 653985.724391 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.094113] ldlm_cn03_001 126605 1858720.530889 2414113 120 0.000000 122444.402279 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.105536] mdt_out03_001 126658 1858707.009052 615 120 0.000000 1208.865268 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.116958] osp-pre-9-3 126725 1858471.259092 214614 120 0.000000 4295.137888 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.128383] osp-pre-35-3 126793 1858707.440405 214673 120 0.000000 8496.406895 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.139808] osp-syn-41-1 126939 1858708.760205 197223 120 0.000000 4644.568748 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.151243] mdt_rdpg03_002 127329 1858452.961002 1581063 120 0.000000 60197.139528 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.162688] mdt_seqm_0002 2903 1858363.715770 1246 120 0.000000 28.282547 0.000000 3 / [Wed Feb 6 09:40:54 2019][1697229.174029] mdt_io03_003 40441 1858148.162177 196898 120 0.000000 10471.603045 0.000000 3 / [Wed Feb 6 09:40:55 2019][1697229.185370] mdt03_018 17810 1858605.884681 216993 120 0.000000 12679.543028 0.000000 3 / [Wed Feb 6 09:40:55 2019][1697229.196717] mdt03_049 18268 1858720.904236 127360 120 0.000000 314322.251311 0.000000 3 / [Wed Feb 6 09:40:55 2019][1697229.208061] pickup 24156 1858415.615743 66 120 0.000000 1320.620337 0.000000 3 / [Wed Feb 6 09:40:55 2019][1697229.219395] ldlm_cn03_017 24531 1858653.962790 8772 120 0.000000 60909.757935 0.000000 3 / [Wed Feb 6 09:40:55 2019][1697229.230733] ldlm_cn03_028 24610 1858330.011892 4322 120 0.000000 61236.644925 0.000000 3 / [Wed Feb 6 09:40:55 2019][1697229.242067] kworker/u260:2 24624 1858617.748338 339 120 0.000000 1462.121918 0.000000 3 / [Wed Feb 6 09:40:55 2019][1697229.253406] kworker/23:0 24710 1738730.439459 2 120 0.000000 0.009969 0.000000 3 / [Wed Feb 6 09:40:55 2019][1697229.264780] [Wed Feb 6 09:40:55 2019][1697229.266451] cpu#24, 1996.233 MHz [Wed Feb 6 09:40:55 2019][1697229.269860] .nr_running : 5 [Wed Feb 6 09:40:55 2019][1697229.274659] .load : 3072 [Wed Feb 6 09:40:55 2019][1697229.279712] .nr_switches : 1082960721 [Wed Feb 6 09:40:55 2019][1697229.285284] .nr_load_updates : 213748740 [Wed Feb 6 09:40:55 2019][1697229.290769] .nr_uninterruptible : -104576 [Wed Feb 6 09:40:55 2019][1697229.296085] .next_balance : 5991.882163 [Wed Feb 6 09:40:55 2019][1697229.301751] .curr->pid : 25140 [Wed Feb 6 09:40:55 2019][1697229.306891] .clock : 1697113243.916509 [Wed Feb 6 09:40:55 2019][1697229.313071] .cpu_load[0] : 3072 [Wed Feb 6 09:40:55 2019][1697229.318124] .cpu_load[1] : 3072 [Wed Feb 6 09:40:55 2019][1697229.323174] .cpu_load[2] : 3072 [Wed Feb 6 09:40:55 2019][1697229.328229] .cpu_load[3] : 3072 [Wed Feb 6 09:40:55 2019][1697229.333280] .cpu_load[4] : 3072 [Wed Feb 6 09:40:55 2019][1697229.338348] .avg_idle : 1256745 [Wed Feb 6 09:40:55 2019][1697229.343665] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:55 2019][1697229.348977] [Wed Feb 6 09:40:55 2019][1697229.350645] cfs_rq[24]:/ [Wed Feb 6 09:40:55 2019][1697229.353359] .exec_clock : 0.000000 [Wed Feb 6 09:40:55 2019][1697229.358759] .MIN_vruntime : 7406502.325776 [Wed Feb 6 09:40:55 2019][1697229.364678] .min_vruntime : 7406514.325776 [Wed Feb 6 09:40:55 2019][1697229.370598] .max_vruntime : 7406502.325776 [Wed Feb 6 09:40:55 2019][1697229.376516] .spread : 0.000000 [Wed Feb 6 09:40:55 2019][1697229.381915] .spread0 : -1436850.613785 [Wed Feb 6 09:40:55 2019][1697229.387923] .nr_spread_over : 0 [Wed Feb 6 09:40:55 2019][1697229.392713] .nr_running : 3 [Wed Feb 6 09:40:55 2019][1697229.397507] .load : 3072 [Wed Feb 6 09:40:55 2019][1697229.402560] .runnable_load_avg : 109 [Wed Feb 6 09:40:55 2019][1697229.407525] .blocked_load_avg : 0 [Wed Feb 6 09:40:55 2019][1697229.412319] .tg_load_avg : 0 [Wed Feb 6 09:40:55 2019][1697229.417113] .tg_load_contrib : 0 [Wed Feb 6 09:40:55 2019][1697229.421905] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:55 2019][1697229.426696] .tg->runnable_avg : 0 [Wed Feb 6 09:40:55 2019][1697229.431490] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:55 2019][1697229.436282] .throttled : 0 [Wed Feb 6 09:40:55 2019][1697229.441074] .throttle_count : 0 [Wed Feb 6 09:40:55 2019][1697229.445868] .avg->runnable_avg_sum : 5406 [Wed Feb 6 09:40:55 2019][1697229.450920] .avg->runnable_avg_period : 47122 [Wed Feb 6 09:40:55 2019][1697229.456064] [Wed Feb 6 09:40:55 2019][1697229.457732] rt_rq[24]:/ [Wed Feb 6 09:40:55 2019][1697229.460357] .rt_nr_running : 1 [Wed Feb 6 09:40:55 2019][1697229.465151] .rt_throttled : 0 [Wed Feb 6 09:40:55 2019][1697229.469944] .rt_time : 0.000000 [Wed Feb 6 09:40:55 2019][1697229.475344] .rt_runtime : 962.046008 [Wed Feb 6 09:40:55 2019][1697229.480916] [Wed Feb 6 09:40:55 2019][1697229.482589] runnable tasks: [Wed Feb 6 09:40:55 2019][1697229.485560] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:55 2019][1697229.496509] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:55 2019][1697229.507383] watchdog/24 131 -11.980744 424271 0 0.000000 65184.322223 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.518719] migration/24 132 0.000000 2981 0 0.000000 175023.540377 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.530055] ksoftirqd/24 133 7406502.325776 2105829 120 0.000000 24488.463263 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.541394] kworker/24:0H 135 4198.955290 10 100 0.000000 0.046296 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.552770] kworker/24:1H 39600 7403960.988766 1862637 100 0.000000 9929.240712 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.564136] mdt_io00_000 126665 4534356.230037 880506 120 0.000000 30894.648610 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.575577] dist_txn-1 127049 6992498.551092 154 120 0.000000 6.997948 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.587001] mdt_rdpg00_002 127213 7171116.588956 1508073 120 0.000000 65447.873837 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.598433] mdt00_013 128092 7213900.185073 85727887 120 0.000000 2257152.299743 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.609867] mdt_io00_023 130240 7183198.031764 1076098 120 0.000000 43808.275176 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.621306] kworker/24:0 108674 7406502.325776 42231 120 0.000000 15430.952383 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.632724] kworker/24:2 123849 5345172.270971 2 120 0.000000 0.010510 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.644150] mdt_fld_0002 11286 6803283.879881 112 120 0.000000 1.123266 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.655489] mdt00_028 17870 7214005.879410 17042 120 0.000000 121585.566262 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.666835] mdt00_055 18001 7213900.186756 245185 120 0.000000 318654.647300 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.678193] R bash 25140 7406507.977140 62 120 0.000000 17.796114 0.000000 0 / [Wed Feb 6 09:40:55 2019][1697229.689532] [Wed Feb 6 09:40:55 2019][1697229.691209] cpu#25, 1996.233 MHz [Wed Feb 6 09:40:55 2019][1697229.694618] .nr_running : 0 [Wed Feb 6 09:40:55 2019][1697229.699408] .load : 0 [Wed Feb 6 09:40:55 2019][1697229.704200] .nr_switches : 702825278 [Wed Feb 6 09:40:55 2019][1697229.709688] .nr_load_updates : 133609237 [Wed Feb 6 09:40:55 2019][1697229.715176] .nr_uninterruptible : 43095 [Wed Feb 6 09:40:55 2019][1697229.720324] .next_balance : 5991.882493 [Wed Feb 6 09:40:55 2019][1697229.725982] .curr->pid : 0 [Wed Feb 6 09:40:55 2019][1697229.730777] .clock : 1697229729.853702 [Wed Feb 6 09:40:55 2019][1697229.736978] .cpu_load[0] : 0 [Wed Feb 6 09:40:55 2019][1697229.741772] .cpu_load[1] : 0 [Wed Feb 6 09:40:55 2019][1697229.746567] .cpu_load[2] : 0 [Wed Feb 6 09:40:55 2019][1697229.751368] .cpu_load[3] : 0 [Wed Feb 6 09:40:55 2019][1697229.756180] .cpu_load[4] : 0 [Wed Feb 6 09:40:55 2019][1697229.760980] .avg_idle : 5958514 [Wed Feb 6 09:40:55 2019][1697229.766291] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:55 2019][1697229.771602] [Wed Feb 6 09:40:55 2019][1697229.773274] cfs_rq[25]:/ [Wed Feb 6 09:40:55 2019][1697229.775985] .exec_clock : 0.000000 [Wed Feb 6 09:40:55 2019][1697229.781386] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:55 2019][1697229.786784] .min_vruntime : 6442727.668721 [Wed Feb 6 09:40:55 2019][1697229.792704] .max_vruntime : 0.000001 [Wed Feb 6 09:40:55 2019][1697229.798103] .spread : 0.000000 [Wed Feb 6 09:40:55 2019][1697229.803503] .spread0 : -2400637.270840 [Wed Feb 6 09:40:55 2019][1697229.809508] .nr_spread_over : 0 [Wed Feb 6 09:40:55 2019][1697229.814302] .nr_running : 0 [Wed Feb 6 09:40:55 2019][1697229.819093] .load : 0 [Wed Feb 6 09:40:55 2019][1697229.823887] .runnable_load_avg : 0 [Wed Feb 6 09:40:55 2019][1697229.828680] .blocked_load_avg : 0 [Wed Feb 6 09:40:55 2019][1697229.833473] .tg_load_avg : 0 [Wed Feb 6 09:40:55 2019][1697229.838264] .tg_load_contrib : 0 [Wed Feb 6 09:40:55 2019][1697229.843057] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:55 2019][1697229.847851] .tg->runnable_avg : 0 [Wed Feb 6 09:40:55 2019][1697229.852642] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:55 2019][1697229.857435] .throttled : 0 [Wed Feb 6 09:40:55 2019][1697229.862228] .throttle_count : 0 [Wed Feb 6 09:40:55 2019][1697229.867020] .avg->runnable_avg_sum : 2 [Wed Feb 6 09:40:55 2019][1697229.871813] .avg->runnable_avg_period : 46830 [Wed Feb 6 09:40:55 2019][1697229.876956] [Wed Feb 6 09:40:55 2019][1697229.878626] rt_rq[25]:/ [Wed Feb 6 09:40:55 2019][1697229.881251] .rt_nr_running : 0 [Wed Feb 6 09:40:55 2019][1697229.886043] .rt_throttled : 0 [Wed Feb 6 09:40:55 2019][1697229.890837] .rt_time : 0.000000 [Wed Feb 6 09:40:55 2019][1697229.896238] .rt_runtime : 979.166667 [Wed Feb 6 09:40:55 2019][1697229.901809] [Wed Feb 6 09:40:55 2019][1697229.903482] runnable tasks: [Wed Feb 6 09:40:55 2019][1697229.906454] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:55 2019][1697229.917401] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:55 2019][1697229.928277] watchdog/25 136 -11.986555 424271 0 0.000000 115964.709631 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697229.939613] migration/25 137 0.000000 3071 0 0.000000 215967.280953 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697229.950950] ksoftirqd/25 138 6442715.683068 2877522 120 0.000000 34146.378449 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697229.962287] kworker/25:0H 140 5344.456971 10 100 0.000000 0.036639 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697229.973648] multipathd 16279 0.000000 1 0 0.000000 0.591564 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697229.984985] multipathd 16358 0.000000 1 0 0.000000 0.564163 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697229.996321] multipathd 16482 0.000000 1 0 0.000000 0.917638 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.007657] multipathd 17731 0.000000 1 0 0.000000 0.644153 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.018992] multipathd 18675 0.000000 1 0 0.000000 0.944088 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.030347] kworker/25:1H 46340 6442715.668742 1763165 100 0.000000 6401.348686 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.041695] ptlrpcd_01_08 126461 6442715.766906 4022904 120 0.000000 65575.914920 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.053123] ldlm_cn01_002 126600 6442704.646287 9349463 120 0.000000 370152.326674 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.064559] osp-syn-15-3 126774 6442715.703816 126896 120 0.000000 3569.549690 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.075986] osp-syn-7-1 126949 6442715.741548 198917 120 0.000000 5087.865670 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.087410] osp-syn-22-1 126969 6442715.739634 198224 120 0.000000 4580.984579 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.098837] mdt01_010 127212 6266563.822297 34569288 120 0.000000 1533158.605368 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.110264] mdt_io01_003 127219 3700296.487906 1587624 120 0.000000 56026.256995 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.121692] mdt_io01_018 127697 6231684.713461 2890143 120 0.000000 99758.804428 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.133119] mdt_io01_020 127710 6442716.000051 2447204 120 0.000000 109746.529631 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.144544] mdt01_015 129229 6442715.677137 35885914 120 0.000000 1800373.500634 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.155969] mdt_io01_027 129300 3699799.935850 3049310 120 0.000000 116899.974114 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.167404] ldlm_cn01_011 1442 6442706.979020 6630813 120 0.000000 321080.138310 0.000000 1 / [Wed Feb 6 09:40:55 2019][1697230.178738] ldlm_cn01_012 1443 6442696.644225 7228486 120 0.000000 326054.133671 0.000000 1 / [Wed Feb 6 09:40:56 2019][1697230.190079] kworker/25:2 37144 4108362.602180 60681 120 0.000000 924.398609 0.000000 1 / [Wed Feb 6 09:40:56 2019][1697230.201411] mdt_out01_002 38264 6128770.517181 201 120 0.000000 48.654916 0.000000 1 / [Wed Feb 6 09:40:56 2019][1697230.212756] ldlm_bl_15 10656 6442715.677167 1326424 120 0.000000 1090709.450028 0.000000 1 / [Wed Feb 6 09:40:56 2019][1697230.224094] kworker/25:1 13104 6442693.935571 16658 120 0.000000 12190.223353 0.000000 1 / [Wed Feb 6 09:40:56 2019][1697230.235440] mdt01_042 17958 6442715.759682 287957 120 0.000000 555455.395150 0.000000 1 / [Wed Feb 6 09:40:56 2019][1697230.246781] mdt01_059 19097 6266563.823560 124924 120 0.000000 166271.412947 0.000000 1 / [Wed Feb 6 09:40:56 2019][1697230.258142] [Wed Feb 6 09:40:56 2019][1697230.259819] cpu#26, 1996.233 MHz [Wed Feb 6 09:40:56 2019][1697230.263228] .nr_running : 0 [Wed Feb 6 09:40:56 2019][1697230.268018] .load : 0 [Wed Feb 6 09:40:56 2019][1697230.272811] .nr_switches : 455366215 [Wed Feb 6 09:40:56 2019][1697230.278298] .nr_load_updates : 146339153 [Wed Feb 6 09:40:56 2019][1697230.283784] .nr_uninterruptible : 43618 [Wed Feb 6 09:40:56 2019][1697230.288923] .next_balance : 5991.883061 [Wed Feb 6 09:40:56 2019][1697230.294582] .curr->pid : 0 [Wed Feb 6 09:40:56 2019][1697230.299374] .clock : 1697230298.854094 [Wed Feb 6 09:40:56 2019][1697230.305555] .cpu_load[0] : 0 [Wed Feb 6 09:40:56 2019][1697230.310348] .cpu_load[1] : 0 [Wed Feb 6 09:40:56 2019][1697230.315154] .cpu_load[2] : 0 [Wed Feb 6 09:40:56 2019][1697230.319952] .cpu_load[3] : 0 [Wed Feb 6 09:40:56 2019][1697230.324744] .cpu_load[4] : 0 [Wed Feb 6 09:40:56 2019][1697230.329537] .avg_idle : 10000000 [Wed Feb 6 09:40:56 2019][1697230.334936] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:56 2019][1697230.340261] [Wed Feb 6 09:40:56 2019][1697230.341935] cfs_rq[26]:/ [Wed Feb 6 09:40:56 2019][1697230.344647] .exec_clock : 0.000000 [Wed Feb 6 09:40:56 2019][1697230.350047] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:56 2019][1697230.355445] .min_vruntime : 5275415.609299 [Wed Feb 6 09:40:56 2019][1697230.361365] .max_vruntime : 0.000001 [Wed Feb 6 09:40:56 2019][1697230.366765] .spread : 0.000000 [Wed Feb 6 09:40:56 2019][1697230.372165] .spread0 : -3567949.330262 [Wed Feb 6 09:40:56 2019][1697230.378169] .nr_spread_over : 0 [Wed Feb 6 09:40:56 2019][1697230.382963] .nr_running : 0 [Wed Feb 6 09:40:56 2019][1697230.387755] .load : 0 [Wed Feb 6 09:40:56 2019][1697230.392549] .runnable_load_avg : 0 [Wed Feb 6 09:40:56 2019][1697230.397340] .blocked_load_avg : 0 [Wed Feb 6 09:40:56 2019][1697230.402133] .tg_load_avg : 0 [Wed Feb 6 09:40:56 2019][1697230.406926] .tg_load_contrib : 0 [Wed Feb 6 09:40:56 2019][1697230.411718] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:56 2019][1697230.416513] .tg->runnable_avg : 0 [Wed Feb 6 09:40:56 2019][1697230.421305] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:56 2019][1697230.426097] .throttled : 0 [Wed Feb 6 09:40:56 2019][1697230.430891] .throttle_count : 0 [Wed Feb 6 09:40:56 2019][1697230.435683] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:40:56 2019][1697230.440477] .avg->runnable_avg_period : 46563 [Wed Feb 6 09:40:56 2019][1697230.445618] [Wed Feb 6 09:40:56 2019][1697230.447287] rt_rq[26]:/ [Wed Feb 6 09:40:56 2019][1697230.449916] .rt_nr_running : 0 [Wed Feb 6 09:40:56 2019][1697230.454707] .rt_throttled : 0 [Wed Feb 6 09:40:56 2019][1697230.459501] .rt_time : 0.000000 [Wed Feb 6 09:40:56 2019][1697230.464900] .rt_runtime : 958.767362 [Wed Feb 6 09:40:56 2019][1697230.470471] [Wed Feb 6 09:40:56 2019][1697230.472145] runnable tasks: [Wed Feb 6 09:40:56 2019][1697230.475117] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:56 2019][1697230.486061] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:56 2019][1697230.496940] watchdog/26 141 -11.985924 424286 0 0.000000 108600.337284 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.508275] migration/26 142 0.000000 1529 0 0.000000 177518.374794 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.519611] ksoftirqd/26 143 5275338.335218 2160749 120 0.000000 39696.830991 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.530949] kworker/26:0H 145 3193.657941 10 100 0.000000 0.044696 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.542303] kworker/26:1 7286 5275348.665913 269791 120 0.000000 14945.969455 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.553658] kworker/26:1H 18900 5240065.093301 1221451 100 0.000000 5425.238367 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.564999] ptlrpc_hr02_002 126394 5275403.685332 15299424 120 0.000000 124981.007595 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.576433] ptlrpcd_02_04 126469 5275403.923719 6884063 120 0.000000 83805.420039 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.587866] mdt_io02_000 126671 5002829.767742 2167597 120 0.000000 97618.618538 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.599292] osp-pre-1-3 126779 5275403.632984 214592 120 0.000000 7803.670496 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.610723] osp-syn-46-1 126899 5275403.694639 196930 120 0.000000 5607.634707 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.622146] osp-syn-23-1 126943 5275390.273276 198630 120 0.000000 4826.995152 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.633575] mdt_io02_007 127264 5275334.036822 2581598 120 0.000000 133593.784719 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.644996] mdt_io02_010 127350 3443602.799622 1940264 120 0.000000 75204.297698 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.656418] mdt_io02_013 127669 5005356.413857 2002229 120 0.000000 93181.356933 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.667878] mdt02_027 17830 5275323.618385 208306 120 0.000000 18813.937898 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.679222] mdt02_052 18269 5034101.526460 156131 120 0.000000 315445.840656 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.690562] ldlm_cn02_026 23739 5275415.609299 25446 120 0.000000 81366.188181 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.701909] kworker/26:0 24918 5014048.250503 2 120 0.000000 0.011701 0.000000 2 / [Wed Feb 6 09:40:56 2019][1697230.713270] [Wed Feb 6 09:40:56 2019][1697230.714946] cpu#27, 1996.233 MHz [Wed Feb 6 09:40:56 2019][1697230.718354] .nr_running : 1 [Wed Feb 6 09:40:56 2019][1697230.723148] .load : 1024 [Wed Feb 6 09:40:56 2019][1697230.728206] .nr_switches : 91085834 [Wed Feb 6 09:40:56 2019][1697230.733607] .nr_load_updates : 49688115 [Wed Feb 6 09:40:56 2019][1697230.739007] .nr_uninterruptible : 763 [Wed Feb 6 09:40:56 2019][1697230.743973] .next_balance : 5991.883616 [Wed Feb 6 09:40:56 2019][1697230.749632] .curr->pid : 10650 [Wed Feb 6 09:40:56 2019][1697230.754772] .clock : 1697230753.857393 [Wed Feb 6 09:40:56 2019][1697230.760950] .cpu_load[0] : 1024 [Wed Feb 6 09:40:56 2019][1697230.766003] .cpu_load[1] : 1024 [Wed Feb 6 09:40:56 2019][1697230.771056] .cpu_load[2] : 1024 [Wed Feb 6 09:40:56 2019][1697230.776109] .cpu_load[3] : 1024 [Wed Feb 6 09:40:56 2019][1697230.781178] .cpu_load[4] : 1024 [Wed Feb 6 09:40:56 2019][1697230.786233] .avg_idle : 3870301 [Wed Feb 6 09:40:56 2019][1697230.791543] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:56 2019][1697230.796857] [Wed Feb 6 09:40:56 2019][1697230.798526] cfs_rq[27]:/ [Wed Feb 6 09:40:56 2019][1697230.801238] .exec_clock : 0.000000 [Wed Feb 6 09:40:56 2019][1697230.806638] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:56 2019][1697230.812038] .min_vruntime : 1269792.803035 [Wed Feb 6 09:40:56 2019][1697230.817957] .max_vruntime : 0.000001 [Wed Feb 6 09:40:56 2019][1697230.823357] .spread : 0.000000 [Wed Feb 6 09:40:56 2019][1697230.828755] .spread0 : -7573572.136526 [Wed Feb 6 09:40:56 2019][1697230.834762] .nr_spread_over : 0 [Wed Feb 6 09:40:56 2019][1697230.839554] .nr_running : 1 [Wed Feb 6 09:40:56 2019][1697230.844348] .load : 1024 [Wed Feb 6 09:40:56 2019][1697230.849400] .runnable_load_avg : 1023 [Wed Feb 6 09:40:56 2019][1697230.854453] .blocked_load_avg : 0 [Wed Feb 6 09:40:56 2019][1697230.859247] .tg_load_avg : 0 [Wed Feb 6 09:40:56 2019][1697230.864039] .tg_load_contrib : 0 [Wed Feb 6 09:40:56 2019][1697230.868833] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:56 2019][1697230.873625] .tg->runnable_avg : 0 [Wed Feb 6 09:40:56 2019][1697230.878416] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:56 2019][1697230.883209] .throttled : 0 [Wed Feb 6 09:40:56 2019][1697230.888001] .throttle_count : 0 [Wed Feb 6 09:40:56 2019][1697230.892795] .avg->runnable_avg_sum : 46453 [Wed Feb 6 09:40:56 2019][1697230.897934] .avg->runnable_avg_period : 46257 [Wed Feb 6 09:40:56 2019][1697230.903075] [Wed Feb 6 09:40:56 2019][1697230.904748] rt_rq[27]:/ [Wed Feb 6 09:40:56 2019][1697230.907374] .rt_nr_running : 0 [Wed Feb 6 09:40:56 2019][1697230.912164] .rt_throttled : 0 [Wed Feb 6 09:40:56 2019][1697230.916959] .rt_time : 0.000000 [Wed Feb 6 09:40:56 2019][1697230.922358] .rt_runtime : 963.162549 [Wed Feb 6 09:40:56 2019][1697230.927931] [Wed Feb 6 09:40:56 2019][1697230.929601] runnable tasks: [Wed Feb 6 09:40:56 2019][1697230.932576] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:56 2019][1697230.943521] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:56 2019][1697230.954398] watchdog/27 146 -11.985232 424306 0 0.000000 62056.985155 0.000000 3 / [Wed Feb 6 09:40:56 2019][1697230.965735] migration/27 147 0.000000 1341 0 0.000000 101246.587692 0.000000 3 / [Wed Feb 6 09:40:56 2019][1697230.977070] ksoftirqd/27 148 1263301.695997 159446 120 0.000000 3927.613815 0.000000 3 / [Wed Feb 6 09:40:56 2019][1697230.988408] kworker/27:0H 150 4837.914852 10 100 0.000000 0.037720 0.000000 3 / [Wed Feb 6 09:40:56 2019][1697230.999761] kworker/27:1 7274 1264157.637956 42892 120 0.000000 7296.592782 0.000000 3 / [Wed Feb 6 09:40:56 2019][1697231.011120] kworker/27:1H 40527 1245110.704892 160940 100 0.000000 492.680204 0.000000 3 / [Wed Feb 6 09:40:56 2019][1697231.022482] ldlm_cn03_000 126604 1264231.019177 2116848 120 0.000000 116198.510060 0.000000 3 / [Wed Feb 6 09:40:56 2019][1697231.033906] osp-syn-12-3 126704 1264147.043373 129354 120 0.000000 3656.291141 0.000000 3 / [Wed Feb 6 09:40:56 2019][1697231.045358] mdt03_008 127317 1152924.485201 8420209 120 0.000000 387051.559555 0.000000 3 / [Wed Feb 6 09:40:56 2019][1697231.056819] R ldlm_bl_09 10650 1270010.118820 1324030 120 0.000000 1460870.267945 0.000000 3 / [Wed Feb 6 09:40:56 2019][1697231.068156] mdt03_015 17805 1264157.711004 42435 120 0.000000 2548.230658 0.000000 3 / [Wed Feb 6 09:40:56 2019][1697231.079487] mdt03_023 17829 1153024.382082 50424 120 0.000000 6151.356724 0.000000 3 / [Wed Feb 6 09:40:56 2019][1697231.090824] mdt03_026 17853 1237802.552618 94739 120 0.000000 6617.332904 0.000000 3 / [Wed Feb 6 09:40:56 2019][1697231.102169] mdt03_044 18177 1255060.433637 79866 120 0.000000 157257.685019 0.000000 3 / [Wed Feb 6 09:40:56 2019][1697231.113508] kworker/27:2 19266 919213.279220 2 120 0.000000 0.009578 0.000000 3 / [Wed Feb 6 09:40:56 2019][1697231.124865] [Wed Feb 6 09:40:56 2019][1697231.126540] cpu#28, 1996.233 MHz [Wed Feb 6 09:40:56 2019][1697231.129949] .nr_running : 0 [Wed Feb 6 09:40:56 2019][1697231.134749] .load : 0 [Wed Feb 6 09:40:56 2019][1697231.139541] .nr_switches : 1790989756 [Wed Feb 6 09:40:56 2019][1697231.145116] .nr_load_updates : 290479731 [Wed Feb 6 09:40:56 2019][1697231.150601] .nr_uninterruptible : -82935 [Wed Feb 6 09:40:56 2019][1697231.155827] .next_balance : 5991.883929 [Wed Feb 6 09:40:56 2019][1697231.161485] .curr->pid 6280] .clock : 1697231166.130599 [Wed Feb 6 09:40:56 2019][1697231.172457] .cpu_load[0] : 0 [Wed Feb 6 09:40:56 2019][1697231.177250] .cpu_load[1] : 0 [Wed Feb 6 09:40:56 2019][1697231.182043] .cpu_load[2] : 0 [Wed Feb 6 09:40:56 2019][1697231.186836] .cpu_load[3] : 0 [Wed Feb 6 09:40:57 2019][1697231.191627] .cpu_load[4] : 0 [Wed Feb 6 09:40:57 2019][1697231.196422] .avg_idle : 523470 [Wed Feb 6 09:40:57 2019][1697231.201649] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:57 2019][1697231.206962] [Wed Feb 6 09:40:57 2019][1697231.208629] cfs_rq[28]:/ [Wed Feb 6 09:40:57 2019][1697231.211342] .exec_clock : 0.000000 [Wed Feb 6 09:40:57 2019][1697231.216742] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:57 2019][1697231.222140] .min_vruntime : 8444209.802160 [Wed Feb 6 09:40:57 2019][1697231.228062] .max_vruntime : 0.000001 [Wed Feb 6 09:40:57 2019][1697231.233459] .spread : 0.000000 [Wed Feb 6 09:40:57 2019][1697231.238861] .spread0 : -399155.137401 [Wed Feb 6 09:40:57 2019][1697231.244780] .nr_spread_over : 0 [Wed Feb 6 09:40:57 2019][1697231.249572] .nr_running : 0 [Wed Feb 6 09:40:57 2019][1697231.254364] .load : 0 [Wed Feb 6 09:40:57 2019][1697231.259158] .runnable_load_avg : 0 [Wed Feb 6 09:40:57 2019][1697231.263951] .blocked_load_avg : 32 [Wed Feb 6 09:40:57 2019][1697231.268830] .tg_load_avg : 0 [Wed Feb 6 09:40:57 2019][1697231.273621] .tg_load_contrib : 0 [Wed Feb 6 09:40:57 2019][1697231.278415] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:57 2019][1697231.283209] .tg->runnable_avg : 0 [Wed Feb 6 09:40:57 2019][1697231.288001] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:57 2019][1697231.292794] .throttled : 0 [Wed Feb 6 09:40:57 2019][1697231.297584] .throttle_count : 0 [Wed Feb 6 09:40:57 2019][1697231.302378] .avg->runnable_avg_sum : 740 [Wed Feb 6 09:40:57 2019][1697231.307345] .avg->runnable_avg_period : 47272 [Wed Feb 6 09:40:57 2019][1697231.312487] [Wed Feb 6 09:40:57 2019][1697231.314156] rt_rq[28]:/ [Wed Feb 6 09:40:57 2019][1697231.316783] .rt_nr_running : 0 [Wed Feb 6 09:40:57 2019][1697231.321575] .rt_throttled : 0 [Wed Feb 6 09:40:57 2019][1697231.326367] .rt_time : 0.009839 [Wed Feb 6 09:40:57 2019][1697231.331769] .rt_runtime : 958.767404 [Wed Feb 6 09:40:57 2019][1697231.337340] [Wed Feb 6 09:40:57 2019][1697231.339014] runnable tasks: [Wed Feb 6 09:40:57 2019][1697231.341985] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:57 2019][1697231.352933] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:57 2019][1697231.363792] kthreadd 2 8435683.589013 13125 120 0.000000 2290.975445 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.375144] watchdog/28 151 -11.987507 424299 0 0.000000 75594.870138 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.386480] migration/28 152 0.000000 1315 0 0.000000 176613.408501 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.397817] ksoftirqd/28 153 8444117.919916 1442779 120 0.000000 20519.894460 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.409154] kworker/28:0H 155 3780.232985 10 100 0.000000 0.041498 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.420525] gmain 16798 3687.091306 3 120 0.000000 0.057186 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.431866] kworker/28:1H 18016 8444117.778915 1583857 100 0.000000 8328.517264 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.443222] ptlrpcd_00_00 126441 8444174.559282 4263275 120 0.000000 69002.924085 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.454645] ptlrpcd_00_06 126447 8444174.464161 4318222 120 0.000000 70855.145467 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.466077] ldlm_cn00_002 126597 8444205.044484 5056640 120 0.000000 229676.106911 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.477502] mdt_rdpg00_000 126635 8431889.288186 5063510 120 0.000000 200288.708851 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.488954] mdt00_004 127134 8336012.667084 80979676 120 0.000000 2494462.694055 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.500382] R mdt00_007 127214 8444290.998105 81959006 120 0.000000 2339178.935269 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.511804] ldlm_cn00_004 127232 8444136.441683 3347493 120 0.000000 200208.925255 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.523227] mdt00_008 127247 8442790.275060 87167788 120 0.000000 2361537.902800 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.534666] mdt00_019 129262 8353042.149292 68479603 120 0.000000 2115111.111496 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.546100] mdt_io00_026 1417 6700526.930418 1394693 120 0.000000 68968.677425 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.557439] kworker/28:0 34393 7922371.869618 8 120 0.000000 0.024588 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.568778] kworker/28:1 39909 8444174.388089 143567 120 0.000000 12751.446567 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.580131] mdt00_032 17889 8351079.338684 209872 120 0.000000 10676.645729 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.591467] mdt00_037 17904 8351973.099004 126650 120 0.000000 164787.838975 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.602805] mdt00_042 17919 8336012.667234 220677 120 0.000000 466767.383814 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.614144] mdt00_054 17983 8350550.100919 178081 120 0.000000 310294.942224 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.625498] mdt_rdpg00_008 24897 8431889.296061 24 120 0.000000 2319.545414 0.000000 0 / [Wed Feb 6 09:40:57 2019][1697231.636845] [Wed Feb 6 09:40:57 2019][1697231.638521] cpu#29, 1996.233 MHz [Wed Feb 6 09:40:57 2019][1697231.641929] .nr_running : 0 [Wed Feb 6 09:40:57 2019][1697231.646723] .load : 0 [Wed Feb 6 09:40:57 2019][1697231.651516] .nr_switches : 1041672478 [Wed Feb 6 09:40:57 2019][1697231.657088] .nr_load_updates : 133970156 [Wed Feb 6 09:40:57 2019][1697231.662572] .nr_uninterruptible : 20780 [Wed Feb 6 09:40:57 2019][1697231.667713] .next_balance : 5991.884441 [Wed Feb 6 09:40:57 2019][1697231.673374] .curr->pid : 0 [Wed Feb 6 09:40:57 2019][1697231.678175] .clock : 1697231677.868277 [Wed Feb 6 09:40:57 2019][1697231.684354] .cpu_load[0] : 0 [Wed Feb 6 09:40:57 2019][1697231.689146] .cpu_load[1] : 1 [Wed Feb 6 09:40:57 2019][1697231.693938] .cpu_load[2] : 3 [Wed Feb 6 09:40:57 2019][1697231.698732] .cpu_load[3] : 8 [Wed Feb 6 09:40:57 2019][1697231.703523] .cpu_load[4] : 10 [Wed Feb 6 09:40:57 2019][1697231.708405] .avg_idle : 431642 [Wed Feb 6 09:40:57 2019][1697231.713629] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:57 2019][1697231.718941] [Wed Feb 6 09:40:57 2019][1697231.720611] cfs_rq[29]:/ [Wed Feb 6 09:40:57 2019][1697231.723323] .exec_clock : 0.000000 [Wed Feb 6 09:40:57 2019][1697231.728723] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:57 2019][1697231.734124] .min_vruntime : 6755526.235758 [Wed Feb 6 09:40:57 2019][1697231.740043] .max_vruntime : 0.000001 [Wed Feb 6 09:40:57 2019][1697231.745441] .spread : 0.000000 [Wed Feb 6 09:40:57 2019][1697231.750840] .spread0 : -2087840.425897 [Wed Feb 6 09:40:57 2019][1697231.756846] .nr_spread_over : 0 [Wed Feb 6 09:40:57 2019][1697231.761640] .nr_running : 0 [Wed Feb 6 09:40:57 2019][1697231.766434] .load : 0 [Wed Feb 6 09:40:57 2019][1697231.771227] .runnable_load_avg : 0 [Wed Feb 6 09:40:57 2019][1697231.776017] .blocked_load_avg : 14 [Wed Feb 6 09:40:57 2019][1697231.780898] .tg_load_avg : 0 [Wed Feb 6 09:40:57 2019][1697231.785691] .tg_load_contrib : 0 [Wed Feb 6 09:40:57 2019][1697231.790482] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:57 2019][1697231.795276] .tg->runnable_avg : 0 [Wed Feb 6 09:40:57 2019][1697231.800067] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:57 2019][1697231.804861] .throttled : 0 [Wed Feb 6 09:40:57 2019][1697231.809653] .throttle_count : 0 [Wed Feb 6 09:40:57 2019][1697231.814447] .avg->runnable_avg_sum : 655 [Wed Feb 6 09:40:57 2019][1697231.819413] .avg->runnable_avg_period : 46788 [Wed Feb 6 09:40:57 2019][1697231.824559] [Wed Feb 6 09:40:57 2019][1697231.826233] rt_rq[29]:/ [Wed Feb 6 09:40:57 2019][1697231.828859] .rt_nr_running : 0 [Wed Feb 6 09:40:57 2019][1697231.833651] .rt_throttled : 0 [Wed Feb 6 09:40:57 2019][1697231.838444] .rt_time : 0.000000 [Wed Feb 6 09:40:57 2019][1697231.843845] .rt_runtime : 938.793106 [Wed Feb 6 09:40:57 2019][1697231.849416] [Wed Feb 6 09:40:57 2019][1697231.851089] runnable tasks: [Wed Feb 6 09:40:57 2019][1697231.854062] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:57 2019][1697231.865008] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:57 2019][1697231.875887] watchdog/29 156 -11.986475 424290 0 0.000000 80643.351680 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697231.887220] migration/29 157 0.000000 5240 0 0.000000 211714.227227 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697231.898557] ksoftirqd/29 158 6753485.917341 1964719 120 0.000000 22265.263180 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697231.909893] kworker/29:0H 160 5269.845950 10 100 0.000000 0.032992 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697231.921266] polkitd 16801 3543.355266 1 120 0.000000 0.026991 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697231.932613] kworker/29:1H 46362 6753087.111273 1241335 100 0.000000 5275.972920 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697231.943948] ptlrpc_hr01_001 126387 6755486.063855 28985457 120 0.000000 218613.431530 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697231.955373] kiblnd_sd_01_02 126427 6755529.969795 816347714 120 0.000000 5804169.533123 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697231.966825] osp-syn-39-1 126897 6755467.105978 196708 120 0.000000 5435.083425 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697231.978252] osp-syn-21-1 126935 6755467.107872 198458 120 0.000000 6112.925132 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697231.989676] osp-pre-29-1 126966 6755462.911388 213960 120 0.000000 4811.633005 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697232.001111] ldlm_cn01_006 127800 6755341.385377 9313460 120 0.000000 367787.504591 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697232.012547] mdt_io01_026 129285 6755103.852880 1848681 120 0.000000 77518.541000 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697232.023967] mdt_io01_028 129301 6660205.856076 3688167 120 0.000000 151135.864481 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697232.035402] mdt_io01_034 3361 6687533.865319 2232189 120 0.000000 98965.060061 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697232.046736] mdt_rdpg01_008 18321 6755521.115402 131359 120 0.000000 9824.562128 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697232.058074] kworker/29:0 54901 6755487.852518 60697 120 0.000000 17183.311669 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697232.069406] kworker/29:1 115868 4648016.024657 3 120 0.000000 0.013215 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697232.080859] mdt01_055 19060 6755521.062653 303061 120 0.000000 472772.852731 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697232.092200] ldlm_cn01_027 23735 6755530.950735 49642 120 0.000000 150987.441100 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697232.103545] kworker/u258:2 24880 6674828.852849 7 120 0.000000 0.478681 0.000000 1 / [Wed Feb 6 09:40:57 2019][1697232.114906] [Wed Feb 6 09:40:57 2019][1697232.116583] cpu#30, 1996.233 MHz [Wed Feb 6 09:40:57 2019][1697232.119990] .nr_running : 0 [Wed Feb 6 09:40:57 2019][1697232.124783] .load : 0 [Wed Feb 6 09:40:57 2019][1697232.129588] .nr_switches : 251710760 [Wed Feb 6 09:40:57 2019][1697232.135080] .nr_load_updates : 106578768 [Wed Feb 6 09:40:57 2019][1697232.140577] .nr_uninterruptible : 16954 [Wed Feb 6 09:40:57 2019][1697232.145728] .next_balance : 5991.884917 [Wed Feb 6 09:40:57 2019][1697232.151395] .curr->pid : 0 [Wed Feb 6 09:40:57 2019][1697232.156195] .clock : 1697232155.868641 [Wed Feb 6 09:40:57 2019][1697232.162383] .cpu_load[0] : 0 [Wed Feb 6 09:40:57 2019][1697232.167182] .cpu_load[1] : 0 [Wed Feb 6 09:40:57 2019][1697232.171984] .cpu_load[2] : 0 [Wed Feb 6 09:40:57 2019][1697232.176784] .cpu_load[3] : 98 [Wed Feb 6 09:40:57 2019][1697232.181678] .cpu_load[4] : 39 [Wed Feb 6 09:40:57 2019][1697232.186566] .avg_idle : 751849 [Wed Feb 6 09:40:57 2019][1697232.191798] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:58 2019][1697232.197120] [Wed Feb 6 09:40:58 2019][1697232.198794] cfs_rq[30]:/ [Wed Feb 6 09:40:58 2019][1697232.201506] .exec_clock : 0.000000 [Wed Feb 6 09:40:58 2019][1697232.206905] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:58 2019][1697232.212303] .min_vruntime : 2987357.708370 [Wed Feb 6 09:40:58 2019][1697232.218226] .max_vruntime : 0.000001 [Wed Feb 6 09:40:58 2019][1697232.223626] .spread : 0.000000 [Wed Feb 6 09:40:58 2019][1697232.229025] .spread0 : -5856029.589053 [Wed Feb 6 09:40:58 2019][1697232.235031] .nr_spread_over : 0 [Wed Feb 6 09:40:58 2019][1697232.239821] .nr_running : 0 [Wed Feb 6 09:40:58 2019][1697232.244614] .load : 0 [Wed Feb 6 09:40:58 2019][1697232.249406] .runnable_load_avg : 0 [Wed Feb 6 09:40:58 2019][1697232.254199] .blocked_load_avg : 0 [Wed Feb 6 09:40:58 2019][1697232.258996] .tg_load_avg : 0 [Wed Feb 6 09:40:58 2019][1697232.263797] .tg_load_contrib : 0 [Wed Feb 6 09:40:58 2019][1697232.268588] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:58 2019][1697232.273379] .tg->runnable_avg : 0 [Wed Feb 6 09:40:58 2019][1697232.278171] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:58 2019][1697232.282965] .throttled : 0 [Wed Feb 6 09:40:58 2019][1697232.287757] .throttle_count : 0 [Wed Feb 6 09:40:58 2019][1697232.292552] .avg->runnable_avg_sum : 10227 [Wed Feb 6 09:40:58 2019][1697232.297692] .avg->runnable_avg_period : 46419 [Wed Feb 6 09:40:58 2019][1697232.302838] [Wed Feb 6 09:40:58 2019][1697232.304511] rt_rq[30]:/ [Wed Feb 6 09:40:58 2019][1697232.307137] .rt_nr_running : 0 [Wed Feb 6 09:40:58 2019][1697232.311929] .rt_throttled : 0 [Wed Feb 6 09:40:58 2019][1697232.316721] .rt_time : 0.000000 [Wed Feb 6 09:40:58 2019][1697232.322120] .rt_runtime : 963.526530 [Wed Feb 6 09:40:58 2019][1697232.327692] [Wed Feb 6 09:40:58 2019][1697232.329366] runnable tasks: [Wed Feb 6 09:40:58 2019][1697232.332339] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:58 2019][1697232.343285] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:58 2019][1697232.354163] watchdog/30 161 -11.985794 424297 0 0.000000 84107.124620 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.365497] migration/30 162 0.000000 1267 0 0.000000 189420.316905 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.376835] ksoftirqd/30 163 2987370.728831 1906496 120 0.000000 26768.797397 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.388169] kworker/30:0 164 1859.323316 22 120 0.000000 0.097312 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.399505] kworker/30:0H 165 4171.668046 10 100 0.000000 0.035688 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.410856] kworker/30:1 2164 2987339.451626 192044 120 0.000000 12117.623769 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.422199] ccp-1-q4 9981 2462.917666 2 120 0.000000 0.012223 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.433550] writer#4 18230 2987331.873969 24921465 120 0.000000 230905.213564 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.444893] kworker/30:1H 40711 2987294.743842 1148382 100 0.000000 4655.243963 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.456234] ptlrpc_hr02_004 126396 2987338.691906 15299424 120 0.000000 124267.498539 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.467658] kiblnd_sd_02_02 126431 2987385.044308 720352841 120 0.000000 4648133.784529 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.479077] kiblnd_sd_02_03 126432 2987385.202880 720371568 120 0.000000 4642942.300145 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.490500] lnet_discovery 126439 2987339.911995 1031264 120 0.000000 13415.950248 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.501926] ptlrpcd_02_02 126467 2987370.969829 6897176 120 0.000000 88205.983680 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.513349] kmmpd-dm-1 126574 2987294.764643 411761 120 0.000000 14389.250471 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.524786] osp-syn-0-3 126738 2987238.322040 128105 120 0.000000 3667.746221 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.536212] osp-syn-6-3 126766 2987292.544303 128399 120 0.000000 3822.107476 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.547635] osp-pre-12-1 126894 2987298.242780 213948 120 0.000000 6376.998926 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.559058] osp-pre-0-1 126928 2987371.314690 213965 120 0.000000 9073.783539 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.570479] osp-syn-33-1 126945 2987338.775633 198720 120 0.000000 5198.533864 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.581903] osp-syn-34-1 126987 2987297.664090 197837 120 0.000000 4665.224916 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.593324] mdt_out02_002 127137 2862481.434228 18543977 120 0.000000 290189.544961 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.604747] mdt_io02_003 127218 1425380.004376 1960652 120 0.000000 72272.080425 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.616167] mdt_io02_004 127224 1425379.982175 1303563 120 0.000000 55825.189345 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.627593] ldlm_cn02_004 127239 2987338.273277 5196619 120 0.000000 309662.702550 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.639021] ldlm_cn02_006 127802 2987338.276143 5778737 120 0.000000 321709.333161 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.650447] mdt02_012 128267 2862499.907256 27257019 120 0.000000 1188090.839042 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.661870] ldlm_cn02_007 129080 2987336.620393 7905884 120 0.000000 342642.800574 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.673311] mdt_rdpg02_005 3266 2987242.662814 6134579 120 0.000000 268309.664715 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.684658] ldlm_cn02_015 14199 2987338.272175 942904 120 0.000000 164922.244887 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.695999] mdt02_028 17834 2987019.050389 63585 120 0.000000 157140.048304 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.707333] mdt02_030 17839 2962740.588363 50955 120 0.000000 6792.175271 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.718667] mdt02_031 17840 2853360.714962 141184 120 0.000000 314237.762518 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.730018] mdt02_049 18265 2986798.025805 207798 120 0.000000 16417.664274 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.741356] ldlm_cn02_017 23719 2987350.277024 19733 120 0.000000 95267.998300 0.000000 2 / [Wed Feb 6 09:40:58 2019][1697232.752751] [Wed Feb 6 09:40:58 2019][1697232.754423] cpu#31, 1996.233 MHz [Wed Feb 6 09:40:58 2019][1697232.757832] .nr_running : 0 [Wed Feb 6 09:40:58 2019][1697232.762632] .load : 0 [Wed Feb 6 09:40:58 2019][1697232.767434] .nr_switches : 100328654 [Wed Feb 6 09:40:58 2019][1697232.772928] .nr_load_updates : 56340454 [Wed Feb 6 09:40:58 2019][1697232.778330] .nr_uninterruptible : 1129 [Wed Feb 6 09:40:58 2019][1697232.783391] .next_balance : 5991.885555 [Wed Feb 6 09:40:58 2019][1697232.789058] .curr->pid : 0 [Wed Feb 6 09:40:58 2019][1697232.793859] .clock : 1697232792.875624 [Wed Feb 6 09:40:58 2019][1697232.800048] .cpu_load[0] : 0 [Wed Feb 6 09:40:58 2019][1697232.804848] .cpu_load[1] : 0 [Wed Feb 6 09:40:58 2019][1697232.809652] .cpu_load[2] : 0 [Wed Feb 6 09:40:58 2019][1697232.814451] .cpu_load[3] : 0 [Wed Feb 6 09:40:58 2019][1697232.819254] .cpu_load[4] : 0 [Wed Feb 6 09:40:58 2019][1697232.824053] .avg_idle : 10000000 [Wed Feb 6 09:40:58 2019][1697232.829463] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:58 2019][1697232.834784] [Wed Feb 6 09:40:58 2019][1697232.836461] cfs_rq[31]:/ [Wed Feb 6 09:40:58 2019][1697232.839172] .exec_clock : 0.000000 [Wed Feb 6 09:40:58 2019][1697232.844572] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:58 2019][1697232.849971] .min_vruntime : 2266950.529806 [Wed Feb 6 09:40:58 2019][1697232.855889] .max_vruntime : 0.000001 [Wed Feb 6 09:40:58 2019][1697232.861290] .spread : 0.000000 [Wed Feb 6 09:40:58 2019][1697232.866689] .spread0 : -6576463.715270 [Wed Feb 6 09:40:58 2019][1697232.872696] .nr_spread_over : 0 [Wed Feb 6 09:40:58 2019][1697232.877487] .nr_running : 0 [Wed Feb 6 09:40:58 2019][1697232.882280] .load : 0 [Wed Feb 6 09:40:58 2019][1697232.887073] .runnable_load_avg : 0 [Wed Feb 6 09:40:58 2019][1697232.891865] .blocked_load_avg : 0 [Wed Feb 6 09:40:58 2019][1697232.896658] .tg_load_avg : 0 [Wed Feb 6 09:40:58 2019][1697232.901450] .tg_load_contrib : 0 [Wed Feb 6 09:40:58 2019][1697232.906243] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:58 2019][1697232.911037] .tg->runnable_avg : 0 [Wed Feb 6 09:40:58 2019][1697232.915831] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:58 2019][1697232.920622] .throttled : 0 [Wed Feb 6 09:40:58 2019][1697232.925415] .throttle_count : 0 [Wed Feb 6 09:40:58 2019][1697232.930207] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:40:58 2019][1697232.935000] .avg->runnable_avg_period : 47785 [Wed Feb 6 09:40:58 2019][1697232.940147] [Wed Feb 6 09:40:58 2019][1697232.941821] rt_rq[31]:/ [Wed Feb 6 09:40:58 2019][1697232.944446] .rt_nr_running : 0 [Wed Feb 6 09:40:58 2019][1697232.949240] .rt_throttled : 0 [Wed Feb 6 09:40:58 2019][1697232.954031] .rt_time : 0.000000 [Wed Feb 6 09:40:58 2019][1697232.959432] .rt_runtime : 963.144286 [Wed Feb 6 09:40:58 2019][1697232.965004] [Wed Feb 6 09:40:58 2019][1697232.966678] runnable tasks: [Wed Feb 6 09:40:58 2019][1697232.969651] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:58 2019][1697232.980596] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:58 2019][1697232.991477] watchdog/31 166 -11.987146 424307 0 0.000000 57917.089210 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.002808] migration/31 167 0.000000 1887 0 0.000000 109211.196383 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.014143] ksoftirqd/31 168 2248002.387667 177464 120 0.000000 6956.152464 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.025481] kworker/31:0H 170 4047.770363 10 100 0.000000 0.036580 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.036833] kworker/31:1 7277 2266887.401463 57196 120 0.000000 10930.150785 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.048208] kworker/31:1H 46420 2266746.898408 248737 100 0.000000 724.926044 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.059560] ptlrpcd_03_04 126481 2266938.532381 5603284 120 0.000000 88683.979775 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.070990] mdt03_002 126634 2074869.984683 7740519 120 0.000000 318136.967046 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.082419] mdt_io03_000 126674 2054250.658420 167972 120 0.000000 7767.125750 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.093851] osp-syn-25-3 126734 2266886.433597 127890 120 0.000000 3644.976333 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.105278] osp-pre-23-3 126755 2266938.549513 214652 120 0.000000 5556.808494 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.116704] osp-syn-34-3 126798 2266938.557147 126610 120 0.000000 3493.901002 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.128131] osp-syn-6-1 126955 2266887.436038 198943 120 0.000000 4931.506814 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.139568] mdt03_010 127390 2075957.444344 7657475 120 0.000000 467135.887986 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.150985] mdt03_012 127456 2074912.283512 8200708 120 0.000000 483857.966115 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.162441] mdt03_013 14718 2232939.240599 7372784 120 0.000000 310890.861067 0.000000 3 / [Wed Feb 6 09:40:58 2019][1697233.173783] mdt_io03_007 40453 2054250.657819 180654 120 0.000000 8207.443854 0.000000 3 / [Wed Feb 6 09:40:59 2019][1697233.185115] ldlm_cn03_004 40888 2266950.529806 3660119 120 0.000000 207593.259037 0.000000 3 / [Wed Feb 6 09:40:59 2019][1697233.196463] mdt03_020 17822 2074869.986015 74158 120 0.000000 4833.906207 0.000000 3 / [Wed Feb 6 09:40:59 2019][1697233.207800] mdt03_029 17903 2232939.237322 127698 120 0.000000 459268.502445 0.000000 3 / [Wed Feb 6 09:40:59 2019][1697233.219160] kworker/31:0 24881 2034393.876037 2 120 0.000000 0.004228 0.000000 3 / [Wed Feb 6 09:40:59 2019][1697233.230540] [Wed Feb 6 09:40:59 2019][1697233.232214] cpu#32, 1996.233 MHz [Wed Feb 6 09:40:59 2019][1697233.235620] .nr_running : 1 [Wed Feb 6 09:40:59 2019][1697233.240416] .load : 1024 [Wed Feb 6 09:40:59 2019][1697233.245475] .nr_switches : 1040341639 [Wed Feb 6 09:40:59 2019][1697233.251050] .nr_load_updates : 206812569 [Wed Feb 6 09:40:59 2019][1697233.256544] .nr_uninterruptible : -110460 [Wed Feb 6 09:40:59 2019][1697233.261867] .next_balance : 5991.886087 [Wed Feb 6 09:40:59 2019][1697233.267535] .curr->pid : 10650 [Wed Feb 6 09:40:59 2019][1697233.272708] .clock : 1697233271.876632 [Wed Feb 6 09:40:59 2019][1697233.278896] .cpu_load[0] : 1024 [Wed Feb 6 09:40:59 2019][1697233.283981] .cpu_load[1] : 1024 [Wed Feb 6 09:40:59 2019][1697233.289035] .cpu_load[2] : 1024 [Wed Feb 6 09:40:59 2019][1697233.294099] .cpu_load[3] : 1024 [Wed Feb 6 09:40:59 2019][1697233.299157] .cpu_load[4] : 1024 [Wed Feb 6 09:40:59 2019][1697233.304223] .avg_idle : 10000000 [Wed Feb 6 09:40:59 2019][1697233.309629] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:59 2019][1697233.314949] [Wed Feb 6 09:40:59 2019][1697233.316626] cfs_rq[32]:/ [Wed Feb 6 09:40:59 2019][1697233.319336] .exec_clock : 0.000000 [Wed Feb 6 09:40:59 2019][1697233.324738] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:59 2019][1697233.330137] .min_vruntime : 10867982.027857 [Wed Feb 6 09:40:59 2019][1697233.336143] .max_vruntime : 0.000001 [Wed Feb 6 09:40:59 2019][1697233.341540] .spread : 0.000000 [Wed Feb 6 09:40:59 2019][1697233.346941] .spread0 : 2024567.782781 [Wed Feb 6 09:40:59 2019][1697233.352860] .nr_spread_over : 0 [Wed Feb 6 09:40:59 2019][1697233.357652] .nr_running : 1 [Wed Feb 6 09:40:59 2019][1697233.362446] .load : 1024 [Wed Feb 6 09:40:59 2019][1697233.367498] .runnable_load_avg : 1023 [Wed Feb 6 09:40:59 2019][1697233.372552] .blocked_load_avg : 0 [Wed Feb 6 09:40:59 2019][1697233.377345] .tg_load_avg : 0 [Wed Feb 6 09:40:59 2019][1697233.382136] .tg_load_contrib : 0 [Wed Feb 6 09:40:59 2019][1697233.386930] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:59 2019][1697233.391721] .tg->runnable_avg : 0 [Wed Feb 6 09:40:59 2019][1697233.396513] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:59 2019][1697233.401306] .throttled : 0 [Wed Feb 6 09:40:59 2019][1697233.406100] .throttle_count : 0 [Wed Feb 6 09:40:59 2019][1697233.410894] .avg->runnable_avg_sum : 46269 [Wed Feb 6 09:40:59 2019][1697233.416033] .avg->runnable_avg_period : 46114 [Wed Feb 6 09:40:59 2019][1697233.421179] [Wed Feb 6 09:40:59 2019][1697233.422854] rt_rq[32]:/ [Wed Feb 6 09:40:59 2019][1697233.425480] .rt_nr_running : 0 [Wed Feb 6 09:40:59 2019][1697233.430271] .rt_throttled : 0 [Wed Feb 6 09:40:59 2019][1697233.435066] .rt_time : 0.000000 [Wed Feb 6 09:40:59 2019][1697233.440465] .rt_runtime : 958.767362 [Wed Feb 6 09:40:59 2019][1697233.446037] [Wed Feb 6 09:40:59 2019][1697233.447710] runnable tasks: [Wed Feb 6 09:40:59 2019][1697233.450683] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:59 2019][1697233.461628] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:59 2019][1697233.472510] watchdog/32 171 -11.989040 424295 0 0.000000 87076.598683 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.483849] migration/32 172 0.000000 37785 0 0.000000 185648.856269 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.495185] ksoftirqd/32 173 10865963.319595 2386383 120 0.000000 27454.433440 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.506521] kworker/32:0H 175 5445.533825 9 100 0.000000 0.031420 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.517893] kworker/32:1H 16048 10866558.053426 2053027 100 0.000000 11114.935881 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.529240] reader#4 18236 14539.097530 1 120 0.000000 0.011121 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.540573] agetty 18468 15820.036127 8 120 0.000000 10.067843 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.551919] clio 126374 2232471.177591 2 100 0.000000 0.011231 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.563366] mdt_attr00_000 126643 2234125.211718 2 120 0.000000 0.044664 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.574789] mdt_attr00_001 126644 2234137.233733 2 120 0.000000 0.022021 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.586214] mdt_out00_000 126651 10588586.111154 7 120 0.000000 0.098027 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.597636] mdt_seqs_0000 126659 2234317.485207 2 120 0.000000 0.047680 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.609057] mdt_seqs_0001 126660 2234329.503045 2 120 0.000000 0.017844 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.620484] osp_up1-3 126701 2234622.872593 2 120 0.000000 0.019747 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.631940] mdt_io00_005 127230 6949202.546947 1725584 120 0.000000 78811.879318 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.643361] mdt_io00_007 127251 6949202.546826 897826 120 0.000000 31940.262333 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.654784] mdt_io00_008 127263 6949202.547708 2235882 120 0.000000 96808.002087 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.666213] mdt_io00_014 127701 6949202.551686 1066295 120 0.000000 37840.294276 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.677630] mdt_io00_016 127709 6949202.547037 733972 120 0.000000 26296.138231 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.689079] mdt_io00_027 1418 6949202.548139 1568237 120 0.000000 71961.291536 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.700417] mdt00_025 2937 10761962.403144 69667491 120 0.000000 2133222.499720 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.711751] mdt00_026 2976 10866422.343204 59172049 120 0.000000 1596129.191609 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.723097] kworker/32:1 7231 10866807.488193 33381 120 0.000000 14282.493221 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.734434] R ldlm_bl_09 10650 10868394.030604 1324035 120 0.000000 1463474.282065 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.745781] mdt00_035 17897 10865980.612055 56371 120 0.000000 551043.174937 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.757116] mdt00_041 17916 10761962.403023 99082 120 0.000000 304989.525073 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.768470] kworker/32:0 23806 10435372.307297 10 120 0.000000 0.083768 0.000000 0 / [Wed Feb 6 09:40:59 2019][1697233.779855] [Wed Feb 6 09:40:59 2019][1697233.781533] cpu#33, 1996.233 MHz [Wed Feb 6 09:40:59 2019][1697233.784940] .nr_running : 0 [Wed Feb 6 09:40:59 2019][1697233.789741] .load : 0 [Wed Feb 6 09:40:59 2019][1697233.794542] .nr_switches : 700272266 [Wed Feb 6 09:40:59 2019][1697233.800039] .nr_load_updates : 137679753 [Wed Feb 6 09:40:59 2019][1697233.805531] .nr_uninterruptible : 44502 [Wed Feb 6 09:40:59 2019][1697233.810681] .next_balance : 5991.886584 [Wed Feb 6 09:40:59 2019][1697233.816348] .curr->pid : 0 [Wed Feb 6 09:40:59 2019][1697233.821150] .clock : 1697233820.946585 [Wed Feb 6 09:40:59 2019][1697233.827337] .cpu_load[0] : 0 [Wed Feb 6 09:40:59 2019][1697233.832142] .cpu_load[1] : 0 [Wed Feb 6 09:40:59 2019][1697233.836941] .cpu_load[2] : 0 [Wed Feb 6 09:40:59 2019][1697233.841745] .cpu_load[3] : 0 [Wed Feb 6 09:40:59 2019][1697233.846541] .cpu_load[4] : 0 [Wed Feb 6 09:40:59 2019][1697233.851337] .avg_idle : 1595480 [Wed Feb 6 09:40:59 2019][1697233.856656] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:59 2019][1697233.861977] [Wed Feb 6 09:40:59 2019][1697233.863646] cfs_rq[33]:/ [Wed Feb 6 09:40:59 2019][1697233.866359] .exec_clock : 0.000000 [Wed Feb 6 09:40:59 2019][1697233.871758] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:59 2019][1697233.877157] .min_vruntime : 8212964.143170 [Wed Feb 6 09:40:59 2019][1697233.883076] .max_vruntime : 0.000001 [Wed Feb 6 09:40:59 2019][1697233.888476] .spread : 0.000000 [Wed Feb 6 09:40:59 2019][1697233.893876] .spread0 : -630450.101906 [Wed Feb 6 09:40:59 2019][1697233.899794] .nr_spread_over : 0 [Wed Feb 6 09:40:59 2019][1697233.904587] .nr_running : 0 [Wed Feb 6 09:40:59 2019][1697233.909381] .load : 0 [Wed Feb 6 09:40:59 2019][1697233.914174] .runnable_load_avg : 0 [Wed Feb 6 09:40:59 2019][1697233.918967] .blocked_load_avg : 3 [Wed Feb 6 09:40:59 2019][1697233.923757] .tg_load_avg : 0 [Wed Feb 6 09:40:59 2019][1697233.928552] .tg_load_contrib : 0 [Wed Feb 6 09:40:59 2019][1697233.933344] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:59 2019][1697233.938136] .tg->runnable_avg : 0 [Wed Feb 6 09:40:59 2019][1697233.942929] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:59 2019][1697233.947721] .throttled : 0 [Wed Feb 6 09:40:59 2019][1697233.952514] .throttle_count : 0 [Wed Feb 6 09:40:59 2019][1697233.957308] .avg->runnable_avg_sum : 88 [Wed Feb 6 09:40:59 2019][1697233.962186] .avg->runnable_avg_period : 46540 [Wed Feb 6 09:40:59 2019][1697233.967332] [Wed Feb 6 09:40:59 2019][1697233.969008] rt_rq[33]:/ [Wed Feb 6 09:40:59 2019][1697233.971634] .rt_nr_running : 0 [Wed Feb 6 09:40:59 2019][1697233.976425] .rt_throttled : 0 [Wed Feb 6 09:40:59 2019][1697233.981220] .rt_time : 0.000000 [Wed Feb 6 09:40:59 2019][1697233.986617] .rt_runtime : 979.166667 [Wed Feb 6 09:40:59 2019][1697233.992191] [Wed Feb 6 09:40:59 2019][1697233.993864] runnable tasks: [Wed Feb 6 09:40:59 2019][1697233.996835] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:59 2019][1697234.007782] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:59 2019][1697234.018667] watchdog/33 176 -11.986083 424290 0 0.000000 125403.057227 0.000000 1 / [Wed Feb 6 09:40:59 2019][1697234.030004] migration/33 177 0.000000 47976 0 0.000000 155264.180510 0.000000 1 / [Wed Feb 6 09:40:59 2019][1697234.041339] ksoftirqd/33 178 8212872.365858 2872066 120 0.000000 32677.301949 0.000000 1 / [Wed Feb 6 09:40:59 2019][1697234.052677] kworker/33:0H 180 15672.140826 10 100 0.000000 0.033963 0.000000 1 / [Wed Feb 6 09:40:59 2019][1697234.064023] netns 256 35.353853 2 100 0.000000 0.010230 0.000000 1 / [Wed Feb 6 09:40:59 2019][1697234.075358] kintegrityd 259 71.381000 2 100 0.000000 0.009037 0.000000 1 / [Wed Feb 6 09:40:59 2019][1697234.086692] bioset 260 83.389245 2 100 0.000000 0.008797 0.000000 1 / [Wed Feb 6 09:40:59 2019][1697234.098028] bioset 261 95.397400 2 100 0.000000 0.008827 0.000000 1 / [Wed Feb 6 09:40:59 2019][1697234.109365] bioset 262 107.405715 2 100 0.000000 0.008916 0.000000 1 / [Wed Feb 6 09:40:59 2019][1697234.120702] kblockd 263 119.414081 2 100 0.000000 0.008957 0.000000 1 / [Wed Feb 6 09:40:59 2019][1697234.132038] md 264 131.422922 2 100 0.000000 0.009659 0.000000 1 / [Wed Feb 6 09:40:59 2019][1697234.143374] edac-poller 265 143.431349 2 100 0.000000 0.009128 0.000000 1 / [Wed Feb 6 09:40:59 2019][1697234.154710] watchdogd 266 155.439735 2 100 0.000000 0.009007 0.000000 1 / [Wed Feb 6 09:40:59 2019][1697234.166046] irq/27-AMD-Vi 271 0.000000 2 49 0.000000 0.017273 0.000000 1 / [Wed Feb 6 09:40:59 2019][1697234.177382] kswapd1 279 456.037119 3 120 0.000000 0.024105 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.188718] ksmd 282 311.705677 2 125 0.000000 0.012493 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.200054] crypto 284 335.801126 2 100 0.000000 0.010801 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.211390] kthrotld 292 432.002762 2 100 0.000000 0.010179 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.222726] kmpath_rdacd 297 480.040884 2 100 0.000000 0.016460 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.234063] kaluad 298 492.055880 2 100 0.000000 0.015300 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.245432] kdmwork-253:4 18673 8004283.563534 415 120 0.000000 3.789187 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.256772] kworker/33:1H 18959 8212872.237355 1912915 100 0.000000 8112.465144 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.268124] ptlrpcd_01_09 126462 8212941.047183 3977960 120 0.000000 56568.222122 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.279554] ldiskfs-rsv-con 126578 2136416.049055 2 100 0.000000 0.011180 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.290977] ldiskfs-rsv-con 126581 2136428.058757 2 100 0.000000 0.010560 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.302426] osp-syn-28-3 126794 8212941.081487 127229 120 0.000000 3565.728896 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.313868] ldlm_cn01_003 127203 8212899.206078 11670768 120 0.000000 406285.136861 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.325290] mdt01_007 127206 8091490.863487 32212201 120 0.000000 1305456.347281 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.336713] mdt01_008 127208 8092126.830236 36112110 120 0.000000 1587781.463760 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.348136] mdt_rdpg01_002 127211 8187091.028644 5976346 120 0.000000 248312.507244 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.359563] ldlm_cn01_005 127252 8212891.551432 11200408 120 0.000000 417052.630642 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.370990] mdt_io01_019 127698 8085041.947726 2878819 120 0.000000 111176.119821 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.382418] mdt_rdpg01_006 128924 8187091.033682 10390253 120 0.000000 451085.187276 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.393836] mdt01_017 129231 8092126.865602 29101974 120 0.000000 1160042.432501 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.405275] mdt_io01_033 131047 8060473.649569 1676158 120 0.000000 58084.656887 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.416702] ldlm_cn01_011 1442 8212899.206238 6630815 120 0.000000 321080.177333 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.428034] ldlm_cn01_013 1444 8212891.551442 8111671 120 0.000000 338983.578475 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.439378] kworker/33:1 39444 8212952.594521 90270 120 0.000000 15294.179128 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.450730] mdt01_038 17823 8088123.185043 145924 120 0.000000 918967.004621 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.462071] mdt01_040 17832 8091490.862264 209377 120 0.000000 167103.107000 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.473405] mdt01_041 17851 8205464.480385 223113 120 0.000000 248000.638908 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.484751] mdt01_046 17976 8205464.461168 34610 120 0.000000 17483.116871 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.496087] mdt01_048 17991 8212880.769715 334337 120 0.000000 317755.178080 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.507421] mdt01_051 17995 8091490.866262 312726 120 0.000000 784530.204624 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.518765] mdt01_056 19083 8087749.058837 54672 120 0.000000 160883.170469 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.530108] ldlm_cn01_020 23721 8212922.656549 49614 120 0.000000 103441.054876 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.541439] ldlm_cn01_021 23723 8212891.551222 54574 120 0.000000 135566.946980 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.552777] ldlm_cn01_023 23727 8212899.206007 48299 120 0.000000 116783.259119 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.564113] ldlm_cn01_028 23736 8212899.206880 56705 120 0.000000 133621.708064 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.575448] ldlm_cn01_029 23737 8212922.657291 48934 120 0.000000 132023.564240 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.586784] ldlm_cn01_030 23738 8212899.206378 46452 120 0.000000 120534.511633 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.598132] kworker/33:2 25047 8186578.714093 8 120 0.000000 0.061654 0.000000 1 / [Wed Feb 6 09:41:00 2019][1697234.609510] [Wed Feb 6 09:41:00 2019][1697234.611186] cpu#34, 1996.233 MHz [Wed Feb 6 09:41:00 2019][1697234.614602] .nr_running : 0 [Wed Feb 6 09:41:00 2019][1697234.619403] .load : 0 [Wed Feb 6 09:41:00 2019][1697234.624204] .nr_switches : 1719988761 [Wed Feb 6 09:41:00 2019][1697234.629786] .nr_load_updates : 157340959 [Wed Feb 6 09:41:00 2019][1697234.635282] .nr_uninterruptible : -1053 [Wed Feb 6 09:41:00 2019][1697234.640427] .next_balance : 5991.887412 [Wed Feb 6 09:41:00 2019][1697234.646091] .curr->pid : 0 [Wed Feb 6 09:41:00 2019][1697234.650891] .clock : 1697234649.892294 [Wed Feb 6 09:41:00 2019][1697234.657080] .cpu_load[0] : 0 [Wed Feb 6 09:41:00 2019][1697234.661879] .cpu_load[1] : 0 [Wed Feb 6 09:41:00 2019][1697234.666681] .cpu_load[2] : 0 [Wed Feb 6 09:41:00 2019][1697234.671480] .cpu_load[3] : 0 [Wed Feb 6 09:41:00 2019][1697234.676277] .cpu_load[4] : 0 [Wed Feb 6 09:41:00 2019][1697234.681075] .avg_idle : 10000000 [Wed Feb 6 09:41:00 2019][1697234.686476] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:00 2019][1697234.691798] [Wed Feb 6 09:41:00 2019][1697234.693474] cfs_rq[34]:/ [Wed Feb 6 09:41:00 2019][1697234.696184] .exec_clock : 0.000000 [Wed Feb 6 09:41:00 2019][1697234.701584] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:00 2019][1697234.706983] .min_vruntime : 10834653.577175 [Wed Feb 6 09:41:00 2019][1697234.712991] .max_vruntime : 0.000001 [Wed Feb 6 09:41:00 2019][1697234.718388] .spread : 0.000000 [Wed Feb 6 09:41:00 2019][1697234.723789] .spread0 : 1991239.332099 [Wed Feb 6 09:41:00 2019][1697234.729709] .nr_spread_over : 0 [Wed Feb 6 09:41:00 2019][1697234.734500] .nr_running : 0 [Wed Feb 6 09:41:00 2019][1697234.739293] .load : 0 [Wed Feb 6 09:41:00 2019][1697234.744085] .runnable_load_avg : 0 [Wed Feb 6 09:41:00 2019][1697234.748880] .blocked_load_avg : 0 [Wed Feb 6 09:41:00 2019][1697234.753673] .tg_load_avg : 0 [Wed Feb 6 09:41:00 2019][1697234.758465] .tg_load_contrib : 0 [Wed Feb 6 09:41:00 2019][1697234.763258] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:00 2019][1697234.768050] .tg->runnable_avg : 0 [Wed Feb 6 09:41:00 2019][1697234.772842] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:00 2019][1697234.777635] .throttled : 0 [Wed Feb 6 09:41:00 2019][1697234.782428] .throttle_count : 0 [Wed Feb 6 09:41:00 2019][1697234.787222] .avg->runnable_avg_sum : 92 [Wed Feb 6 09:41:00 2019][1697234.792100] .avg->runnable_avg_period : 47392 [Wed Feb 6 09:41:00 2019][1697234.797247] [Wed Feb 6 09:41:00 2019][1697234.798920] rt_rq[34]:/ [Wed Feb 6 09:41:00 2019][1697234.801547] .rt_nr_running : 0 [Wed Feb 6 09:41:00 2019][1697234.806341] .rt_throttled : 0 [Wed Feb 6 09:41:00 2019][1697234.811133] .rt_time : 0.000000 [Wed Feb 6 09:41:00 2019][1697234.816531] .rt_runtime : 964.415724 [Wed Feb 6 09:41:00 2019][1697234.822105] [Wed Feb 6 09:41:00 2019][1697234.823776] runnable tasks: [Wed Feb 6 09:41:00 2019][1697234.826749] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:00 2019][1697234.837696] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:00 2019][1697234.848581] watchdog/34 181 -11.987226 424297 0 0.000000 88037.083105 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697234.859917] migration/34 182 0.000000 22746 0 0.000000 193266.533427 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697234.871252] ksoftirqd/34 183 10834641.420782 781472 120 0.000000 30000.809809 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697234.882589] kworker/34:0H 185 15147.604104 10 100 0.000000 0.027731 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697234.893937] kpsmoused 302 95.049004 2 100 0.000000 0.019928 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697234.905270] ipv6_addrconf 304 107.067560 2 100 0.000000 0.019226 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697234.916606] deferwq 318 179.238372 2 100 0.000000 0.011902 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697234.927944] kworker/34:1 325 10834613.899663 357107 120 0.000000 19720.659517 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697234.939305] xprtiod 16732 10433.991392 2 100 0.000000 0.009558 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697234.950646] mlx4 16966 10761.816939 2 100 0.000000 0.013406 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697234.961978] mlx4_ib 16981 10818.283593 2 100 0.000000 0.010821 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697234.973313] mlx4_ib_mcg 16982 10830.292180 2 100 0.000000 0.009297 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697234.984655] writer#1 18227 10834609.121291 24939600 120 0.000000 227184.928785 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697234.995987] nfsv4.1-svc 18374 14733.893820 2 120 0.000000 0.009438 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697235.007324] kworker/34:1H 18978 10834491.822024 285679 100 0.000000 1641.265329 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697235.018678] ptlrpcd_02_10 126475 10834641.708554 6886880 120 0.000000 83246.530948 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697235.030105] lc_watchdogd 126586 10828875.837085 7592 120 0.000000 18845.004989 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697235.041539] mdt_out02_001 126656 10834640.344703 17686332 120 0.000000 286353.554275 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697235.052973] osp-syn-38-3 126772 10834641.596892 126428 120 0.000000 3526.871087 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697235.064403] osp-syn-18-1 126921 10834641.648878 198639 120 0.000000 4754.224382 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697235.075858] mdt02_016 129250 10829955.524731 23432783 120 0.000000 1614554.955440 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697235.087287] mdt02_020 129278 10728011.338193 26714516 120 0.000000 1615559.312158 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697235.098717] mdt_out02_006 2325 10834640.350574 11764906 120 0.000000 192088.167671 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697235.110052] ldlm_cb02_004 18369 10708373.365860 6882261 120 0.000000 90477.875476 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697235.121407] mdt02_035 17906 10727493.923815 13027 120 0.000000 158943.673962 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697235.132746] mdt02_042 17999 10727426.370728 188319 120 0.000000 639516.894773 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697235.144085] mdt02_046 18224 10727384.000743 22498 120 0.000000 161815.858261 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697235.155442] kworker/34:0 24916 10727229.831712 6 120 0.000000 0.049504 0.000000 2 / [Wed Feb 6 09:41:00 2019][1697235.166790] [Wed Feb 6 09:41:00 2019][1697235.168468] cpu#35, 1996.233 MHz [Wed Feb 6 09:41:00 2019][1697235.171876] .nr_running : 0 [Wed Feb 6 09:41:00 2019][1697235.176678] .load : 0 [Wed Feb 6 09:41:00 2019][1697235.181476] .nr_switches : 90776325 [Wed Feb 6 09:41:00 2019][1697235.186880] .nr_load_updates : 52722332 [Wed Feb 6 09:41:01 2019][1697235.192286] .nr_uninterruptible : 1120 [Wed Feb 6 09:41:01 2019][1697235.197349] .next_balance : 5991.887970 [Wed Feb 6 09:41:01 2019][1697235.203014] .curr->pid : 0 [Wed Feb 6 09:41:01 2019][1697235.207809] .clock : 1697235206.897293 [Wed Feb 6 09:41:01 2019][1697235.213995] .cpu_load[0] : 0 [Wed Feb 6 09:41:01 2019][1697235.218790] .cpu_load[1] : 0 [Wed Feb 6 09:41:01 2019][1697235.223591] .cpu_load[2] : 0 [Wed Feb 6 09:41:01 2019][1697235.228393] .cpu_load[3] : 0 [Wed Feb 6 09:41:01 2019][1697235.233193] .cpu_load[4] : 0 [Wed Feb 6 09:41:01 2019][1697235.237993] .avg_idle : 10000000 [Wed Feb 6 09:41:01 2019][1697235.243392] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:01 2019][1697235.248706] [Wed Feb 6 09:41:01 2019][1697235.250384] cfs_rq[35]:/ [Wed Feb 6 09:41:01 2019][1697235.253095] .exec_clock : 0.000000 [Wed Feb 6 09:41:01 2019][1697235.258496] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:01 2019][1697235.263893] .min_vruntime : 4148808.498596 [Wed Feb 6 09:41:01 2019][1697235.269813] .max_vruntime : 0.000001 [Wed Feb 6 09:41:01 2019][1697235.275213] .spread : 0.000000 [Wed Feb 6 09:41:01 2019][1697235.280612] .spread0 : -4694605.746480 [Wed Feb 6 09:41:01 2019][1697235.286618] .nr_spread_over : 0 [Wed Feb 6 09:41:01 2019][1697235.291411] .nr_running : 0 [Wed Feb 6 09:41:01 2019][1697235.296204] .load : 0 [Wed Feb 6 09:41:01 2019][1697235.300995] .runnable_load_avg : 0 [Wed Feb 6 09:41:01 2019][1697235.305789] .blocked_load_avg : 0 [Wed Feb 6 09:41:01 2019][1697235.310582] .tg_load_avg : 0 [Wed Feb 6 09:41:01 2019][1697235.315374] .tg_load_contrib : 0 [Wed Feb 6 09:41:01 2019][1697235.320168] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:01 2019][1697235.324960] .tg->runnable_avg : 0 [Wed Feb 6 09:41:01 2019][1697235.329752] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:01 2019][1697235.334545] .throttled : 0 [Wed Feb 6 09:41:01 2019][1697235.339338] .throttle_count : 0 [Wed Feb 6 09:41:01 2019][1697235.344130] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:41:01 2019][1697235.348924] .avg->runnable_avg_period : 46956 [Wed Feb 6 09:41:01 2019][1697235.354072] [Wed Feb 6 09:41:01 2019][1697235.355745] rt_rq[35]:/ [Wed Feb 6 09:41:01 2019][1697235.358370] .rt_nr_running : 0 [Wed Feb 6 09:41:01 2019][1697235.363163] .rt_throttled : 0 [Wed Feb 6 09:41:01 2019][1697235.367955] .rt_time : 0.007114 [Wed Feb 6 09:41:01 2019][1697235.373355] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:01 2019][1697235.379014] [Wed Feb 6 09:41:01 2019][1697235.380687] runnable tasks: [Wed Feb 6 09:41:01 2019][1697235.383660] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:01 2019][1697235.394605] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:01 2019][1697235.405490] watchdog/35 186 -11.986815 424308 0 0.000000 61705.150217 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.416827] migration/35 187 0.000000 43736 0 0.000000 102493.699173 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.428163] ksoftirqd/35 188 4124093.659219 183356 120 0.000000 3478.064932 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.439500] kworker/35:0H 190 2751.922180 9 100 0.000000 0.024416 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.450850] scsi_tmf_0 2266 865.150467 2 100 0.000000 0.008016 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.462183] mlx5_page_alloc 7370 2292.087419 2 100 0.000000 0.011782 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.473519] kworker/u265:0 7440 2513.878064 4 100 0.000000 0.013946 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.484852] ib-comp-wq 7468 2525.878034 2 100 0.000000 0.009979 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.496190] ib-comp-unb-wq 7469 2537.886526 2 100 0.000000 0.009608 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.507526] ib_mcast 7470 2549.894561 2 100 0.000000 0.008646 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.518860] ib_nl_sa_wq 7471 2561.903307 2 100 0.000000 0.009317 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.530198] mlx5_ib_sigerr_ 7487 2592.109045 2 100 0.000000 0.012344 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.541534] ib_mad1 7492 2604.118335 2 100 0.000000 0.010059 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.552870] to_fifo 7493 2616.126631 2 100 0.000000 0.008937 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.564204] mkey_cache 7496 2628.135588 2 100 0.000000 0.009578 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.575543] kworker/35:1H 7575 4148418.930619 190654 100 0.000000 1467.394484 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.586893] auditd 16728 4131770.922295 13528 116 0.000000 338.824014 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.598235] JS Sour~ Thread 16806 11557.869895 5 120 0.000000 0.103244 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.609568] gssproxy 16815 11581.862877 1 120 0.000000 0.027482 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.620903] gssproxy 16816 11593.869313 1 120 0.000000 0.006442 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.632239] gssproxy 16818 11605.875529 1 120 0.000000 0.006222 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.643575] gssproxy 16819 11617.881705 1 120 0.000000 0.006182 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.654912] mlx5i 17081 3964917.567513 3 100 0.000000 0.059481 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.666248] ipoib_wq 17093 3964917.567653 4 100 0.000000 206.648022 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.677593] slapd 18606 16041.869136 7 120 0.000000 8.438084 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.688954] ptlrpcd_03_11 126488 4148796.732217 5590326 120 0.000000 80975.936009 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.700403] osp-pre-30-3 126797 4148796.513674 214703 120 0.000000 6716.098890 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.711835] osp-pre-43-1 126918 4148796.511290 213922 120 0.000000 3052.332148 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.723259] osp-syn-25-1 126923 4148796.525738 198136 120 0.000000 4709.234289 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.734692] mdt03_004 127132 4019289.614146 6851171 120 0.000000 731288.341663 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.746124] mdt03_011 127411 4019334.198360 7574840 120 0.000000 463488.240481 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.757582] ldlm_cn03_007 51807 4148739.468121 1316706 120 0.000000 161247.440964 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.768917] kworker/35:1 120937 4148796.505890 27555 120 0.000000 11791.767621 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.780344] ldlm_cn03_009 12924 4148758.455017 1442242 120 0.000000 128677.555743 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.791697] mdt03_035 17959 4021145.698334 132494 120 0.000000 6401.574279 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.803037] mdt03_038 17969 4148796.835781 88565 120 0.000000 160737.666975 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.814373] mdt03_040 17982 4019334.201907 56881 120 0.000000 5883.562256 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.825710] mdt03_042 18164 4019233.485117 21205 120 0.000000 1904.526352 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.837047] mdt03_048 18261 4148796.500740 135155 120 0.000000 7897.268286 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.848395] kworker/35:2 24048 3790733.412267 2 120 0.000000 0.004519 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.859728] ldlm_cn03_021 24584 4148739.466939 2697 120 0.000000 28445.528735 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.871064] ldlm_cn03_024 24587 4148739.467460 5998 120 0.000000 80131.350252 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.882398] ldlm_cn03_025 24588 4148806.520699 6499 120 0.000000 30718.728968 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.893736] ldlm_cn03_029 24611 4148758.114529 11806 120 0.000000 80232.682703 0.000000 3 / [Wed Feb 6 09:41:01 2019][1697235.905092] [Wed Feb 6 09:41:01 2019][1697235.906767] cpu#36, 1996.233 MHz [Wed Feb 6 09:41:01 2019][1697235.910184] .nr_running : 1 [Wed Feb 6 09:41:01 2019][1697235.914981] .load : 1024 [Wed Feb 6 09:41:01 2019][1697235.920037] .nr_switches : 1708849516 [Wed Feb 6 09:41:01 2019][1697235.925615] .nr_load_updates : 280510200 [Wed Feb 6 09:41:01 2019][1697235.931106] .nr_uninterruptible : -98481 [Wed Feb 6 09:41:01 2019][1697235.936337] .next_balance : 5991.888718 [Wed Feb 6 09:41:01 2019][1697235.942007] .curr->pid : 10649 [Wed Feb 6 09:41:01 2019][1697235.947153] .clock : 1697235946.898956 [Wed Feb 6 09:41:01 2019][1697235.953336] .cpu_load[0] : 1024 [Wed Feb 6 09:41:01 2019][1697235.958395] .cpu_load[1] : 1024 [Wed Feb 6 09:41:01 2019][1697235.963456] .cpu_load[2] : 1024 [Wed Feb 6 09:41:01 2019][1697235.968517] .cpu_load[3] : 1024 [Wed Feb 6 09:41:01 2019][1697235.973581] .cpu_load[4] : 1024 [Wed Feb 6 09:41:01 2019][1697235.978639] .avg_idle : 10000000 [Wed Feb 6 09:41:01 2019][1697235.984050] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:01 2019][1697235.989366] [Wed Feb 6 09:41:01 2019][1697235.991038] cfs_rq[36]:/ [Wed Feb 6 09:41:01 2019][1697235.993749] .exec_clock : 0.000000 [Wed Feb 6 09:41:01 2019][1697235.999150] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:01 2019][1697236.004547] .min_vruntime : 10909698.025597 [Wed Feb 6 09:41:01 2019][1697236.010553] .max_vruntime : 0.000001 [Wed Feb 6 09:41:01 2019][1697236.015955] .spread : 0.000000 [Wed Feb 6 09:41:01 2019][1697236.021354] .spread0 : 2066282.069667 [Wed Feb 6 09:41:01 2019][1697236.027271] .nr_spread_over : 0 [Wed Feb 6 09:41:01 2019][1697236.032065] .nr_running : 1 [Wed Feb 6 09:41:01 2019][1697236.036857] .load : 1024 [Wed Feb 6 09:41:01 2019][1697236.041911] .runnable_load_avg : 1023 [Wed Feb 6 09:41:01 2019][1697236.046962] .blocked_load_avg : 0 [Wed Feb 6 09:41:01 2019][1697236.051756] .tg_load_avg : 0 [Wed Feb 6 09:41:01 2019][1697236.056547] .tg_load_contrib : 0 [Wed Feb 6 09:41:01 2019][1697236.061342] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:01 2019][1697236.066134] .tg->runnable_avg : 0 [Wed Feb 6 09:41:01 2019][1697236.070927] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:01 2019][1697236.075719] .throttled : 0 [Wed Feb 6 09:41:01 2019][1697236.080512] .throttle_count : 0 [Wed Feb 6 09:41:01 2019][1697236.085305] .avg->runnable_avg_sum : 46453 [Wed Feb 6 09:41:01 2019][1697236.090444] .avg->runnable_avg_period : 46284 [Wed Feb 6 09:41:01 2019][1697236.095594] [Wed Feb 6 09:41:01 2019][1697236.097265] rt_rq[36]:/ [Wed Feb 6 09:41:01 2019][1697236.099890] .rt_nr_running : 0 [Wed Feb 6 09:41:01 2019][1697236.104685] .rt_throttled : 0 [Wed Feb 6 09:41:01 2019][1697236.109475] .rt_time : 0.000000 [Wed Feb 6 09:41:01 2019][1697236.114875] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:01 2019][1697236.120535] [Wed Feb 6 09:41:01 2019][1697236.122208] runnable tasks: [Wed Feb 6 09:41:01 2019][1697236.125181] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:01 2019][1697236.136127] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:01 2019][1697236.147011] watchdog/36 191 -11.985853 424298 0 0.000000 73818.371463 0.000000 0 / [Wed Feb 6 09:41:01 2019][1697236.158348] migration/36 192 0.000000 21238 0 0.000000 190643.734837 0.000000 0 / [Wed Feb 6 09:41:01 2019][1697236.169684] ksoftirqd/36 193 10895036.962784 1560260 120 0.000000 24675.040350 0.000000 0 / [Wed Feb 6 09:41:01 2019][1697236.181020] kworker/36:0H 195 16089.817056 10 100 0.000000 0.030299 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.192373] scsi_tmf_2 7310 1733.735734 2 100 0.000000 0.009999 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.203710] mlx5_fc 7450 1866.588197 2 100 0.000000 0.012183 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.215056] ccp-5-q2 10003 3153.849442 2 120 0.000000 0.010099 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.226390] ccp-5-q3 10004 3165.857733 2 120 0.000000 0.008297 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.237727] ccp-5-q4 10005 3177.867846 2 120 0.000000 0.010119 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.249063] ccp-6-q0 10006 3189.876557 2 120 0.000000 0.008717 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.260399] ccp-6-q1 10007 3201.885066 2 120 0.000000 0.008515 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.271736] ccp-6-q2 10008 3213.893215 2 120 0.000000 0.008155 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.283072] ccp-6-q3 10009 3225.901554 2 120 0.000000 0.008345 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.294409] ccp-6-q4 10010 3237.909533 2 120 0.000000 0.007985 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.305747] ccp-8-q0 10014 3249.918284 2 120 0.000000 0.008757 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.317080] ccp-8-q1 10015 3261.926955 2 120 0.000000 0.008677 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.328416] ccp-8-q2 10016 3273.934994 2 120 0.000000 0.008045 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.339751] ccp-8-q3 10017 3285.943363 2 120 0.000000 0.008375 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.351088] ccp-8-q4 10018 3297.951102 2 120 0.000000 0.007745 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.362435] ipoib_flush 17079 12301.713410 2 100 0.000000 0.012473 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.373777] kworker/36:1H 18810 10895036.940858 1663354 100 0.000000 10031.730334 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.385121] kworker/36:1 124458 10895041.038656 351934 120 0.000000 16452.974625 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.396578] mdt_io00_001 126666 9599562.417885 1405367 120 0.000000 49608.537782 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.408043] mdt00_009 127250 10764052.822583 78294076 120 0.000000 2475359.511702 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.419461] mdt_io00_009 127266 10740598.762546 2030889 120 0.000000 94609.073156 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.430891] mdt00_010 127970 10764052.822081 60574157 120 0.000000 2353111.612182 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.442317] mdt00_012 128085 10764052.825247 85102212 120 0.000000 2496894.036470 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.453746] mdt00_015 129251 10764052.824987 75404157 120 0.000000 2370421.756224 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.465169] mdt00_017 129258 10890247.745237 79567881 120 0.000000 2465276.574033 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.476593] mdt_io00_020 129277 10759161.759114 1582838 120 0.000000 64786.025568 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.488042] R ldlm_bl_08 10649 10910188.029770 1325166 120 0.000000 1666595.728319 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.499388] mdt00_029 17875 10764052.826702 52537 120 0.000000 156891.024807 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.510754] kworker/36:2 24836 10681562.943636 2 120 0.000000 0.010921 0.000000 0 / [Wed Feb 6 09:41:02 2019][1697236.522113] [Wed Feb 6 09:41:02 2019][1697236.523789] cpu#37, 1996.233 MHz [Wed Feb 6 09:41:02 2019][1697236.527205] .nr_running : 0 [Wed Feb 6 09:41:02 2019][1697236.532008] .load : 0 [Wed Feb 6 09:41:02 2019][1697236.536805] .nr_switches : 1022927623 [Wed Feb 6 09:41:02 2019][1697236.542382] .nr_load_updates : 139634081 [Wed Feb 6 09:41:02 2019][1697236.547875] .nr_uninterruptible : 21150 [Wed Feb 6 09:41:02 2019][1697236.553023] .next_balance : 5991.889325 [Wed Feb 6 09:41:02 2019][1697236.558690] .curr->pid : 0 [Wed Feb 6 09:41:02 2019][1697236.563493] .clock : 1697236562.906637 [Wed Feb 6 09:41:02 2019][1697236.569680] .cpu_load[0] : 0 [Wed Feb 6 09:41:02 2019][1697236.574482] .cpu_load[1] : 0 [Wed Feb 6 09:41:02 2019][1697236.579283] .cpu_load[2] : 0 [Wed Feb 6 09:41:02 2019][1697236.584085] .cpu_load[3] : 0 [Wed Feb 6 09:41:02 2019][1697236.588887] .cpu_load[4] : 0 [Wed Feb 6 09:41:02 2019][1697236.593704] .avg_idle : 10000000 [Wed Feb 6 09:41:02 2019][1697236.599110] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:02 2019][1697236.604427] [Wed Feb 6 09:41:02 2019][1697236.606101] cfs_rq[37]:/ [Wed Feb 6 09:41:02 2019][1697236.608812] .exec_clock : 0.000000 [Wed Feb 6 09:41:02 2019][1697236.614213] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:02 2019][1697236.619612] .min_vruntime : 8904852.214059 [Wed Feb 6 09:41:02 2019][1697236.625532] .max_vruntime : 0.000001 [Wed Feb 6 09:41:02 2019][1697236.630932] .spread : 0.000000 [Wed Feb 6 09:41:02 2019][1697236.636331] .spread0 : 61433.931659 [Wed Feb 6 09:41:02 2019][1697236.642077] .nr_spread_over : 0 [Wed Feb 6 09:41:02 2019][1697236.646870] .nr_running : 0 [Wed Feb 6 09:41:02 2019][1697236.651660] .load : 0 [Wed Feb 6 09:41:02 2019][1697236.656453] .runnable_load_avg : 0 [Wed Feb 6 09:41:02 2019][1697236.661246] .blocked_load_avg : 0 [Wed Feb 6 09:41:02 2019][1697236.666041] .tg_load_avg : 0 [Wed Feb 6 09:41:02 2019][1697236.670834] .tg_load_contrib : 0 [Wed Feb 6 09:41:02 2019][1697236.675626] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:02 2019][1697236.680418] .tg->runnable_avg : 0 [Wed Feb 6 09:41:02 2019][1697236.685211] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:02 2019][1697236.690005] .throttled : 0 [Wed Feb 6 09:41:02 2019][1697236.694797] .throttle_count : 0 [Wed Feb 6 09:41:02 2019][1697236.699589] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:41:02 2019][1697236.704382] .avg->runnable_avg_period : 46816 [Wed Feb 6 09:41:02 2019][1697236.709529] [Wed Feb 6 09:41:02 2019][1697236.711203] rt_rq[37]:/ [Wed Feb 6 09:41:02 2019][1697236.713829] .rt_nr_running : 0 [Wed Feb 6 09:41:02 2019][1697236.718622] .rt_throttled : 0 [Wed Feb 6 09:41:02 2019][1697236.723414] .rt_time : 0.000000 [Wed Feb 6 09:41:02 2019][1697236.728812] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:02 2019][1697236.734472] [Wed Feb 6 09:41:02 2019][1697236.736145] runnable tasks: [Wed Feb 6 09:41:02 2019][1697236.739117] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:02 2019][1697236.750065] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:02 2019][1697236.760951] watchdog/37 196 -11.986966 424285 0 0.000000 126665.988026 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.772285] migration/37 197 0.000000 40534 0 0.000000 169129.098273 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.783622] ksoftirqd/37 198 8903818.654923 1982082 120 0.000000 26171.884406 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.794958] kworker/37:0H 200 14257.505752 10 100 0.000000 0.033845 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.806322] ccp-2-q0 9990 3017.716652 2 120 0.000000 0.011292 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.817655] ccp-2-q1 9991 3029.725822 2 120 0.000000 0.009176 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.828992] ccp-2-q2 9992 3041.734733 2 120 0.000000 0.008917 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.840329] ccp-4-q0 9998 3053.742632 2 120 0.000000 0.007905 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.851664] ccp-4-q1 9999 3065.750151 2 120 0.000000 0.007525 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.863001] ccp-4-q2 10000 3077.757488 2 120 0.000000 0.007343 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.874337] ccp-4-q3 10001 3089.765156 2 120 0.000000 0.007674 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.885673] ccp-4-q4 10002 3101.772744 2 120 0.000000 0.007594 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.897020] gdbus 16800 8903940.211749 462887 120 0.000000 19814.986205 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.908362] nfsiod 18306 12503.020043 2 100 0.000000 0.010299 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.919702] kworker/37:1H 18813 8903818.636237 1453840 100 0.000000 6780.808132 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.931055] ptlrpcd_01_04 126457 8904840.226192 4022314 120 0.000000 63480.639646 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.942475] ptlrpcd_01_11 126464 8904840.224268 3970524 120 0.000000 62491.231722 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.953922] osp-syn-43-3 126728 8904840.172962 128280 120 0.000000 3620.233675 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.965369] osp-pre-29-1 126966 8904831.350331 213961 120 0.000000 4811.652352 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.976792] mdt01_005 127197 8677821.475545 31369433 120 0.000000 1704937.441031 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.988213] mdt01_009 127210 8904521.285890 30585315 120 0.000000 1254559.315896 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697236.999652] mdt01_020 129244 8677821.475535 39602290 120 0.000000 2626055.926046 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697237.011082] mdt_io01_029 129974 8649974.327489 1089800 120 0.000000 44994.660679 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697237.022514] mdt_io01_035 3384 6710540.638196 1628033 120 0.000000 56491.701784 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697237.033858] kworker/37:0 111915 8904840.129830 62605 120 0.000000 14223.069719 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697237.045320] kworker/37:1 24125 8411427.068992 2 120 0.000000 0.010380 0.000000 1 / [Wed Feb 6 09:41:02 2019][1697237.056675] [Wed Feb 6 09:41:02 2019][1697237.058356] cpu#38, 1996.233 MHz [Wed Feb 6 09:41:02 2019][1697237.061773] .nr_running : 0 [Wed Feb 6 09:41:02 2019][1697237.066574] .load : 0 [Wed Feb 6 09:41:02 2019][1697237.071374] .nr_switches : 247352984 [Wed Feb 6 09:41:02 2019][1697237.076874] .nr_load_updates : 110207925 [Wed Feb 6 09:41:02 2019][1697237.082362] .nr_uninterruptible : 13651 [Wed Feb 6 09:41:02 2019][1697237.087515] .next_balance : 5991.889859 [Wed Feb 6 09:41:02 2019][1697237.093198] .curr->pid : 0 [Wed Feb 6 09:41:02 2019][1697237.097997] .clock : 1697237097.908743 [Wed Feb 6 09:41:02 2019][1697237.104175] .cpu_load[0] : 0 [Wed Feb 6 09:41:02 2019][1697237.108969] .cpu_load[1] : 0 [Wed Feb 6 09:41:02 2019][1697237.113761] .cpu_load[2] : 0 [Wed Feb 6 09:41:02 2019][1697237.118553] .cpu_load[3] : 0 [Wed Feb 6 09:41:02 2019][1697237.123344] .cpu_load[4] : 0 [Wed Feb 6 09:41:02 2019][1697237.128139] .avg_idle : 1880644 [Wed Feb 6 09:41:02 2019][1697237.133453] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:02 2019][1697237.138765] [Wed Feb 6 09:41:02 2019][1697237.140435] cfs_rq[38]:/ [Wed Feb 6 09:41:02 2019][1697237.143146] .exec_clock : 0.000000 [Wed Feb 6 09:41:02 2019][1697237.148545] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:02 2019][1697237.153944] .min_vruntime : 6024589.629231 [Wed Feb 6 09:41:02 2019][1697237.159863] .max_vruntime : 0.000001 [Wed Feb 6 09:41:02 2019][1697237.165264] .spread : 0.000000 [Wed Feb 6 09:41:02 2019][1697237.170662] .spread0 : -2818830.778552 [Wed Feb 6 09:41:03 2019][1697237.176669] .nr_spread_over : 0 [Wed Feb 6 09:41:03 2019][1697237.181463] .nr_running : 0 [Wed Feb 6 09:41:03 2019][1697237.186256] .load : 0 [Wed Feb 6 09:41:03 2019][1697237.191049] .runnable_load_avg : 0 [Wed Feb 6 09:41:03 2019][1697237.195839] .blocked_load_avg : 0 [Wed Feb 6 09:41:03 2019][1697237.200633] .tg_load_avg : 0 [Wed Feb 6 09:41:03 2019][1697237.205425] .tg_load_contrib : 0 [Wed Feb 6 09:41:03 2019][1697237.210217] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:03 2019][1697237.215011] .tg->runnable_avg : 0 [Wed Feb 6 09:41:03 2019][1697237.219803] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:03 2019][1697237.224597] .throttled : 0 [Wed Feb 6 09:41:03 2019][1697237.229390] .throttle_count : 0 [Wed Feb 6 09:41:03 2019][1697237.234184] .avg->runnable_avg_sum : 793 [Wed Feb 6 09:41:03 2019][1697237.239148] .avg->runnable_avg_period : 46909 [Wed Feb 6 09:41:03 2019][1697237.244294] [Wed Feb 6 09:41:03 2019][1697237.245968] rt_rq[38]:/ [Wed Feb 6 09:41:03 2019][1697237.248594] .rt_nr_running : 0 [Wed Feb 6 09:41:03 2019][1697237.253386] .rt_throttled : 0 [Wed Feb 6 09:41:03 2019][1697237.258179] .rt_time : 0.000000 [Wed Feb 6 09:41:03 2019][1697237.263579] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:03 2019][1697237.269239] [Wed Feb 6 09:41:03 2019][1697237.270911] runnable tasks: [Wed Feb 6 09:41:03 2019][1697237.273885] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:03 2019][1697237.284830] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:03 2019][1697237.295717] watchdog/38 201 -11.986725 424297 0 0.000000 82272.111389 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.307052] migration/38 202 0.000000 9803 0 0.000000 219524.879166 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.318388] ksoftirqd/38 203 6024559.557496 1854201 120 0.000000 24545.202915 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.329723] kworker/38:0H 205 5969.545038 9 100 0.000000 0.034455 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.341072] ata_sff 7243 1936.717859 2 100 0.000000 0.009107 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.352406] kworker/38:1 7280 6024566.119079 270148 120 0.000000 13848.644284 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.363746] ext4-rsv-conver 7869 2942.614159 2 100 0.000000 0.009659 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.375078] kmpathd 9274 3655.895545 2 100 0.000000 0.016842 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.386421] ccp-7-q2 10011 3987.972740 2 120 0.000000 0.010220 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.397759] ccp-7-q3 10012 3999.980689 2 120 0.000000 0.007955 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.409093] ccp-7-q4 10013 4011.988938 2 120 0.000000 0.008255 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.420431] kworker/38:1H 16092 6023914.403001 1086064 100 0.000000 6514.763895 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.431772] gssproxy 16802 11055.320737 4 120 0.000000 3.155072 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.443108] writer#4 18230 6024578.861402 24921523 120 0.000000 230906.342900 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.454463] ptlrpcd_02_06 126471 6024572.196368 6827502 120 0.000000 86545.826509 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.465887] ptlrpcd_02_08 126473 6024578.796490 6840512 120 0.000000 83763.349167 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.477319] ldlm_cn02_002 126603 6024448.319041 7993860 120 0.000000 348705.162038 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.488743] mdt02_000 126629 5882444.103417 27464201 120 0.000000 1459916.451697 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.500177] osp-pre-43-3 126729 6024550.628744 214723 120 0.000000 7513.129922 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.511602] osp-pre-24-3 126769 6024550.096160 214687 120 0.000000 2831.105525 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.523032] osp-pre-8-3 126791 6024579.664100 214662 120 0.000000 5869.107640 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.534457] osp-pre-18-1 126922 6024579.550517 213948 120 0.000000 6361.569585 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.545895] mdt_rdpg02_002 127215 6024433.372527 2483783 120 0.000000 112357.332752 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.557313] ldlm_cb02_002 127217 5872119.380940 10744979 120 0.000000 157205.192061 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.568742] mdt02_009 127273 5898205.596350 28120348 120 0.000000 1364658.092434 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.580209] ldlm_cn02_013 13011 6024448.320022 2069033 120 0.000000 128568.203935 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.591547] ldlm_cn02_014 13387 6024448.319431 1418736 120 0.000000 130800.436776 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.602902] mdt02_043 18165 6024174.330495 57894 120 0.000000 222751.403241 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.614240] mdt02_056 18566 5898843.637801 64222 120 0.000000 6763.033254 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.625574] mdt02_061 18682 6024593.979961 34806 120 0.000000 8274.184288 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.636919] ldlm_cn02_027 23741 6024562.079599 20341 120 0.000000 70749.275422 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.648264] kworker/38:2 24998 5945973.561624 7 120 0.000000 0.048941 0.000000 2 / [Wed Feb 6 09:41:03 2019][1697237.659630] [Wed Feb 6 09:41:03 2019][1697237.661301] cpu#39, 1996.233 MHz [Wed Feb 6 09:41:03 2019][1697237.664708] .nr_running : 0 [Wed Feb 6 09:41:03 2019][1697237.669501] .load : 0 [Wed Feb 6 09:41:03 2019][1697237.674293] .nr_switches : 97166854 [Wed Feb 6 09:41:03 2019][1697237.679692] .nr_load_updates : 56351526 [Wed Feb 6 09:41:03 2019][1697237.685093] .nr_uninterruptible : 1576 [Wed Feb 6 09:41:03 2019][1697237.690159] .next_balance : 5991.890462 [Wed Feb 6 09:41:03 2019][1697237.695822] .curr->pid : 0 [Wed Feb 6 09:41:03 2019][1697237.700615] .clock : 1697237699.913890 [Wed Feb 6 09:41:03 2019][1697237.706794] .cpu_load[0] : 0 [Wed Feb 6 09:41:03 2019][1697237.711587] .cpu_load[1] : 0 [Wed Feb 6 09:41:03 2019][1697237.716378] .cpu_load[2] : 0 [Wed Feb 6 09:41:03 2019][1697237.721174] .cpu_load[3] : 0 [Wed Feb 6 09:41:03 2019][1697237.725975] .cpu_load[4] : 0 [Wed Feb 6 09:41:03 2019][1697237.730776] .avg_idle : 6292675 [Wed Feb 6 09:41:03 2019][1697237.736096] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:03 2019][1697237.741426] [Wed Feb 6 09:41:03 2019][1697237.743095] cfs_rq[39]:/ [Wed Feb 6 09:41:03 2019][1697237.745808] .exec_clock : 0.000000 [Wed Feb 6 09:41:03 2019][1697237.751206] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:03 2019][1697237.756606] .min_vruntime : 3405550.119537 [Wed Feb 6 09:41:03 2019][1697237.762527] .max_vruntime : 0.000001 [Wed Feb 6 09:41:03 2019][1697237.767925] .spread : 0.000000 [Wed Feb 6 09:41:03 2019][1697237.773326] .spread0 : -5437870.288246 [Wed Feb 6 09:41:03 2019][1697237.779331] .nr_spread_over : 0 [Wed Feb 6 09:41:03 2019][1697237.784123] .nr_running : 0 [Wed Feb 6 09:41:03 2019][1697237.788916] .load : 0 [Wed Feb 6 09:41:03 2019][1697237.793710] .runnable_load_avg : 0 [Wed Feb 6 09:41:03 2019][1697237.798503] .blocked_load_avg : 1 [Wed Feb 6 09:41:03 2019][1697237.803295] .tg_load_avg : 0 [Wed Feb 6 09:41:03 2019][1697237.808088] .tg_load_contrib : 0 [Wed Feb 6 09:41:03 2019][1697237.812881] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:03 2019][1697237.817673] .tg->runnable_avg : 0 [Wed Feb 6 09:41:03 2019][1697237.822465] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:03 2019][1697237.827259] .throttled : 0 [Wed Feb 6 09:41:03 2019][1697237.832051] .throttle_count : 0 [Wed Feb 6 09:41:03 2019][1697237.836843] .avg->runnable_avg_sum : 91 [Wed Feb 6 09:41:03 2019][1697237.841722] .avg->runnable_avg_period : 47212 [Wed Feb 6 09:41:03 2019][1697237.846872] [Wed Feb 6 09:41:03 2019][1697237.848543] rt_rq[39]:/ [Wed Feb 6 09:41:03 2019][1697237.851169] .rt_nr_running : 0 [Wed Feb 6 09:41:03 2019][1697237.855962] .rt_throttled : 0 [Wed Feb 6 09:41:03 2019][1697237.860755] .rt_time : 0.000000 [Wed Feb 6 09:41:03 2019][1697237.866155] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:03 2019][1697237.871814] [Wed Feb 6 09:41:03 2019][1697237.873487] runnable tasks: [Wed Feb 6 09:41:03 2019][1697237.876458] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:03 2019][1697237.887406] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:03 2019][1697237.898291] watchdog/39 206 -11.986876 424308 0 0.000000 63287.248863 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697237.909626] migration/39 207 0.000000 16304 0 0.000000 100941.530752 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697237.920963] ksoftirqd/39 208 3405040.315957 167293 120 0.000000 5885.346880 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697237.932298] kworker/39:0 209 3405527.802487 89399 120 0.000000 5481.577732 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697237.943637] kworker/39:0H 210 7497.056911 9 100 0.000000 0.029393 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697237.955005] kworker/39:1H 16540 3405040.297233 209244 100 0.000000 615.823675 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697237.966345] JS GC Helper 16803 7926.841190 3 120 0.000000 0.106490 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697237.977678] mdadm 16792 3405409.420341 1698 120 0.000000 106.518378 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697237.989015] sssd_be 16813 3405478.794539 790367 120 0.000000 856966.082346 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.000373] kiblnd_sd_03_02 126435 3405550.693645 87900823 120 0.000000 652642.023823 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.011799] ptlrpcd_03_03 126480 3405538.341747 5576249 120 0.000000 81702.846332 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.023233] mdt_out03_000 126657 3148213.449792 474236 120 0.000000 6922.629948 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.034659] osp-syn-9-3 126724 3405526.116075 127273 120 0.000000 6025.163562 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.046076] osp-syn-18-3 126732 3405526.115063 129178 120 0.000000 3608.002242 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.057510] osp-pre-13-1 126896 3405537.966961 213953 120 0.000000 4775.787229 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.068931] osp-pre-10-1 126902 3405537.773087 213965 120 0.000000 7551.317359 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.080362] osp-pre-15-1 126964 3405536.434321 213949 120 0.000000 7978.658688 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.091785] osp-pre-8-1 126980 3405537.734905 213947 120 0.000000 9782.321884 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.103255] mdt_out03_003 3908 3320425.970812 508255 120 0.000000 9437.030663 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.114593] mdt_io03_005 40451 3405041.212016 139409 120 0.000000 9490.712478 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.125933] mdt_io03_008 40479 3311283.969321 139719 120 0.000000 7015.064285 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.137290] mdt03_031 17907 3338047.679508 76095 120 0.000000 11971.590404 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.148628] mdt03_041 18154 3338548.818859 142681 120 0.000000 165196.989636 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.159968] mdt03_045 18186 3339147.269944 50909 120 0.000000 2543.423701 0.000000 3 / [Wed Feb 6 09:41:03 2019][1697238.171304] mdt03_050 18271 3334226.314072 21346 120 0.000000 1907.314164 0.000000 3 / [Wed Feb 6 09:41:04 2019][1697238.182643] kworker/u260:0 22241 3320426.485812 2421 120 0.000000 2784.390152 0.000000 3 / [Wed Feb 6 09:41:04 2019][1697238.193975] mdt_out03_004 23366 3320425.971654 37 120 0.000000 0.833370 0.000000 3 / [Wed Feb 6 09:41:04 2019][1697238.205321] ldlm_cn03_027 24596 3405478.777467 8911 120 0.000000 50165.608822 0.000000 3 / [Wed Feb 6 09:41:04 2019][1697238.216659] kworker/39:2 24707 3259097.190145 2 120 0.000000 0.008045 0.000000 3 / [Wed Feb 6 09:41:04 2019][1697238.228011] [Wed Feb 6 09:41:04 2019][1697238.229686] cpu#40, 1996.233 MHz [Wed Feb 6 09:41:04 2019][1697238.233093] .nr_running : 0 [Wed Feb 6 09:41:04 2019][1697238.237886] .load : 0 [Wed Feb 6 09:41:04 2019][1697238.242679] .nr_switches : 1084511698 [Wed Feb 6 09:41:04 2019][1697238.248261] .nr_load_updates : 213087104 [Wed Feb 6 09:41:04 2019][1697238.253756] .nr_uninterruptible : -114181 [Wed Feb 6 09:41:04 2019][1697238.259077] .next_balance : 5991.891031 [Wed Feb 6 09:41:04 2019][1697238.264745] .curr->pid : 0 [Wed Feb 6 09:41:04 2019][1697238.269564] .clock : 1697238268.918249 [Wed Feb 6 09:41:04 2019][1697238.275753] .cpu_load[0] : 0 [Wed Feb 6 09:41:04 2019][1697238.280552] .cpu_load[1] : 0 [Wed Feb 6 09:41:04 2019][1697238.285346] .cpu_load[2] : 0 [Wed Feb 6 09:41:04 2019][1697238.290147] .cpu_load[3] : 0 [Wed Feb 6 09:41:04 2019][1697238.294950] .cpu_load[4] : 0 [Wed Feb 6 09:41:04 2019][1697238.299749] .avg_idle : 196716 [Wed Feb 6 09:41:04 2019][1697238.304973] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:04 2019][1697238.310286] [Wed Feb 6 09:41:04 2019][1697238.311957] cfs_rq[40]:/ [Wed Feb 6 09:41:04 2019][1697238.314669] .exec_clock : 0.000000 [Wed Feb 6 09:41:04 2019][1697238.320069] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:04 2019][1697238.325469] .min_vruntime : 7352096.163908 [Wed Feb 6 09:41:04 2019][1697238.331388] .max_vruntime : 0.000001 [Wed Feb 6 09:41:04 2019][1697238.336787] .spread : 0.000000 [Wed Feb 6 09:41:04 2019][1697238.342184] .spread0 : -1491324.243875 [Wed Feb 6 09:41:04 2019][1697238.348192] .nr_spread_over : 0 [Wed Feb 6 09:41:04 2019][1697238.352986] .nr_running : 0 [Wed Feb 6 09:41:04 2019][1697238.357779] .load : 0 [Wed Feb 6 09:41:04 2019][1697238.362570] .runnable_load_avg : 0 [Wed Feb 6 09:41:04 2019][1697238.367362] .blocked_load_avg : 17 [Wed Feb 6 09:41:04 2019][1697238.372242] .tg_load_avg : 0 [Wed Feb 6 09:41:04 2019][1697238.377035] .tg_load_contrib : 0 [Wed Feb 6 09:41:04 2019][1697238.381828] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:04 2019][1697238.386621] .tg->runnable_avg : 0 [Wed Feb 6 09:41:04 2019][1697238.391416] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:04 2019][1697238.396207] .throttled : 0 [Wed Feb 6 09:41:04 2019][1697238.400998] .throttle_count : 0 [Wed Feb 6 09:41:04 2019][1697238.405792] .avg->runnable_avg_sum : 565 [Wed Feb 6 09:41:04 2019][1697238.410756] .avg->runnable_avg_period : 46872 [Wed Feb 6 09:41:04 2019][1697238.415905] [Wed Feb 6 09:41:04 2019][1697238.417580] rt_rq[40]:/ [Wed Feb 6 09:41:04 2019][1697238.420205] .rt_nr_running : 0 [Wed Feb 6 09:41:04 2019][1697238.424996] .rt_throttled : 0 [Wed Feb 6 09:41:04 2019][1697238.429788] .rt_time : 0.000000 [Wed Feb 6 09:41:04 2019][1697238.435190] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:04 2019][1697238.440849] [Wed Feb 6 09:41:04 2019][1697238.442522] runnable tasks: [Wed Feb 6 09:41:04 2019][1697238.445493] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:04 2019][1697238.456440] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:04 2019][1697238.467327] watchdog/40 211 -11.990252 424303 0 0.000000 92414.307078 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.478661] migration/40 212 0.000000 1767 0 0.000000 152591.037038 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.489997] ksoftirqd/40 213 7344145.025321 2085953 120 0.000000 23275.944124 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.501332] kworker/40:0H 215 5184.572414 10 100 0.000000 0.053491 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.512680] kworker/40:1 320 7352083.949048 325766 120 0.000000 14179.690172 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.524043] gssproxy 16814 4741.095675 2 120 0.000000 0.044535 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.535382] in:imjournal 18241 7352090.513120 1999330 120 0.000000 35678.572776 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.546719] kworker/40:1H 39599 7351820.197327 1936401 100 0.000000 11067.413175 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.558069] kiblnd_sd_00_02 126423 7352098.708132 1556611850 120 0.000000 9507436.853834 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.569580] ptlrpcd_00_01 126442 7352079.813354 4282414 120 0.000000 63439.366781 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.581002] ptlrpcd_00_09 126450 7352084.514821 4237557 120 0.000000 63265.736103 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.592435] ldlm_cn00_000 126595 7352034.759491 3379164 120 0.000000 182973.897740 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.603861] mdt_seqm_0000 126661 6556806.706095 230 120 0.000000 3.297969 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.615284] mdt_fld_0001 126664 6713400.022323 3052 120 0.000000 49.216999 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.626716] osp-pre-21-3 126747 7352080.102619 214712 120 0.000000 6040.472751 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.638138] osp-pre-41-3 126751 7352086.701330 214715 120 0.000000 7106.936639 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.649576] osp-syn-17-1 126925 7352077.834880 198630 120 0.000000 6889.928291 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.661031] mdt00_021 129276 7347257.136652 83359605 120 0.000000 2460191.680125 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.672455] mdt00_023 129587 7347257.138065 78484750 120 0.000000 2301647.971063 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.683874] mdt_io00_022 129737 7253658.956751 2053587 120 0.000000 91170.465642 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.695306] mdt_rdpg00_005 1270 7352088.938558 954179 120 0.000000 50903.968675 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.706641] mdt_rdpg00_006 1271 7224601.546350 6056632 120 0.000000 274224.708892 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.718006] mdt00_030 17877 7231283.175052 149785 120 0.000000 467956.281309 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.729343] mdt00_040 17915 7347257.138625 21956 120 0.000000 157971.165174 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.740677] mdt00_044 17932 7352095.528297 49253 120 0.000000 163731.808230 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.752014] mdt00_048 17952 7254029.669702 188672 120 0.000000 164577.835555 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.763373] kworker/40:2 24636 7101346.067861 7 120 0.000000 0.248807 0.000000 0 / [Wed Feb 6 09:41:04 2019][1697238.774750] [Wed Feb 6 09:41:04 2019][1697238.776429] cpu#41, 1996.233 MHz [Wed Feb 6 09:41:04 2019][1697238.779838] .nr_running : 0 [Wed Feb 6 09:41:04 2019][1697238.784638] .load : 0 [Wed Feb 6 09:41:04 2019][1697238.789432] .nr_switches : 702230717 [Wed Feb 6 09:41:04 2019][1697238.794929] .nr_load_updates : 134951593 [Wed Feb 6 09:41:04 2019][1697238.800422] .nr_uninterruptible : 43261 [Wed Feb 6 09:41:04 2019][1697238.805568] .next_balance : 5991.891579 [Wed Feb 6 09:41:04 2019][1697238.811230] .curr->pid : 0 [Wed Feb 6 09:41:04 2019][1697238.816031] .clock : 1697238815.927111 [Wed Feb 6 09:41:04 2019][1697238.822217] .cpu_load[0] : 0 [Wed Feb 6 09:41:04 2019][1697238.827013] .cpu_load[1] : 0 [Wed Feb 6 09:41:04 2019][1697238.831811] .cpu_load[2] : 0 [Wed Feb 6 09:41:04 2019][1697238.836605] .cpu_load[3] : 0 [Wed Feb 6 09:41:04 2019][1697238.841400] .cpu_load[4] : 0 [Wed Feb 6 09:41:04 2019][1697238.846199] .avg_idle : 10000000 [Wed Feb 6 09:41:04 2019][1697238.851598] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:04 2019][1697238.856919] [Wed Feb 6 09:41:04 2019][1697238.858589] cfs_rq[41]:/ [Wed Feb 6 09:41:04 2019][1697238.861301] .exec_clock : 0.000000 [Wed Feb 6 09:41:04 2019][1697238.866700] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:04 2019][1697238.872102] .min_vruntime : 6421986.295560 [Wed Feb 6 09:41:04 2019][1697238.878020] .max_vruntime : 0.000001 [Wed Feb 6 09:41:04 2019][1697238.883419] .spread : 0.000000 [Wed Feb 6 09:41:04 2019][1697238.888816] .spread0 : -2421434.112223 [Wed Feb 6 09:41:04 2019][1697238.894823] .nr_spread_over : 0 [Wed Feb 6 09:41:04 2019][1697238.899615] .nr_running : 0 [Wed Feb 6 09:41:04 2019][1697238.904408] .load : 0 [Wed Feb 6 09:41:04 2019][1697238.909202] .runnable_load_avg : 0 [Wed Feb 6 09:41:04 2019][1697238.913995] .blocked_load_avg : 0 [Wed Feb 6 09:41:04 2019][1697238.918788] .tg_load_avg : 0 [Wed Feb 6 09:41:04 2019][1697238.923579] .tg_load_contrib : 0 [Wed Feb 6 09:41:04 2019][1697238.928373] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:04 2019][1697238.933164] .tg->runnable_avg : 0 [Wed Feb 6 09:41:04 2019][1697238.937957] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:04 2019][1697238.942751] .throttled : 0 [Wed Feb 6 09:41:04 2019][1697238.947543] .throttle_count : 0 [Wed Feb 6 09:41:04 2019][1697238.952337] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:41:04 2019][1697238.957128] .avg->runnable_avg_period : 47777 [Wed Feb 6 09:41:04 2019][1697238.962277] [Wed Feb 6 09:41:04 2019][1697238.963949] rt_rq[41]:/ [Wed Feb 6 09:41:04 2019][1697238.966575] .rt_nr_running : 0 [Wed Feb 6 09:41:04 2019][1697238.971369] .rt_throttled : 0 [Wed Feb 6 09:41:04 2019][1697238.976162] .rt_time : 0.000000 [Wed Feb 6 09:41:04 2019][1697238.981560] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:04 2019][1697238.987220] [Wed Feb 6 09:41:04 2019][1697238.988892] runnable tasks: [Wed Feb 6 09:41:04 2019][1697238.991865] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:04 2019][1697239.002810] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:04 2019][1697239.013700] watchdog/41 216 -11.986445 424291 0 0.000000 118752.475031 0.000000 1 / [Wed Feb 6 09:41:04 2019][1697239.025033] migration/41 217 0.000000 19690 0 0.000000 183527.390114 0.000000 1 / [Wed Feb 6 09:41:04 2019][1697239.036369] ksoftirqd/41 218 6412504.626525 2864282 120 0.000000 28537.237072 0.000000 1 / [Wed Feb 6 09:41:04 2019][1697239.047706] kworker/41:0H 220 19055.044198 10 100 0.000000 0.041278 0.000000 1 / [Wed Feb 6 09:41:04 2019][1697239.059052] scsi_tmf_1 3706 963.917968 2 100 0.000000 0.008246 0.000000 1 / [Wed Feb 6 09:41:04 2019][1697239.070404] kdmwork-253:0 16278 6208093.762035 4918 120 0.000000 44.353659 0.000000 1 / [Wed Feb 6 09:41:04 2019][1697239.081758] kworker/41:1H 49497 6412504.507235 1750333 100 0.000000 6242.723249 0.000000 1 / [Wed Feb 6 09:41:04 2019][1697239.093096] ptlrpc_hr01_000 126386 6421974.335595 28985429 120 0.000000 219646.933304 0.000000 1 / [Wed Feb 6 09:41:04 2019][1697239.104526] ptlrpcd_01_10 126463 6421974.374157 4034755 120 0.000000 61141.013777 0.000000 1 / [Wed Feb 6 09:41:04 2019][1697239.116001] mdt_io01_006 127227 3965182.911157 2993217 120 0.000000 106056.488693 0.000000 1 / [Wed Feb 6 09:41:04 2019][1697239.127429] mdt_io01_013 127657 1558419.520329 1395747 120 0.000000 48162.987667 0.000000 1 / [Wed Feb 6 09:41:04 2019][1697239.138871] mdt_io01_023 129282 1558419.520319 672730 120 0.000000 33318.510043 0.000000 1 / [Wed Feb 6 09:41:04 2019][1697239.150297] mdt_io01_032 130112 6317086.895763 3194010 120 0.000000 138720.522843 0.000000 1 / [Wed Feb 6 09:41:04 2019][1697239.161721] mdt01_036 847 6319187.737747 24411571 120 0.000000 1948266.453075 0.000000 1 / [Wed Feb 6 09:41:04 2019][1697239.173059] mdt_rdpg01_007 16573 6421783.801310 4599111 120 0.000000 193998.771911 0.000000 1 / [Wed Feb 6 09:41:04 2019][1697239.184407] kworker/41:0 7616 6421974.310568 32268 120 0.000000 19297.051385 0.000000 1 / [Wed Feb 6 09:41:05 2019][1697239.195772] mdt01_061 19115 6421784.242852 181181 120 0.000000 18283.953183 0.000000 1 / [Wed Feb 6 09:41:05 2019][1697239.207105] kworker/41:2 22569 5056689.451765 9 120 0.000000 0.041128 0.000000 1 / [Wed Feb 6 09:41:05 2019][1697239.218444] ldlm_cn01_026 23733 6421949.828648 53093 120 0.000000 119519.225704 0.000000 1 / [Wed Feb 6 09:41:05 2019][1697239.229804] [Wed Feb 6 09:41:05 2019][1697239.231478] cpu#42, 1996.233 MHz [Wed Feb 6 09:41:05 2019][1697239.234885] .nr_running : 0 [Wed Feb 6 09:41:05 2019][1697239.239682] .load : 0 [Wed Feb 6 09:41:05 2019][1697239.244480] .nr_switches : 265488756 [Wed Feb 6 09:41:05 2019][1697239.249984] .nr_load_updates : 134935370 [Wed Feb 6 09:41:05 2019][1697239.255468] .nr_uninterruptible : 34511 [Wed Feb 6 09:41:05 2019][1697239.260608] .next_balance : 5991.892033 [Wed Feb 6 09:41:05 2019][1697239.266268] .curr->pid : 0 [Wed Feb 6 09:41:05 2019][1697239.271062] .clock : 1697239270.929618 [Wed Feb 6 09:41:05 2019][1697239.277249] .cpu_load[0] : 0 [Wed Feb 6 09:41:05 2019][1697239.282041] .cpu_load[1] : 0 [Wed Feb 6 09:41:05 2019][1697239.286834] .cpu_load[2] : 0 [Wed Feb 6 09:41:05 2019][1697239.291629] .cpu_load[3] : 0 [Wed Feb 6 09:41:05 2019][1697239.296427] .cpu_load[4] : 0 [Wed Feb 6 09:41:05 2019][1697239.301221] .avg_idle : 1291744 [Wed Feb 6 09:41:05 2019][1697239.306550] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:05 2019][1697239.311862] [Wed Feb 6 09:41:05 2019][1697239.313534] cfs_rq[42]:/ [Wed Feb 6 09:41:05 2019][1697239.316246] .exec_clock : 0.000000 [Wed Feb 6 09:41:05 2019][1697239.321644] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:05 2019][1697239.327045] .min_vruntime : 4352970.273557 [Wed Feb 6 09:41:05 2019][1697239.332964] .max_vruntime : 0.000001 [Wed Feb 6 09:41:05 2019][1697239.338363] .spread : 0.000000 [Wed Feb 6 09:41:05 2019][1697239.343762] .spread0 : -4490450.134226 [Wed Feb 6 09:41:05 2019][1697239.349769] .nr_spread_over : 0 [Wed Feb 6 09:41:05 2019][1697239.354562] .nr_running : 0 [Wed Feb 6 09:41:05 2019][1697239.359355] .load : 0 [Wed Feb 6 09:41:05 2019][1697239.364146] .runnable_load_avg : 0 [Wed Feb 6 09:41:05 2019][1697239.368938] .blocked_load_avg : 7 [Wed Feb 6 09:41:05 2019][1697239.373731] .tg_load_avg : 0 [Wed Feb 6 09:41:05 2019][1697239.378525] .tg_load_contrib : 0 [Wed Feb 6 09:41:05 2019][1697239.383317] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:05 2019][1697239.388111] .tg->runnable_avg : 0 [Wed Feb 6 09:41:05 2019][1697239.392902] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:05 2019][1697239.397695] .throttled : 0 [Wed Feb 6 09:41:05 2019][1697239.402489] .throttle_count : 0 [Wed Feb 6 09:41:05 2019][1697239.407283] .avg->runnable_avg_sum : 201 [Wed Feb 6 09:41:05 2019][1697239.412249] .avg->runnable_avg_period : 46215 [Wed Feb 6 09:41:05 2019][1697239.417396] [Wed Feb 6 09:41:05 2019][1697239.419070] rt_rq[42]:/ [Wed Feb 6 09:41:05 2019][1697239.421695] .rt_nr_running : 0 [Wed Feb 6 09:41:05 2019][1697239.426488] .rt_throttled : 0 [Wed Feb 6 09:41:05 2019][1697239.431280] .rt_time : 0.007564 [Wed Feb 6 09:41:05 2019][1697239.436679] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:05 2019][1697239.442339] [Wed Feb 6 09:41:05 2019][1697239.444009] runnable tasks: [Wed Feb 6 09:41:05 2019][1697239.446982] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:05 2019][1697239.457931] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:05 2019][1697239.468817] watchdog/42 221 -11.988529 424301 0 0.000000 118926.021674 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.480151] migration/42 222 0.000000 1788 0 0.000000 162211.714696 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.491487] ksoftirqd/42 223 4352958.306860 1950048 120 0.000000 31820.112445 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.502823] kworker/42:0 224 4352956.639174 12326 120 0.000000 11650.961785 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.514159] kworker/42:0H 225 4443.608397 10 100 0.000000 0.045845 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.525504] scsi_eh_1 3704 935.047955 2 120 0.000000 0.009298 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.536845] mlx5_health0000 7348 1628.974665 2 100 0.000000 0.007363 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.548199] rdma_cm 17508 4158.282124 2 100 0.000000 0.011782 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.559548] kworker/42:1H 20448 4352958.274840 1195032 100 0.000000 3926.111314 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.570920] osp-pre-2-3 126698 4352940.007904 205473 120 0.000000 5178.700130 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.582354] osp-pre-28-3 126795 4352941.541313 214711 120 0.000000 7439.883198 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.593779] osp-pre-19-1 126920 4352947.885831 213949 120 0.000000 5080.532626 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.605205] osp-pre-25-1 126924 4352944.730720 213949 120 0.000000 4823.810118 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.616643] mdt_io02_005 127226 4273536.741015 1913775 120 0.000000 78410.408878 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.628078] ldlm_cn02_005 127801 4352860.959463 7439390 120 0.000000 272268.540402 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.639509] mdt02_017 129260 4284544.883453 22128860 120 0.000000 960702.361903 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.650935] mdt_io02_022 129309 4352833.523896 1491000 120 0.000000 74504.023175 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.662355] mdt_io02_024 130203 4352958.577350 1851886 120 0.000000 94523.699164 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.673784] ldlm_cn02_010 3385 4352860.959162 5014031 120 0.000000 246471.167284 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.685136] ldlm_cn02_011 12935 4352860.959001 1264504 120 0.000000 141034.526088 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.696474] ldlm_cn02_012 12966 4352860.959352 1335362 120 0.000000 144055.579849 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.707816] mdt02_029 17838 4284501.010128 73373 120 0.000000 164645.090322 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.719159] mdt02_038 17930 4352971.021189 195222 120 0.000000 178330.479032 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.730494] mdt02_039 17984 4352407.589184 51937 120 0.000000 6988.476076 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.741832] mdt02_054 18272 4284544.883432 185183 120 0.000000 20562.553040 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.753167] kworker/u259:1 20339 4352671.249945 876 120 0.000000 2681.537272 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.764500] ldlm_cn02_016 23607 4352860.960164 15569 120 0.000000 70545.203060 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.775838] ldlm_cn02_020 23724 4352860.959382 37075 120 0.000000 105949.133767 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.787174] ldlm_cn02_024 23732 4352860.961166 34628 120 0.000000 117596.695095 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.798508] ldlm_cn02_025 23734 4352926.248065 18406 120 0.000000 107025.472528 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.809846] ldlm_cn02_028 23742 4352860.959102 24045 120 0.000000 93243.105053 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.821180] ldlm_cn02_029 23743 4352860.960123 33443 120 0.000000 109823.155473 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.832517] ldlm_cn02_030 23747 4352860.959733 14244 120 0.000000 84785.622690 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.843861] kworker/42:2 24882 4274510.786256 11 120 0.000000 181.918086 0.000000 2 / [Wed Feb 6 09:41:05 2019][1697239.855245] [Wed Feb 6 09:41:05 2019][1697239.856916] cpu#43, 1996.233 MHz [Wed Feb 6 09:41:05 2019][1697239.860323] .nr_running : 0 [Wed Feb 6 09:41:05 2019][1697239.865116] .load : 0 [Wed Feb 6 09:41:05 2019][1697239.869910] .nr_switches : 87996206 [Wed Feb 6 09:41:05 2019][1697239.875309] .nr_load_updates : 55163843 [Wed Feb 6 09:41:05 2019][1697239.880708] .nr_uninterruptible : 853 [Wed Feb 6 09:41:05 2019][1697239.885674] .next_balance : 5991.892658 [Wed Feb 6 09:41:05 2019][1697239.891335] .curr->pid : 0 [Wed Feb 6 09:41:05 2019][1697239.896136] .clock : 1697239895.934925 [Wed Feb 6 09:41:05 2019][1697239.902316] .cpu_load[0] : 0 [Wed Feb 6 09:41:05 2019][1697239.907114] .cpu_load[1] : 0 [Wed Feb 6 09:41:05 2019][1697239.911907] .cpu_load[2] : 0 [Wed Feb 6 09:41:05 2019][1697239.916700] .cpu_load[3] : 0 [Wed Feb 6 09:41:05 2019][1697239.921492] .cpu_load[4] : 0 [Wed Feb 6 09:41:05 2019][1697239.926286] .avg_idle : 4599077 [Wed Feb 6 09:41:05 2019][1697239.931600] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:05 2019][1697239.936921] [Wed Feb 6 09:41:05 2019][1697239.938590] cfs_rq[43]:/ [Wed Feb 6 09:41:05 2019][1697239.941302] .exec_clock : 0.000000 [Wed Feb 6 09:41:05 2019][1697239.946703] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:05 2019][1697239.952102] .min_vruntime : 1243111.688639 [Wed Feb 6 09:41:05 2019][1697239.958021] .max_vruntime : 0.000001 [Wed Feb 6 09:41:05 2019][1697239.963420] .spread : 0.000000 [Wed Feb 6 09:41:05 2019][1697239.968818] .spread0 : -7600309.311665 [Wed Feb 6 09:41:05 2019][1697239.974824] .nr_spread_over : 0 [Wed Feb 6 09:41:05 2019][1697239.979618] .nr_running : 0 [Wed Feb 6 09:41:05 2019][1697239.984409] .load : 0 [Wed Feb 6 09:41:05 2019][1697239.989204] .runnable_load_avg : 0 [Wed Feb 6 09:41:05 2019][1697239.993995] .blocked_load_avg : 0 [Wed Feb 6 09:41:05 2019][1697239.998790] .tg_load_avg : 0 [Wed Feb 6 09:41:05 2019][1697240.003580] .tg_load_contrib : 0 [Wed Feb 6 09:41:05 2019][1697240.008374] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:05 2019][1697240.013167] .tg->runnable_avg : 0 [Wed Feb 6 09:41:05 2019][1697240.017959] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:05 2019][1697240.022752] .throttled : 0 [Wed Feb 6 09:41:05 2019][1697240.027545] .throttle_count : 0 [Wed Feb 6 09:41:05 2019][1697240.032339] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:41:05 2019][1697240.037132] .avg->runnable_avg_period : 47264 [Wed Feb 6 09:41:05 2019][1697240.042279] [Wed Feb 6 09:41:05 2019][1697240.043951] rt_rq[43]:/ [Wed Feb 6 09:41:05 2019][1697240.046578] .rt_nr_running : 0 [Wed Feb 6 09:41:05 2019][1697240.051369] .rt_throttled : 0 [Wed Feb 6 09:41:05 2019][1697240.056164] .rt_time : 0.000000 [Wed Feb 6 09:41:05 2019][1697240.061563] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:05 2019][1697240.067221] [Wed Feb 6 09:41:05 2019][1697240.068895] runnable tasks: [Wed Feb 6 09:41:05 2019][1697240.071866] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:05 2019][1697240.082813] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:05 2019][1697240.093703] watchdog/43 226 -11.985913 424309 0 0.000000 59145.351776 0.000000 3 / [Wed Feb 6 09:41:05 2019][1697240.105034] migration/43 227 0.000000 2735 0 0.000000 105893.573166 0.000000 3 / [Wed Feb 6 09:41:05 2019][1697240.116369] ksoftirqd/43 228 1242921.941116 168651 120 0.000000 2957.038562 0.000000 3 / [Wed Feb 6 09:41:05 2019][1697240.127708] kworker/43:0H 230 4279.648784 10 100 0.000000 0.036118 0.000000 3 / [Wed Feb 6 09:41:05 2019][1697240.139054] mlx5_cmd_0000:8 7362 1036778.753120 4 100 0.000000 0.026529 0.000000 3 / [Wed Feb 6 09:41:05 2019][1697240.150391] poll_mpt3sas0_s 7601 1036778.753083 12 100 0.000000 3313.079236 0.000000 3 / [Wed Feb 6 09:41:05 2019][1697240.161729] ccp-3-q2 9995 2163.037669 2 120 0.000000 0.007313 0.000000 3 / [Wed Feb 6 09:41:05 2019][1697240.173085] kworker/43:1H 40567 1242921.928791 183579 100 0.000000 527.548176 0.000000 3 / [Wed Feb 6 09:41:05 2019][1697240.184452] mdt_rdpg03_001 126642 1242298.326861 2561424 120 0.000000 88397.443711 0.000000 3 / [Wed Feb 6 09:41:06 2019][1697240.195876] osp-pre-46-3 126711 1243099.699179 214655 120 0.000000 10159.679073 0.000000 3 / [Wed Feb 6 09:41:06 2019][1697240.207296] osp-pre-9-3 126725 1243099.702234 214616 120 0.000000 4295.171591 0.000000 3 / [Wed Feb 6 09:41:06 2019][1697240.218721] osp-pre-6-3 126767 1243099.709900 214616 120 0.000000 3860.967396 0.000000 3 / [Wed Feb 6 09:41:06 2019][1697240.230141] osp-pre-15-3 126775 1243099.700381 214717 120 0.000000 7932.477725 0.000000 3 / [Wed Feb 6 09:41:06 2019][1697240.241568] osp-syn-14-1 126905 1243099.711872 198590 120 0.000000 4682.793905 0.000000 3 / [Wed Feb 6 09:41:06 2019][1697240.253004] osp-syn-36-1 126977 1243099.778990 196577 120 0.000000 5311.176305 0.000000 3 / [Wed Feb 6 09:41:06 2019][1697240.264482] ldlm_cn03_010 12925 1243108.691198 1936027 120 0.000000 170045.465276 0.000000 3 / [Wed Feb 6 09:41:06 2019][1697240.275815] mdt_rdpg03_004 13294 1242983.867740 1023704 120 0.000000 35278.162398 0.000000 3 / [Wed Feb 6 09:41:06 2019][1697240.287151] kworker/43:2 16827 1243099.705200 10125 120 0.000000 9335.926365 0.000000 3 / [Wed Feb 6 09:41:06 2019][1697240.298485] mdt03_014 17804 1243099.760995 65113 120 0.000000 4837.646755 0.000000 3 / [Wed Feb 6 09:41:06 2019][1697240.309855] ldlm_cn03_026 24595 1243109.443557 7294 120 0.000000 51616.382231 0.000000 3 / [Wed Feb 6 09:41:06 2019][1697240.321193] kworker/43:1 24661 898541.164401 4 120 0.000000 0.018636 0.000000 3 / [Wed Feb 6 09:41:06 2019][1697240.332544] [Wed Feb 6 09:41:06 2019][1697240.334221] cpu#44, 1996.233 MHz [Wed Feb 6 09:41:06 2019][1697240.337626] .nr_running : 0 [Wed Feb 6 09:41:06 2019][1697240.342422] .load : 0 [Wed Feb 6 09:41:06 2019][1697240.347215] .nr_switches : 1795325096 [Wed Feb 6 09:41:06 2019][1697240.352789] .nr_load_updates : 289392746 [Wed Feb 6 09:41:06 2019][1697240.358281] .nr_uninterruptible : -130581 [Wed Feb 6 09:41:06 2019][1697240.363595] .next_balance : 5991.893136 [Wed Feb 6 09:41:06 2019][1697240.369255] .curr->pid : 0 [Wed Feb 6 09:41:06 2019][1697240.374063] .clock : 1697240373.939176 [Wed Feb 6 09:41:06 2019][1697240.380244] .cpu_load[0] : 0 [Wed Feb 6 09:41:06 2019][1697240.385036] .cpu_load[1] : 0 [Wed Feb 6 09:41:06 2019][1697240.389828] .cpu_load[2] : 0 [Wed Feb 6 09:41:06 2019][1697240.394620] .cpu_load[3] : 0 [Wed Feb 6 09:41:06 2019][1697240.399414] .cpu_load[4] : 0 [Wed Feb 6 09:41:06 2019][1697240.404209] .avg_idle : 10000000 [Wed Feb 6 09:41:06 2019][1697240.409616] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:06 2019][1697240.414928] [Wed Feb 6 09:41:06 2019][1697240.416596] cfs_rq[44]:/ [Wed Feb 6 09:41:06 2019][1697240.419310] .exec_clock : 0.000000 [Wed Feb 6 09:41:06 2019][1697240.424708] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:06 2019][1697240.430108] .min_vruntime : 8078740.909007 [Wed Feb 6 09:41:06 2019][1697240.436028] .max_vruntime : 0.000001 [Wed Feb 6 09:41:06 2019][1697240.441428] .spread : 0.000000 [Wed Feb 6 09:41:06 2019][1697240.446827] .spread0 : -764680.211234 [Wed Feb 6 09:41:06 2019][1697240.452746] .nr_spread_over : 0 [Wed Feb 6 09:41:06 2019][1697240.457538] .nr_running : 0 [Wed Feb 6 09:41:06 2019][1697240.462332] .load : 0 [Wed Feb 6 09:41:06 2019][1697240.467125] .runnable_load_avg : 0 [Wed Feb 6 09:41:06 2019][1697240.471918] .blocked_load_avg : 0 [Wed Feb 6 09:41:06 2019][1697240.476710] .tg_load_avg : 0 [Wed Feb 6 09:41:06 2019][1697240.481503] .tg_load_contrib : 0 [Wed Feb 6 09:41:06 2019][1697240.486296] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:06 2019][1697240.491089] .tg->runnable_avg : 0 [Wed Feb 6 09:41:06 2019][1697240.495880] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:06 2019][1697240.500674] .throttled : 0 [Wed Feb 6 09:41:06 2019][1697240.505466] .throttle_count : 0 [Wed Feb 6 09:41:06 2019][1697240.510260] .avg->runnable_avg_sum : 31 [Wed Feb 6 09:41:06 2019][1697240.515137] .avg->runnable_avg_period : 48067 [Wed Feb 6 09:41:06 2019][1697240.520280] [Wed Feb 6 09:41:06 2019][1697240.521950] rt_rq[44]:/ [Wed Feb 6 09:41:06 2019][1697240.524576] .rt_nr_running : 0 [Wed Feb 6 09:41:06 2019][1697240.529368] .rt_throttled : 0 [Wed Feb 6 09:41:06 2019][1697240.534161] .rt_time : 0.000000 [Wed Feb 6 09:41:06 2019][1697240.539562] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:06 2019][1697240.545220] [Wed Feb 6 09:41:06 2019][1697240.546893] runnable tasks: [Wed Feb 6 09:41:06 2019][1697240.549866] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:06 2019][1697240.560812] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:06 2019][1697240.571699] watchdog/44 231 -11.988468 424291 0 0.000000 112055.242619 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.583033] migration/44 232 0.000000 1735 0 0.000000 157941.547191 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.594368] ksoftirqd/44 233 8078728.951247 1644890 120 0.000000 20362.030505 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.605705] kworker/44:0H 235 3943.996444 10 100 0.000000 0.048181 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.617058] ccp-2-q3 9993 2181.424850 2 120 0.000000 0.009598 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.628395] ccp-2-q4 9994 2193.432618 2 120 0.000000 0.007774 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.639738] kdmwork-253:2 16480 8078729.064289 33969103 120 0.000000 404396.656646 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.651087] kworker/44:1H 19058 8078728.909322 1810529 100 0.000000 9959.955291 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.662433] ptlrpcd_00_08 126449 8078729.017051 4295746 120 0.000000 70893.060132 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.673853] ptlrpcd_00_11 126452 8078728.941199 4270083 120 0.000000 66717.602951 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.685296] osp-pre-10-3 126713 8078728.931459 214649 120 0.000000 7253.410924 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.696719] osp-syn-40-3 126762 8078728.933904 126274 120 0.000000 4926.328792 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.708142] osp_up3-1 126891 7645813.446719 255 120 0.000000 7.472543 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.719562] osp-syn-12-1 126893 8078728.978779 198340 120 0.000000 4673.305319 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.730992] osp-pre-7-1 126950 8078728.929245 213818 120 0.000000 6363.415987 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.742414] osp-pre-24-1 126958 8078728.931059 213934 120 0.000000 6836.977826 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.753841] osp-pre-28-1 126984 8078728.935898 213907 120 0.000000 5027.930034 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.765279] ldlm_cn00_007 128033 8078650.609533 2762152 120 0.000000 199874.064578 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.776702] mdt_rdpg00_003 128100 7785798.305037 8606114 120 0.000000 372719.012742 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.788126] mdt_rdpg00_004 128546 7785543.708577 8928447 120 0.000000 397534.714036 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.799548] mdt00_014 129245 7828666.048421 85010135 120 0.000000 3030982.337001 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.810975] mdt_io00_019 129268 8078729.057512 1438193 120 0.000000 54720.216883 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.822401] ldlm_cn00_024 130332 8078675.078314 4024832 120 0.000000 245542.212461 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.833824] ldlm_cn00_027 130337 8078729.999993 3414033 120 0.000000 213198.832602 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.845253] kworker/44:0 17593 8078728.932351 189338 120 0.000000 16374.966691 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.856596] kworker/44:2 76594 6463802.593980 2 120 0.000000 0.009208 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.867936] kworker/u257:1 17401 8078729.358313 3209 120 0.000000 3722.935231 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.879281] mdt00_043 17931 8078663.931734 327121 120 0.000000 12639.289234 0.000000 0 / [Wed Feb 6 09:41:06 2019][1697240.890671] [Wed Feb 6 09:41:06 2019][1697240.892343] cpu#45, 1996.233 MHz [Wed Feb 6 09:41:06 2019][1697240.895753] .nr_running : 0 [Wed Feb 6 09:41:06 2019][1697240.900544] .load : 0 [Wed Feb 6 09:41:06 2019][1697240.905338] .nr_switches : 1040500386 [Wed Feb 6 09:41:06 2019][1697240.910910] .nr_load_updates : 133208473 [Wed Feb 6 09:41:06 2019][1697240.916398] .nr_uninterruptible : 20981 [Wed Feb 6 09:41:06 2019][1697240.921545] .next_balance : 5991.893696 [Wed Feb 6 09:41:06 2019][1697240.927204] .curr->pid : 0 [Wed Feb 6 09:41:06 2019][1697240.931998] .clock : 1697240931.942395 [Wed Feb 6 09:41:06 2019][1697240.938178] .cpu_load[0] : 0 [Wed Feb 6 09:41:06 2019][1697240.942979] .cpu_load[1] : 0 [Wed Feb 6 09:41:06 2019][1697240.947779] .cpu_load[2] : 0 [Wed Feb 6 09:41:06 2019][1697240.952571] .cpu_load[3] : 0 [Wed Feb 6 09:41:06 2019][1697240.957363] .cpu_load[4] : 0 [Wed Feb 6 09:41:06 2019][1697240.962156] .avg_idle : 9921412 [Wed Feb 6 09:41:06 2019][1697240.967469] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:06 2019][1697240.972782] [Wed Feb 6 09:41:06 2019][1697240.974452] cfs_rq[45]:/ [Wed Feb 6 09:41:06 2019][1697240.977165] .exec_clock : 0.000000 [Wed Feb 6 09:41:06 2019][1697240.982564] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:06 2019][1697240.987962] .min_vruntime : 7081002.666927 [Wed Feb 6 09:41:06 2019][1697240.993883] .max_vruntime : 0.000001 [Wed Feb 6 09:41:06 2019][1697240.999281] .spread : 0.000000 [Wed Feb 6 09:41:06 2019][1697241.004681] .spread0 : -1762418.571106 [Wed Feb 6 09:41:06 2019][1697241.010688] .nr_spread_over : 0 [Wed Feb 6 09:41:06 2019][1697241.015480] .nr_running : 0 [Wed Feb 6 09:41:06 2019][1697241.020271] .load : 0 [Wed Feb 6 09:41:06 2019][1697241.025066] .runnable_load_avg : 0 [Wed Feb 6 09:41:06 2019][1697241.029858] .blocked_load_avg : 2 [Wed Feb 6 09:41:06 2019][1697241.034650] .tg_load_avg : 0 [Wed Feb 6 09:41:06 2019][1697241.039442] .tg_load_contrib : 0 [Wed Feb 6 09:41:06 2019][1697241.044237] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:06 2019][1697241.049029] .tg->runnable_avg : 0 [Wed Feb 6 09:41:06 2019][1697241.053822] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:06 2019][1697241.058615] .throttled : 0 [Wed Feb 6 09:41:06 2019][1697241.063406] .throttle_count : 0 [Wed Feb 6 09:41:06 2019][1697241.068199] .avg->runnable_avg_sum : 170 [Wed Feb 6 09:41:06 2019][1697241.073166] .avg->runnable_avg_period : 47176 [Wed Feb 6 09:41:06 2019][1697241.078309] [Wed Feb 6 09:41:06 2019][1697241.079977] rt_rq[45]:/ [Wed Feb 6 09:41:06 2019][1697241.082603] .rt_nr_running : 0 [Wed Feb 6 09:41:06 2019][1697241.087397] .rt_throttled : 0 [Wed Feb 6 09:41:06 2019][1697241.092189] .rt_time : 0.000000 [Wed Feb 6 09:41:06 2019][1697241.097590] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:06 2019][1697241.103247] [Wed Feb 6 09:41:06 2019][1697241.104921] runnable tasks: [Wed Feb 6 09:41:06 2019][1697241.107895] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:06 2019][1697241.118841] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:06 2019][1697241.129698] systemd 1 7080992.358033 892809 120 0.000000 181417.748794 0.000000 1 / [Wed Feb 6 09:41:06 2019][1697241.141066] watchdog/45 236 -11.984651 424296 0 0.000000 147947.324837 0.000000 1 / [Wed Feb 6 09:41:06 2019][1697241.152398] migration/45 237 0.000000 11156 0 0.000000 141848.930626 0.000000 1 / [Wed Feb 6 09:41:06 2019][1697241.163732] ksoftirqd/45 238 7080645.260085 1973350 120 0.000000 26038.392765 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.175070] kworker/45:0H 240 69017.343357 10 100 0.000000 0.036428 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.186416] kworker/45:2 7817 7080990.682216 253248 120 0.000000 23982.043286 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.197775] kworker/45:1H 117225 7080645.246253 1233631 100 0.000000 4191.902567 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.209210] ptlrpcd_01_06 126459 7080990.711772 4022381 120 0.000000 59687.051987 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.220639] ldlm_cn01_001 126599 7081000.421741 10109755 120 0.000000 402628.746433 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.232068] mdt_io01_001 126669 4742691.426211 3020095 120 0.000000 123263.207789 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.243494] osp-syn-27-3 126784 7080990.580495 127776 120 0.000000 3605.178087 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.254922] osp-syn-18-1 126921 7080990.689900 198641 120 0.000000 4754.268966 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.266363] mdt_rdpg01_004 128484 7052567.329637 9997539 120 0.000000 440667.892602 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.277789] mdt_rdpg01_005 128517 7081000.742544 11798470 120 0.000000 505187.957364 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.289212] mdt01_012 128538 7080877.767676 24162097 120 0.000000 864085.634102 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.300639] mdt01_024 129254 6998165.484736 33765823 120 0.000000 1585973.034112 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.312058] mdt01_026 129256 7078923.634675 39494489 120 0.000000 1905462.157309 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.323481 38213127 120 0.000000 2260810.822093 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.334905] mdt_io01_025 129284 4741457.094936 1721781 120 0.000000 70329.163247 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.346326] mdt01_030 129372 7078923.632491 34582012 120 0.000000 1892358.404091 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.357748] mdt01_032 129374 7078923.631910 39894175 120 0.000000 2040847.100397 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.369171] mdt01_033 129795 6998165.483904 31886539 120 0.000000 1601555.155000 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.380610] kworker/45:0 103954 7059266.891008 4 120 0.000000 0.047870 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.392078] kworker/45:1 25099 7059410.251868 7 120 0.000000 0.073619 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.403413] kworker/u256:1 25137 7080990.719435 773 120 0.000000 10.642178 0.000000 1 / [Wed Feb 6 09:41:07 2019][1697241.414760] [Wed Feb 6 09:41:07 2019][1697241.416433] cpu#46, 1996.233 MHz [Wed Feb 6 09:41:07 2019][1697241.419841] .nr_running : 0 [Wed Feb 6 09:41:07 2019][1697241.424634] .load : 0 [Wed Feb 6 09:41:07 2019][1697241.429426] .nr_switches : 258666639 [Wed Feb 6 09:41:07 2019][1697241.434911] .nr_load_updates : 111853276 [Wed Feb 6 09:41:07 2019][1697241.440399] .nr_uninterruptible : 16074 [Wed Feb 6 09:41:07 2019][1697241.445539] .next_balance : 5991.894216 [Wed Feb 6 09:41:07 2019][1697241.451198] .curr->pid : 0 [Wed Feb 6 09:41:07 2019][1697241.455990] .clock : 1697241455.943692 [Wed Feb 6 09:41:07 2019][1697241.462170] .cpu_load[0] : 0 [Wed Feb 6 09:41:07 2019][1697241.466962] .cpu_load[1] : 0 [Wed Feb 6 09:41:07 2019][1697241.471753] .cpu_load[2] : 0 [Wed Feb 6 09:41:07 2019][1697241.476548] .cpu_load[3] : 0 [Wed Feb 6 09:41:07 2019][1697241.481341] .cpu_load[4] : 0 [Wed Feb 6 09:41:07 2019][1697241.486134] .avg_idle : 7010171 [Wed Feb 6 09:41:07 2019][1697241.491446] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:07 2019][1697241.496759] [Wed Feb 6 09:41:07 2019][1697241.498428] cfs_rq[46]:/ [Wed Feb 6 09:41:07 2019][1697241.501139] .exec_clock : 0.000000 [Wed Feb 6 09:41:07 2019][1697241.506539] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:07 2019][1697241.511939] .min_vruntime : 3450069.790707 [Wed Feb 6 09:41:07 2019][1697241.517859] .max_vruntime : 0.000001 [Wed Feb 6 09:41:07 2019][1697241.523258] .spread : 0.000000 [Wed Feb 6 09:41:07 2019][1697241.528658] .spread0 : -5393351.554797 [Wed Feb 6 09:41:07 2019][1697241.534664] .nr_spread_over : 0 [Wed Feb 6 09:41:07 2019][1697241.539455] .nr_running : 0 [Wed Feb 6 09:41:07 2019][1697241.544250] .load : 0 [Wed Feb 6 09:41:07 2019][1697241.549040] .runnable_load_avg : 0 [Wed Feb 6 09:41:07 2019][1697241.553835] .blocked_load_avg : 8 [Wed Feb 6 09:41:07 2019][1697241.558629] .tg_load_avg : 0 [Wed Feb 6 09:41:07 2019][1697241.563419] .tg_load_contrib : 0 [Wed Feb 6 09:41:07 2019][1697241.568212] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:07 2019][1697241.573006] .tg->runnable_avg : 0 [Wed Feb 6 09:41:07 2019][1697241.577798] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:07 2019][1697241.582590] .throttled : 0 [Wed Feb 6 09:41:07 2019][1697241.587382] .throttle_count : 0 [Wed Feb 6 09:41:07 2019][1697241.592176] .avg->runnable_avg_sum : 572 [Wed Feb 6 09:41:07 2019][1697241.597143] .avg->runnable_avg_period : 46730 [Wed Feb 6 09:41:07 2019][1697241.602283] [Wed Feb 6 09:41:07 2019][1697241.603955] rt_rq[46]:/ [Wed Feb 6 09:41:07 2019][1697241.606580] .rt_nr_running : 0 [Wed Feb 6 09:41:07 2019][1697241.611372] .rt_throttled : 0 [Wed Feb 6 09:41:07 2019][1697241.616167] .rt_time : 0.000000 [Wed Feb 6 09:41:07 2019][1697241.621565] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:07 2019][1697241.627225] [Wed Feb 6 09:41:07 2019][1697241.628896] runnable tasks: [Wed Feb 6 09:41:07 2019][1697241.631868] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:07 2019][1697241.642816] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:07 2019][1697241.653705] watchdog/46 241 -11.986806 424300 0 0.000000 73783.602540 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.665036] migration/46 242 0.000000 1533 0 0.000000 190122.125322 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.676374] ksoftirqd/46 243 3449545.347080 1809803 120 0.000000 29637.872906 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.687709] kworker/46:0H 245 7122.260333 10 100 0.000000 0.030658 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.699077] kworker/46:1H 40156 3441023.079643 1104651 100 0.000000 5322.987029 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.710422] ptlrpcd_02_02 126467 3450057.000047 6897199 120 0.000000 88206.273676 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.721840] ptlrpcd_02_08 126473 3450057.083425 6840523 120 0.000000 83763.546319 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.733262] ptlrpcd_02_09 126474 3450057.018322 6832780 120 0.000000 85097.656299 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.744694] mdt02_002 126631 3440568.578924 28433487 120 0.000000 1188854.349209 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.756116] mdt_rdpg02_000 126639 3439576.551646 5068043 120 0.000000 246045.002420 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.767542] mdt_io02_002 126673 1834846.923082 891140 120 0.000000 33536.335368 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.778968] osp-syn-20-3 126758 3450038.571225 128176 120 0.000000 4324.383482 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.790387] osp-syn-29-3 126776 3450046.268674 128155 120 0.000000 5195.407680 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.801808] osp-pre-29-3 126777 3450046.249959 214638 120 0.000000 6680.051343 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.813231] osp-pre-1-3 126779 3450039.966332 214593 120 0.000000 7803.681566 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.824662] osp-pre-32-1 126934 3450046.682665 213947 120 0.000000 7258.024874 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.836090] mdt02_004 127135 3360860.424053 24902324 120 0.000000 1038674.908380 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.847513] mdt_out02_003 127237 3369285.548651 18025508 120 0.000000 282754.306284 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.858931] mdt02_008 127248 3450073.748315 26228871 120 0.000000 1170828.983851 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.870355] mdt02_011 127501 3445930.340915 25403344 120 0.000000 1234537.505249 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.881784] mdt02_014 129247 3369310.617792 26844496 120 0.000000 1357067.355539 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.893213] mdt_io02_020 129292 1707950.621209 639462 120 0.000000 23886.628790 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.904631] mdt02_023 129595 3369285.548430 22845886 120 0.000000 1694953.758019 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.916053] mdt02_025 129805 3440568.575819 24532178 120 0.000000 1223344.199491 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.927493] kworker/46:1 10785 3450056.972745 25028 120 0.000000 11726.627046 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.938837] mdt02_034 17859 3369857.312661 118561 120 0.000000 168130.819298 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.950177] mdt02_036 17908 3360860.423652 132326 120 0.000000 163006.527740 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.961509] mdt02_037 17909 3360860.423812 3159 120 0.000000 3640.635839 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.972855] mdt02_053 18270 3360860.423542 90677 120 0.000000 71628.377538 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.984192] mdt02_059 18661 3361506.130792 5204 120 0.000000 8469.473095 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697241.995538] kworker/46:0 24693 3364805.961680 8 120 0.000000 1407.152322 0.000000 2 / [Wed Feb 6 09:41:07 2019][1697242.006901] [Wed Feb 6 09:41:07 2019][1697242.008573] cpu#47, 1996.233 MHz [Wed Feb 6 09:41:07 2019][1697242.011982] .nr_running : 0 [Wed Feb 6 09:41:07 2019][1697242.016773] .load : 0 [Wed Feb 6 09:41:07 2019][1697242.021566] .nr_switches : 100722802 [Wed Feb 6 09:41:07 2019][1697242.027054] .nr_load_updates : 56995175 [Wed Feb 6 09:41:07 2019][1697242.032461] .nr_uninterruptible : 1209 [Wed Feb 6 09:41:07 2019][1697242.037514] .next_balance : 5991.894810 [Wed Feb 6 09:41:07 2019][1697242.043173] .curr->pid : 0 [Wed Feb 6 09:41:07 2019][1697242.047965] .clock : 1697242047.949404 [Wed Feb 6 09:41:07 2019][1697242.054147] .cpu_load[0] : 0 [Wed Feb 6 09:41:07 2019][1697242.058949] .cpu_load[1] : 0 [Wed Feb 6 09:41:07 2019][1697242.063748] .cpu_load[2] : 192 [Wed Feb 6 09:41:07 2019][1697242.068715] .cpu_load[3] : 56 [Wed Feb 6 09:41:07 2019][1697242.073594] .cpu_load[4] : 28 [Wed Feb 6 09:41:07 2019][1697242.078475] .avg_idle : 1846644 [Wed Feb 6 09:41:07 2019][1697242.083795] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:07 2019][1697242.089107] [Wed Feb 6 09:41:07 2019][1697242.090776] cfs_rq[47]:/ [Wed Feb 6 09:41:07 2019][1697242.093489] .exec_clock : 0.000000 [Wed Feb 6 09:41:07 2019][1697242.098887] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:07 2019][1697242.104288] .min_vruntime : 1450432.362447 [Wed Feb 6 09:41:07 2019][1697242.110208] .max_vruntime : 0.000001 [Wed Feb 6 09:41:07 2019][1697242.115605] .spread : 0.000000 [Wed Feb 6 09:41:07 2019][1697242.121006] .spread0 : -7392989.102025 [Wed Feb 6 09:41:07 2019][1697242.127013] .nr_spread_over : 0 [Wed Feb 6 09:41:07 2019][1697242.131803] .nr_running : 0 [Wed Feb 6 09:41:07 2019][1697242.136596] .load : 0 [Wed Feb 6 09:41:07 2019][1697242.141389] .runnable_load_avg : 0 [Wed Feb 6 09:41:07 2019][1697242.146182] .blocked_load_avg : 2 [Wed Feb 6 09:41:07 2019][1697242.150976] .tg_load_avg : 0 [Wed Feb 6 09:41:07 2019][1697242.155766] .tg_load_contrib : 0 [Wed Feb 6 09:41:07 2019][1697242.160560] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:07 2019][1697242.165353] .tg->runnable_avg : 0 [Wed Feb 6 09:41:07 2019][1697242.170147] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:07 2019][1697242.174938] .throttled : 0 [Wed Feb 6 09:41:07 2019][1697242.179732] .throttle_count : 0 [Wed Feb 6 09:41:08 2019][1697242.184523] .avg->runnable_avg_sum : 166 [Wed Feb 6 09:41:08 2019][1697242.189490] .avg->runnable_avg_period : 46555 [Wed Feb 6 09:41:08 2019][1697242.194631] [Wed Feb 6 09:41:08 2019][1697242.196303] rt_rq[47]:/ [Wed Feb 6 09:41:08 2019][1697242.198927] .rt_nr_running : 0 [Wed Feb 6 09:41:08 2019][1697242.203720] .rt_throttled : 0 [Wed Feb 6 09:41:08 2019][1697242.208515] .rt_time : 0.000000 [Wed Feb 6 09:41:08 2019][1697242.213914] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:08 2019][1697242.219573] [Wed Feb 6 09:41:08 2019][1697242.221244] runnable tasks: [Wed Feb 6 09:41:08 2019][1697242.224217] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:08 2019][1697242.235164] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:08 2019][1697242.246050] watchdog/47 246 -11.986395 424309 0 0.000000 59563.413107 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.257386] migration/47 247 0.000000 2543 0 0.000000 120630.842913 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.268721] ksoftirqd/47 248 1450184.964136 147668 120 0.000000 4584.105377 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.280057] kworker/47:0H 250 4654.973800 10 100 0.000000 0.035635 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.291399] kworker/47:1 7285 1450410.454326 59893 120 0.000000 6491.055844 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.302735] ccp-3-q3 9996 2198.676014 2 120 0.000000 0.008386 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.314065] ccp-3-q4 9997 2210.683612 2 120 0.000000 0.007604 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.325411] polkitd 16810 3642.289479 3 120 0.000000 0.077206 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.336752] qmgr 18610 1442538.552671 10309 120 0.000000 525.314408 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.348088] kworker/47:1H 46363 1450184.948660 207927 100 0.000000 928.684946 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.359425] ptlrpc_hr03_002 126401 1450414.439811 1678762 120 0.000000 14336.623649 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.370845] kiblnd_sd_03_02 126435 1450433.855229 87904376 120 0.000000 652666.297281 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.382272] ptlrpcd_03_05 126482 1450412.859945 5615407 120 0.000000 82199.372543 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.393701] ll_cfg_requeue 126610 1255708.764637 289 120 0.000000 10.552937 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.405121] mdt_rdpg03_000 126641 1450421.864104 2045679 120 0.000000 79887.986881 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.416548] osp-syn-14-3 126716 1450395.324005 128824 120 0.000000 5456.350141 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.427969] osp-syn-33-3 126756 1450420.369970 128517 120 0.000000 3791.152047 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.439390] osp-pre-35-3 126793 1450422.118677 214676 120 0.000000 8496.466877 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.450814] osp-pre-11-1 126904 1450395.185221 213773 120 0.000000 8942.746160 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.462234] osp-syn-9-1 126913 1450409.642401 198075 120 0.000000 4788.867661 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.473703] ldlm_cn03_013 13042 1450361.438475 1217107 120 0.000000 92696.529825 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.485037] ldlm_cn03_014 13459 1435807.522352 794160 120 0.000000 124760.922451 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.496374] mdt03_024 17848 1254498.160134 56404 120 0.000000 3996.073056 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.507708] mdt03_028 17890 1450395.117374 44562 120 0.000000 5443.115669 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.519048] mdt03_037 17968 1254747.131077 95663 120 0.000000 308511.159749 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.530380] mdt03_039 17977 1254965.832700 148652 120 0.000000 308627.038442 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.541728] kworker/47:0 24086 1062455.317370 3 120 0.000000 0.016402 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.553065] ldlm_cn03_028 24610 1450329.074010 4323 120 0.000000 61236.649013 0.000000 3 / [Wed Feb 6 09:41:08 2019][1697242.564410] [Wed Feb 6 09:41:08 2019][1697242.566954] INFO: rcu_sched detected stalls on CPUs/tasks: { 2 6 7 32 36} (detected by 32, t=7 jiffies, g=72717088, c=72717087, q=427542) [Wed Feb 6 09:41:08 2019][1697242.579678] Task dump for CPU 2: [Wed Feb 6 09:41:08 2019][1697242.583085] ldlm_bl_01 R running task 0 126607 2 0x00000080 [Wed Feb 6 09:41:08 2019][1697242.590443] Call Trace: [Wed Feb 6 09:41:08 2019][1697242.593115] [] ? ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.601746] [] ? ldlm_cancel_locks_for_export_cb+0x3f/0x50 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.609843] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [Wed Feb 6 09:41:08 2019][1697242.617439] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.626072] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.634683] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [Wed Feb 6 09:41:08 2019][1697242.642194] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.649786] [] ? ldlm_bl_thread_main+0x4c8/0x700 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.657008] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:41:08 2019][1697242.662787] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.670352] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:41:08 2019][1697242.675577] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:08 2019][1697242.681844] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:41:08 2019][1697242.688630] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:08 2019][1697242.694895] Task dump for CPU 7: [Wed Feb 6 09:41:08 2019][1697242.698300] ldlm_bl_09 R running task 0 10650 2 0x00000080 [Wed Feb 6 09:41:08 2019][1697242.705571] Call Trace: [Wed Feb 6 09:41:08 2019][1697242.708224] [] ? ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.716855] [] ? ldlm_cancel_locks_for_export_cb+0x3f/0x50 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.724950] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [Wed Feb 6 09:41:08 2019][1697242.732542] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.741174] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.749787] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [Wed Feb 6 09:41:08 2019][1697242.757296] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.764890] [] ? ldlm_bl_thread_main+0x4c8/0x700 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.772109] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:41:08 2019][1697242.777880] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.785445] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:41:08 2019][1697242.790671] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:08 2019][1697242.796937] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:41:08 2019][1697242.803723] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:08 2019][1697242.809989] Task dump for CPU 32: [Wed Feb 6 09:41:08 2019][1697242.813483] swapper/32 R running task 0 0 1 0x00000008 [Wed Feb 6 09:41:08 2019][1697242.820753] Call Trace: [Wed Feb 6 09:41:08 2019][1697242.823380] [] sched_show_task+0xa8/0x110 [Wed Feb 6 09:41:08 2019][1697242.829847] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 09:41:08 2019][1697242.835420] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 09:41:08 2019][1697242.841511] [] rcu_check_callbacks+0x6f2/0x730 [Wed Feb 6 09:41:08 2019][1697242.847778] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 09:41:08 2019][1697242.854043] [] update_process_times+0x46/0x80 [Wed Feb 6 09:41:08 2019][1697242.860222] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 09:41:08 2019][1697242.866143] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 09:41:08 2019][1697242.871974] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 09:41:08 2019][1697242.878240] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 09:41:08 2019][1697242.884248] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 09:41:08 2019][1697242.890948] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 09:41:08 2019][1697242.897474] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 09:41:08 2019][1697242.903823] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:08 2019][1697242.910638] [] ? native_safe_halt+0x6/0x10 [Wed Feb 6 09:41:08 2019][1697242.916556] [] default_idle+0x1e/0xc0 [Wed Feb 6 09:41:08 2019][1697242.922044] [] arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:08 2019][1697242.927617] [] cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:08 2019][1697242.933708] [] start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:08 2019][1697242.939630] [] start_cpu+0x5/0x14 [Wed Feb 6 09:41:08 2019][1697242.944766] Task dump for CPU 36: [Wed Feb 6 09:41:08 2019][1697242.948258] ldlm_bl_08 R running task 0 10649 2 0x00000080 [Wed Feb 6 09:41:08 2019][1697242.955528] Call Trace: [Wed Feb 6 09:41:08 2019][1697242.958183] [] ? ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.966813] [] ? ldlm_cancel_locks_for_export_cb+0x3f/0x50 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.974909] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [Wed Feb 6 09:41:08 2019][1697242.982502] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.991131] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697242.999746] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [Wed Feb 6 09:41:08 2019][1697243.007254] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697243.014847] [] ? ldlm_bl_thread_main+0x4c8/0x700 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697243.022067] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:41:08 2019][1697243.027841] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:41:08 2019][1697243.035403] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:41:08 2019][1697243.040630] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:08 2019][1697243.046896] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:41:08 2019][1697243.053681] [] ? insert_kthread_work+0x40/0x40