[Fri Jan 25 11:42:10 2019][667006.487552] LNet: HW NUMA nodes: 4, HW CPU cores: 48, npartitions: 4 [Fri Jan 25 11:42:10 2019][667006.495364] alg: No test for adler32 (adler32-zlib) [Fri Jan 25 11:42:10 2019][667007.297917] Lustre: Lustre: Build Version: 2.12.0 [Fri Jan 25 11:42:11 2019][667007.419464] LNet: Using FastReg for registration [Fri Jan 25 11:42:11 2019][667007.436329] LNet: Added LNI 10.0.10.51@o2ib7 [8/256/0/180] [Fri Jan 25 11:42:12 2019][667008.569167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [Fri Jan 25 11:42:12 2019][667008.895347] Lustre: MGS: Connection restored to 92c3d183-6c80-ac05-2c13-91ea1d1f805c (at 0@lo) [Fri Jan 25 11:42:17 2019][667014.273677] Lustre: MGS: Connection restored to 6caeea3e-5b74-9a8d-d8ac-3bf9f06c410e (at 10.9.0.1@o2ib4) [Fri Jan 25 11:42:19 2019][667016.177153] Lustre: MGS: Connection restored to ea1f2e42-f42b-9f18-7c47-4ca0e038dce5 (at 10.0.10.103@o2ib7) [Fri Jan 25 11:42:19 2019][667016.186980] Lustre: Skipped 1 previous similar message [Fri Jan 25 11:42:21 2019][667018.192644] Lustre: MGS: Connection restored to 352bc83a-a73d-57af-a05c-bb6431f8c6a2 (at 10.0.10.105@o2ib7) [Fri Jan 25 11:42:21 2019][667018.202477] Lustre: Skipped 4 previous similar messages [Fri Jan 25 11:42:26 2019][667023.174267] Lustre: MGS: Connection restored to 92348c57-fd1b-b54c-db37-9f18c062f166 (at 10.0.10.101@o2ib7) [Fri Jan 25 11:42:26 2019][667023.184097] Lustre: Skipped 3 previous similar messages [Fri Jan 25 11:42:44 2019][667040.879010] Lustre: MGS: Connection restored to 88130480-2e5c-e317-8933-a7c5aa30a9a9 (at 10.9.0.2@o2ib4) [Fri Jan 25 11:42:58 2019][667054.800496] LDISKFS-fs (dm-3): file extents enabled[667054.804480] LDISKFS-fs (dm-4): file extents enabled, maximum tree depth=5 [Fri Jan 25 11:42:58 2019] [Fri Jan 25 11:42:58 2019][667054.812201] , maximum tree depth=5 [Fri Jan 25 11:42:58 2019][667055.061274] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [Fri Jan 25 11:42:58 2019][667055.073057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [Fri Jan 25 11:43:00 2019][667056.593638] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Jan 25 11:43:00 2019][667056.618437] LustreError: 11-0: fir-OST0012-osc-MDT0000: operation ost_connect to node 10.0.10.103@o2ib7 failed: rc = -16 [Fri Jan 25 11:43:00 2019][667056.670978] Lustre: fir-MDT0000: Imperative Recovery not enabled, recovery window 300-900 [Fri Jan 25 11:43:00 2019][667056.740990] Lustre: fir-MDD0000: changelog on [Fri Jan 25 11:43:00 2019][667056.746561] Lustre: fir-MDT0000: in recovery but waiting for the first client to connect [Fri Jan 25 11:43:01 2019][667057.406061] Lustre: MGS: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) [Fri Jan 25 11:43:01 2019][667057.414072] Lustre: Skipped 3 previous similar messages [Fri Jan 25 11:43:01 2019][667057.547582] Lustre: fir-MDT0002: Will be in recovery for at least 5:00, or until 8 clients reconnect [Fri Jan 25 11:43:02 2019][667058.945308] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.0.1@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Jan 25 11:43:02 2019][667058.962502] LustreError: Skipped 1 previous similar message [Fri Jan 25 11:43:05 2019][667062.054940] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.101.59@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Jan 25 11:43:15 2019][667072.187405] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.0.10.103@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Jan 25 11:43:21 2019][667078.232126] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.0.10.106@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Jan 25 11:43:21 2019][667078.249590] LustreError: Skipped 39 previous similar messages [Fri Jan 25 11:43:25 2019][667081.869368] LustreError: 11-0: fir-OST0012-osc-MDT0000: operation ost_connect to node 10.0.10.103@o2ib7 failed: rc = -16 [Fri Jan 25 11:43:25 2019][667081.880346] LustreError: Skipped 84 previous similar messages [Fri Jan 25 11:43:40 2019][667097.263075] Lustre: fir-MDT0000: Connection restored to 10.0.10.108@o2ib7 (at 10.0.10.108@o2ib7) [Fri Jan 25 11:43:40 2019][667097.271995] Lustre: Skipped 61 previous similar messages [Fri Jan 25 11:43:50 2019][667106.958081] LustreError: 11-0: fir-OST0012-osc-MDT0000: operation ost_connect to node 10.0.10.103@o2ib7 failed: rc = -16 [Fri Jan 25 11:43:50 2019][667106.969073] LustreError: Skipped 39 previous similar messages [Fri Jan 25 11:43:51 2019][667107.688573] Lustre: fir-MDT0000: Recovery over after 0:50, of 10 clients 10 recovered and 0 were evicted. [Fri Jan 25 13:08:06 2019][672163.046605] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Fri Jan 25 15:37:50 2019][681146.737059] Lustre: MGS: Connection restored to e421d59a-4482-eafc-ea85-c173e2c58e62 (at 10.8.3.11@o2ib6) [Fri Jan 25 15:37:50 2019][681146.746714] Lustre: Skipped 45 previous similar messages [Fri Jan 25 15:54:06 2019][682123.558844] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Fri Jan 25 16:06:31 2019][682868.101205] perf: interrupt took too long (2504 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 [Fri Jan 25 16:10:10 2019][683087.438505] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Fri Jan 25 16:10:10 2019][683087.450154] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Fri Jan 25 16:16:32 2019][683469.368141] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Fri Jan 25 16:18:01 2019][683558.622610] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Fri Jan 25 16:28:07 2019][684163.834934] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Fri Jan 25 16:28:07 2019][684163.846585] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Fri Jan 25 16:29:18 2019][684234.935559] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Fri Jan 25 16:29:18 2019][684234.947216] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Fri Jan 25 17:10:55 2019][686731.821969] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:10:55 2019][686731.833626] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Fri Jan 25 17:26:40 2019][687676.879193] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:28:01 2019][687758.396463] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:37:52 2019][688349.270702] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Fri Jan 25 17:37:52 2019][688349.282362] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Fri Jan 25 17:59:52 2019][689668.855034] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Fri Jan 25 18:01:46 2019][689783.508669] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Fri Jan 25 18:01:46 2019][689783.520329] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Fri Jan 25 18:21:58 2019][690995.836911] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Fri Jan 25 19:28:34 2019][694991.470447] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Fri Jan 25 19:43:46 2019][695903.537783] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Fri Jan 25 19:56:55 2019][696692.307428] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Fri Jan 25 20:44:49 2019][699566.183704] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Fri Jan 25 20:44:49 2019][699566.195362] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Fri Jan 25 21:17:35 2019][701533.036066] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Fri Jan 25 21:27:15 2019][702112.596127] Lustre: 5929:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1548480466/real 1548480466] req@ffff91456b750c00 x1623664064378224/t0(0) o41->fir-MDT0003-osp-MDT0002@10.0.10.52@o2ib7:24/4 lens 224/368 e 0 to 1 dl 1548480475 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Jan 25 21:27:15 2019][702112.624248] Lustre: fir-MDT0003-osp-MDT0002: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Fri Jan 25 21:27:15 2019][702112.640316] Lustre: Skipped 1 previous similar message [Fri Jan 25 21:27:16 2019][702113.224276] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Fri Jan 25 21:27:16 2019][702113.235252] Lustre: fir-MDT0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) [Fri Jan 25 21:27:16 2019][702113.243951] Lustre: Skipped 11 previous similar messages [Fri Jan 25 21:27:23 2019][702121.156018] Lustre: MGS: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Fri Jan 25 21:27:23 2019][702121.156039] Lustre: fir-MDT0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) [Fri Jan 25 21:27:23 2019][702121.156041] Lustre: Skipped 1 previous similar message [Fri Jan 25 21:27:24 2019][702121.180204] Lustre: Skipped 5 previous similar messages [Fri Jan 25 21:27:28 2019][702125.911733] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.0.10.104@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Jan 25 21:27:28 2019][702125.929192] LustreError: Skipped 35 previous similar messages [Fri Jan 25 21:27:32 2019][702129.591099] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.6.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Jan 25 21:27:32 2019][702129.608385] LustreError: Skipped 1 previous similar message [Fri Jan 25 21:27:35 2019][702133.149733] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.0.10.101@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Jan 25 21:27:35 2019][702133.167191] LustreError: Skipped 9 previous similar messages [Fri Jan 25 21:27:40 2019][702137.980942] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Jan 25 21:27:40 2019][702137.997271] LustreError: Skipped 27 previous similar messages [Fri Jan 25 21:27:59 2019][702156.306707] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.3.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Jan 25 21:27:59 2019][702156.323992] LustreError: Skipped 2 previous similar messages [Fri Jan 25 21:28:05 2019][702163.069672] Lustre: fir-MDT0003-osp-MDT0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) [Fri Jan 25 21:28:05 2019][702163.079422] Lustre: Skipped 4 previous similar messages [Fri Jan 25 21:28:22 2019][702179.505503] Lustre: MGS: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Fri Jan 25 21:28:22 2019][702179.505558] Lustre: fir-MDT0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) [Fri Jan 25 21:28:22 2019][702179.505560] Lustre: Skipped 3 previous similar messages [Fri Jan 25 21:28:22 2019][702179.529768] Lustre: Skipped 1 previous similar message [Fri Jan 25 22:53:32 2019][707290.206899] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Fri Jan 25 22:57:38 2019][707535.569775] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Fri Jan 25 23:05:10 2019][707987.738357] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Fri Jan 25 23:16:41 2019][708678.637226] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Fri Jan 25 23:38:05 2019][709962.739846] perf: interrupt took too long (3132 > 3130), lowering kernel.perf_event_max_sample_rate to 63000 [Fri Jan 25 23:41:06 2019][710143.516950] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Fri Jan 25 23:50:39 2019][710717.031550] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Fri Jan 25 23:50:39 2019][710717.043202] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 00:51:08 2019][714346.173620] Lustre: fir-MDT0000: Client f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6) reconnecting [Sat Jan 26 00:51:08 2019][714346.173656] Lustre: fir-MDT0002: Connection restored to 6aadd60a-4be2-83fa-2199-3d4225023e35 (at 10.8.6.20@o2ib6) [Sat Jan 26 00:51:08 2019][714346.173657] Lustre: Skipped 1 previous similar message [Sat Jan 26 00:51:08 2019][714346.174174] LustreError: 11093:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff9139f678c850 x1623670968620880/t0(0) o4->f9afcafe-b7fa-076a-0d87-fc0f78fd02d0@10.8.6.20@o2ib6:584/0 lens 488/448 e 0 to 0 dl 1548492749 ref 1 fl Interpret:/0/0 rc 0/0 [Sat Jan 26 00:51:08 2019][714346.174196] Lustre: fir-MDT0002: Bulk IO write error with f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6), client will retry: rc = -110 [Sat Jan 26 00:51:08 2019][714346.236458] Lustre: Skipped 1 previous similar message [Sat Jan 26 00:52:57 2019][714454.566845] Lustre: 6985:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1548492808/real 1548492808] req@ffff912e3ae0c500 x1623664751288816/t0(0) o104->fir-MDT0000@10.8.3.11@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1548492817 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Sat Jan 26 00:52:57 2019][714454.594013] Lustre: 6985:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [Sat Jan 26 00:53:09 2019][714466.836852] Lustre: fir-MDT0002: Client 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6) reconnecting [Sat Jan 26 00:53:09 2019][714466.836915] Lustre: MGS: Received new LWP connection from 10.8.3.11@o2ib6, removing former export from same NID [Sat Jan 26 00:53:09 2019][714466.836938] Lustre: MGS: Connection restored to e421d59a-4482-eafc-ea85-c173e2c58e62 (at 10.8.3.11@o2ib6) [Sat Jan 26 00:53:09 2019][714466.836940] Lustre: Skipped 1 previous similar message [Sat Jan 26 00:53:09 2019][714467.129555] LustreError: 6909:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff9129a3c3fc50 x1623568220881696/t0(0) o4->9b3a0481-8c2f-a606-18ad-7a06b856368e@10.8.3.11@o2ib6:707/0 lens 488/448 e 0 to 0 dl 1548492872 ref 1 fl Interpret:/0/0 rc 0/0 [Sat Jan 26 00:53:09 2019][714467.129588] Lustre: fir-MDT0002: Bulk IO write error with 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6), client will retry: rc = -110 [Sat Jan 26 00:53:09 2019][714467.129590] Lustre: Skipped 2 previous similar messages [Sat Jan 26 00:53:09 2019][714467.171972] LustreError: 6909:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 3 previous similar messages [Sat Jan 26 00:53:52 2019][714510.123967] Lustre: MGS: Received new LWP connection from 10.8.3.11@o2ib6, removing former export from same NID [Sat Jan 26 00:53:52 2019][714510.134168] Lustre: MGS: Connection restored to e421d59a-4482-eafc-ea85-c173e2c58e62 (at 10.8.3.11@o2ib6) [Sat Jan 26 00:53:52 2019][714510.143826] Lustre: Skipped 1 previous similar message [Sat Jan 26 01:48:13 2019][717770.693050] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Sat Jan 26 01:48:13 2019][717770.703221] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.201@o2ib7 (5): c: 4, oc: 0, rc: 8 [Sat Jan 26 01:48:14 2019][717771.694087] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Sat Jan 26 01:48:14 2019][717771.704264] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 2 previous similar messages [Sat Jan 26 01:48:14 2019][717771.714430] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.209@o2ib7 (6): c: 7, oc: 0, rc: 8 [Sat Jan 26 01:48:14 2019][717771.726414] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 2 previous similar messages [Sat Jan 26 01:48:14 2019][717771.941089] Lustre: 5934:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1548496127/real 0] req@ffff9159562e5a00 x1623664975827104/t0(0) o13->fir-OST0015-osc-MDT0002@10.0.10.104@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1548496134 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Sat Jan 26 01:48:14 2019][717771.968423] Lustre: 5934:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message [Sat Jan 26 01:48:14 2019][717771.978084] Lustre: fir-OST0015-osc-MDT0002: Connection to fir-OST0015 (at 10.0.10.104@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Sat Jan 26 01:48:14 2019][717771.994231] Lustre: Skipped 2 previous similar messages [Sat Jan 26 01:48:14 2019][717772.195866] Lustre: fir-MDT0002: Client 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) reconnecting [Sat Jan 26 01:48:14 2019][717772.205980] Lustre: fir-MDT0002: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Sat Jan 26 01:48:15 2019][717772.774425] Lustre: fir-MDT0000: Client 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) reconnecting [Sat Jan 26 01:48:15 2019][717772.784427] Lustre: Skipped 1 previous similar message [Sat Jan 26 01:48:15 2019][717772.789679] Lustre: fir-MDT0000: Connection restored to 6caeea3e-5b74-9a8d-d8ac-3bf9f06c410e (at 10.9.0.1@o2ib4) [Sat Jan 26 01:48:15 2019][717772.799932] Lustre: Skipped 1 previous similar message [Sat Jan 26 01:48:16 2019][717773.693128] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Sat Jan 26 01:48:16 2019][717773.703298] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (8): c: 3, oc: 0, rc: 8 [Sat Jan 26 01:48:18 2019][717775.921553] Lustre: MGS: Received new LWP connection from 10.8.2.5@o2ib6, removing former export from same NID [Sat Jan 26 01:48:18 2019][717775.931652] Lustre: MGS: Connection restored to 16f622df-9b71-a16e-7fd5-e6f7aaabcc11 (at 10.8.2.5@o2ib6) [Sat Jan 26 01:48:18 2019][717775.949186] Lustre: 5937:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1548496131/real 0] req@ffff91594979b000 x1623664975827728/t0(0) o13->fir-OST0028-osc-MDT0002@10.0.10.107@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1548496138 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Sat Jan 26 01:48:18 2019][717775.965195] Lustre: fir-MDT0003-osp-MDT0002: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Sat Jan 26 01:48:18 2019][717775.992578] Lustre: 5937:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [Sat Jan 26 01:48:19 2019][717777.002211] Lustre: 5937:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1548496132/real 0] req@ffff915949798900 x1623664975828432/t0(0) o13->fir-OST0024-osc-MDT0002@10.0.10.107@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1548496139 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Sat Jan 26 01:48:19 2019][717777.013217] Lustre: fir-OST0011-osc-MDT0000: Connection to fir-OST0011 (at 10.0.10.104@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Sat Jan 26 01:48:19 2019][717777.013218] Lustre: Skipped 11 previous similar messages [Sat Jan 26 01:48:19 2019][717777.051091] Lustre: 5937:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [Sat Jan 26 01:48:21 2019][717779.283690] Lustre: fir-MDT0002: Client 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) reconnecting [Sat Jan 26 01:48:21 2019][717779.293818] Lustre: fir-MDT0002: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Sat Jan 26 01:48:22 2019][717780.235299] LustreError: 6215:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff9149d5624c50 x1623623379077904/t0(0) o4->85c45b71-10aa-be9f-071b-9a9c50548db2@10.8.15.7@o2ib6:237/0 lens 488/448 e 0 to 0 dl 1548496177 ref 1 fl Interpret:/2/0 rc 0/0 [Sat Jan 26 01:48:22 2019][717780.259288] Lustre: fir-MDT0002: Bulk IO write error with 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6), client will retry: rc = -110 [Sat Jan 26 01:48:22 2019][717780.272416] Lustre: Skipped 1 previous similar message [Sat Jan 26 01:48:24 2019][717781.693323] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Sat Jan 26 01:48:24 2019][717781.703491] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 1 previous similar message [Sat Jan 26 01:48:24 2019][717781.713572] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (8): c: 4, oc: 0, rc: 8 [Sat Jan 26 01:48:24 2019][717781.725470] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 1 previous similar message [Sat Jan 26 01:48:28 2019][717785.597502] Lustre: fir-MDT0000: Client 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) reconnecting [Sat Jan 26 01:48:28 2019][717785.607591] Lustre: Skipped 1 previous similar message [Sat Jan 26 01:48:28 2019][717785.612864] Lustre: fir-MDT0000: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Sat Jan 26 01:48:28 2019][717785.623208] Lustre: Skipped 1 previous similar message [Sat Jan 26 01:48:28 2019][717786.125444] Lustre: 5936:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1548496141/real 0] req@ffff911f54e31e00 x1623664975829664/t0(0) o13->fir-OST0000-osc-MDT0000@10.0.10.101@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1548496148 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Sat Jan 26 01:48:28 2019][717786.152781] Lustre: 5936:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [Sat Jan 26 01:48:28 2019][717786.162526] Lustre: fir-OST0000-osc-MDT0000: Connection to fir-OST0000 (at 10.0.10.101@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Sat Jan 26 01:48:28 2019][717786.178676] Lustre: Skipped 7 previous similar messages [Sat Jan 26 01:48:29 2019][717787.122451] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [mdt_io01_012:7223] [Sat Jan 26 01:48:29 2019][717787.130365] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd[717787.157454] NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [mdt_io01_014:8952] [Sat Jan 26 01:48:29 2019][717787.157456] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:29 2019][717787.157530] CPU: 13 PID: 8952 Comm: mdt_io01_014 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:29 2019][717787.157531] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:29 2019][717787.157533] task: ffff91599d3d1040 ti: ffff9159f42a4000 task.ti: ffff9159f42a4000 [Sat Jan 26 01:48:29 2019][717787.157534] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x1ce/0x200 [Sat Jan 26 01:48:29 2019][717787.157542] RSP: 0018:ffff9159f42a7798 EFLAGS: 00000202 [Sat Jan 26 01:48:29 2019][717787.157543] RAX: 0000000000000001 RBX: ffff911e33293a30 RCX: 0000000000000001 [Sat Jan 26 01:48:29 2019][717787.157544] RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:29 2019][717787.157545] RBP: ffff9159f42a7798 R08: 0000000000000101 R09: ffffffffc12b0d1a [Sat Jan 26 01:48:29 2019][717787.157546] R10: ffff9139ff6df140 R11: ffffdbb24f22f600 R12: ffff9159f42a7738 [Sat Jan 26 01:48:29 2019][717787.157546] R13: ffff911e33293b18 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:29 2019][717787.157548] FS: 00007f277afbf700(0000) GS:ffff9139ff6c0000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:29 2019][717787.157549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:29 2019][717787.157550] CR2: 00007f2781327000 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:29 2019][717787.157551] Call Trace: [Sat Jan 26 01:48:29 2019][717787.157556] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:29 2019][717787.157560] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:29 2019][717787.157574] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:29 2019][717787.157586] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:29 2019][717787.157593] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Sat Jan 26 01:48:29 2019][717787.157599] [] ? crypto_create_tfm+0x32/0xd0 [Sat Jan 26 01:48:29 2019][717787.157609] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Sat Jan 26 01:48:29 2019][717787.157618] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:29 2019][717787.157625] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Sat Jan 26 01:48:29 2019][717787.157633] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Sat Jan 26 01:48:29 2019][717787.157647] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:29 2019][717787.157657] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Sat Jan 26 01:48:29 2019][717787.157667] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Sat Jan 26 01:48:29 2019][717787.157689] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Sat Jan 26 01:48:29 2019][717787.157742] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Sat Jan 26 01:48:29 2019][717787.157784] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Sat Jan 26 01:48:29 2019][717787.157821] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Sat Jan 26 01:48:29 2019][717787.157828] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:29 2019][717787.157830] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:29 2019][717787.157862] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Sat Jan 26 01:48:29 2019][717787.157903] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:29 2019][717787.157943] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:29 2019][717787.157951] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:30 2019][717787.157989] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.158026] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.158029] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:30 2019][717787.158032] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:30 2019][717787.158070] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.158106] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.158110] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:30 2019][717787.158112] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:30 2019][717787.158115] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:30 2019][717787.158117] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:30 2019][717787.158117] Code: 37 81 fe 00 01 00 00 74 f4 e9 93 fe ff ff 0f 1f 80 00 00 00 00 83 fa 01 75 11 0f 1f 00 e9 68 fe ff ff 0f 1f 00 85 c0 74 0c f3 90 <8b> 07 0f b6 c0 83 f8 03 75 f0 b8 01 00 00 00 66 89 07 5d c3 66 [Sat Jan 26 01:48:30 2019][717787.165452] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [mdt_io02_012:7222] [Sat Jan 26 01:48:30 2019][717787.165453] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:30 2019][717787.165524] CPU: 14 PID: 7222 Comm: mdt_io02_012 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:30 2019][717787.165525] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:30 2019][717787.165526] task: ffff9129fb998000 ti: ffff9159f1fa0000 task.ti: ffff9159f1fa0000 [Sat Jan 26 01:48:30 2019][717787.165527] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:30 2019][717787.165535] RSP: 0018:ffff9159f1fa3798 EFLAGS: 00000246 [Sat Jan 26 01:48:30 2019][717787.165535] RAX: 0000000000000000 RBX: ffff912d52184ef8 RCX: 0000000000710000 [Sat Jan 26 01:48:30 2019][717787.165536] RDX: ffff9139ff71b780 RSI: 0000000000890101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:30 2019][717787.165537] RBP: ffff9159f1fa3798 R08: ffff9149ff6db780 R09: 0000000000000000 [Sat Jan 26 01:48:30 2019][717787.165537] R10: ffff9149ff6df140 R11: ffffdbb2bf141000 R12: ffff9159f1fa3738 [Sat Jan 26 01:48:30 2019][717787.165538] R13: ffff912d52184fe0 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:30 2019][717787.165539] FS: 00007f21dc71d740(0000) GS:ffff9149ff6c0000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:30 2019][717787.165540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:30 2019][717787.165541] CR2: 0000000001e66020 CR3: 0000004031fb0000 CR4: 00000000003407e0 [Sat Jan 26 01:48:30 2019][717787.165542] Call Trace: [Sat Jan 26 01:48:30 2019][717787.165547] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:30 2019][717787.165552] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:30 2019][717787.165566] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:30 2019][717787.165577] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:30 2019][717787.165585] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Sat Jan 26 01:48:30 2019][717787.165589] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Sat Jan 26 01:48:30 2019][717787.165594] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:30 2019][717787.165606] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Sat Jan 26 01:48:30 2019][717787.165613] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:30 2019][717787.165619] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Sat Jan 26 01:48:30 2019][717787.165626] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Sat Jan 26 01:48:30 2019][717787.165639] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:30 2019][717787.165648] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Sat Jan 26 01:48:30 2019][717787.165656] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Sat Jan 26 01:48:30 2019][717787.165678] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Sat Jan 26 01:48:30 2019][717787.165735] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.165771] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.165803] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.165833] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.165835] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:30 2019][717787.165837] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:30 2019][717787.165863] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.165898] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.165931] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.165939] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:30 2019][717787.165971] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.165975] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:30 2019][717787.165977] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:30 2019][717787.166009] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.166040] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.166044] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:30 2019][717787.166046] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:30 2019][717787.166049] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:30 2019][717787.166050] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:30 2019][717787.166051] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:30 2019][717787.191454] NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [mdt_io01_006:6788] [Sat Jan 26 01:48:30 2019][717787.191455] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:30 2019][717787.191526] CPU: 17 PID: 6788 Comm: mdt_io01_006 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:30 2019][717787.191527] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:30 2019][717787.191528] task: ffff9159ed232080 ti: ffff9159f6e50000 task.ti: ffff9159f6e50000 [Sat Jan 26 01:48:30 2019][717787.191529] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:30 2019][717787.191538] RSP: 0018:ffff9159f6e53798 EFLAGS: 00000246 [Sat Jan 26 01:48:30 2019][717787.191539] RAX: 0000000000000000 RBX: ffff912d520c2990 RCX: 0000000000890000 [Sat Jan 26 01:48:30 2019][717787.191540] RDX: ffff9139ff69b780 RSI: 0000000000490101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:30 2019][717787.191540] RBP: ffff9159f6e53798 R08: ffff9139ff71b780 R09: 0000000000000000 [Sat Jan 26 01:48:30 2019][717787.191541] R10: ffff9139ff71f140 R11: ffffdbb251ee2800 R12: ffff9159f6e53738 [Sat Jan 26 01:48:30 2019][717787.191542] R13: ffff912d520c2a78 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:30 2019][717787.191543] FS: 00007f63fb2d1700(0000) GS:ffff9139ff700000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:30 2019][717787.191544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:30 2019][717787.191545] CR2: 00007f63fb343000 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:30 2019][717787.191546] Call Trace: [Sat Jan 26 01:48:30 2019][717787.191552] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:30 2019][717787.191556] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:30 2019][717787.191571] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:30 2019][717787.191583] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:30 2019][717787.191590] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Sat Jan 26 01:48:30 2019][717787.191595] [] ? crypto_create_tfm+0x32/0xd0 [Sat Jan 26 01:48:30 2019][717787.191607] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Sat Jan 26 01:48:30 2019][717787.191615] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:30 2019][717787.191622] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Sat Jan 26 01:48:30 2019][717787.191630] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Sat Jan 26 01:48:30 2019][717787.191644] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:30 2019][717787.191654] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Sat Jan 26 01:48:30 2019][717787.191664] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Sat Jan 26 01:48:30 2019][717787.191686] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Sat Jan 26 01:48:30 2019][717787.191739] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.191781] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.191818] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.191855] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.191861] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:30 2019][717787.191864] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:30 2019][717787.191897] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.191938] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.191978] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.191985] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:30 2019][717787.192023] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.192060] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.192064] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:30 2019][717787.192067] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:30 2019][717787.192104] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.192142] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:30 2019][717787.192146] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:30 2019][717787.192148] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:30 2019][717787.192151] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:30 2019][717787.192153] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:30 2019][717787.192153] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:31 2019][717787.225454] NMI watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [mdt_io01_016:8954] [Sat Jan 26 01:48:31 2019][717787.225455] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:31 2019][717787.225506] CPU: 21 PID: 8954 Comm: mdt_io01_016 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:31 2019][717787.225507] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:31 2019][717787.225508] task: ffff91599d3d0000 ti: ffff9159a9a24000 task.ti: ffff9159a9a24000 [Sat Jan 26 01:48:31 2019][717787.225509] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:31 2019][717787.225513] RSP: 0018:ffff9159a9a27798 EFLAGS: 00000246 [Sat Jan 26 01:48:31 2019][717787.225514] RAX: 0000000000000000 RBX: ffff912d521cf460 RCX: 0000000000a90000 [Sat Jan 26 01:48:31 2019][717787.225515] RDX: ffff9129ff09b780 RSI: 0000000001410101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:31 2019][717787.225516] RBP: ffff9159a9a27798 R08: ffff9139ff75b780 R09: 0000000000000000 [Sat Jan 26 01:48:31 2019][717787.225516] R10: ffff9139ff75f140 R11: ffffdbb24f1efe00 R12: ffff9159a9a27738 [Sat Jan 26 01:48:31 2019][717787.225517] R13: ffff912d521cf548 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:31 2019][717787.225518] FS: 00007f277afbf700(0000) GS:ffff9139ff740000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:31 2019][717787.225519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:31 2019][717787.225520] CR2: 00007f2781327000 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:31 2019][717787.225521] Call Trace: [Sat Jan 26 01:48:31 2019][717787.225524] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:31 2019][717787.225526] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:31 2019][717787.225538] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:31 2019][717787.225551] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:31 2019][717787.225556] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Sat Jan 26 01:48:31 2019][717787.225558] [] ? crypto_create_tfm+0x32/0xd0 [Sat Jan 26 01:48:31 2019][717787.225566] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Sat Jan 26 01:48:31 2019][717787.225574] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:31 2019][717787.225581] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Sat Jan 26 01:48:31 2019][717787.225589] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Sat Jan 26 01:48:31 2019][717787.225600] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:31 2019][717787.225610] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Sat Jan 26 01:48:31 2019][717787.225620] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Sat Jan 26 01:48:31 2019][717787.225635] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Sat Jan 26 01:48:31 2019][717787.225680] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.225721] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.225724] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:31 2019][717787.225726] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:31 2019][717787.225757] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.225798] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.225838] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.225844] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:31 2019][717787.225882] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.225921] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.225923] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:31 2019][717787.225924] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:31 2019][717787.225961] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.225998] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.226000] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:31 2019][717787.226001] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:31 2019][717787.226003] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:31 2019][717787.226005] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:31 2019][717787.226005] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:31 2019][717787.294456] NMI watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [mdt_io01_004:6783] [Sat Jan 26 01:48:31 2019][717787.294457] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:31 2019][717787.294499] CPU: 29 PID: 6783 Comm: mdt_io01_004 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:31 2019][717787.294500] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:31 2019][717787.294501] task: ffff9159ed231040 ti: ffff9159efbd4000 task.ti: ffff9159efbd4000 [Sat Jan 26 01:48:31 2019][717787.294502] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x158/0x200 [Sat Jan 26 01:48:31 2019][717787.294506] RSP: 0018:ffff9159efbd7798 EFLAGS: 00000202 [Sat Jan 26 01:48:31 2019][717787.294507] RAX: 0000000000000101 RBX: ffff912d521d14c8 RCX: 0000000000e90000 [Sat Jan 26 01:48:31 2019][717787.294508] RDX: 0000000000d10101 RSI: 0000000000000101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:31 2019][717787.294508] RBP: ffff9159efbd7798 R08: ffff9139ff7db780 R09: 0000000000000000 [Sat Jan 26 01:48:31 2019][717787.294509] R10: ffff9139ff7df140 R11: ffffdbb27ea72200 R12: ffff9159efbd7738 [Sat Jan 26 01:48:31 2019][717787.294510] R13: ffff912d521d15b0 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:31 2019][717787.294511] FS: 00007fd22b997940(0000) GS:ffff9139ff7c0000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:31 2019][717787.294512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:31 2019][717787.294513] CR2: 00007f0efc872000 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:31 2019][717787.294514] Call Trace: [Sat Jan 26 01:48:31 2019][717787.294517] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:31 2019][717787.294519] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:31 2019][717787.294531] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:31 2019][717787.294543] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:31 2019][717787.294549] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Sat Jan 26 01:48:31 2019][717787.294554] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Sat Jan 26 01:48:31 2019][717787.294557] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:31 2019][717787.294564] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Sat Jan 26 01:48:31 2019][717787.294572] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:31 2019][717787.294580] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Sat Jan 26 01:48:31 2019][717787.294587] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Sat Jan 26 01:48:31 2019][717787.294598] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:31 2019][717787.294608] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Sat Jan 26 01:48:31 2019][717787.294618] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Sat Jan 26 01:48:31 2019][717787.294633] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Sat Jan 26 01:48:31 2019][717787.294676] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.294717] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.294719] [] ? load_balance+0x1be/0x9a0 [Sat Jan 26 01:48:31 2019][717787.294721] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:31 2019][717787.294723] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:31 2019][717787.294754] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.294795] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.294834] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.294841] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:31 2019][717787.294878] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.294915] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.294917] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:31 2019][717787.294918] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:31 2019][717787.294955] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.294992] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:31 2019][717787.294993] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:31 2019][717787.294995] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:31 2019][717787.294997] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:31 2019][717787.294999] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:31 2019][717787.294999] Code: 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 8b 17 <0f> b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84 00 00 [Sat Jan 26 01:48:31 2019][717787.391459] NMI watchdog: BUG: soft lockup - CPU#40 stuck for 22s! [mdt_io00_008:7094] [Sat Jan 26 01:48:31 2019][717787.391460] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:32 2019][717787.391532] CPU: 40 PID: 7094 Comm: mdt_io00_008 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:32 2019][717787.391533] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:32 2019][717787.391534] task: ffff9159f6ada080 ti: ffff9159ac4c0000 task.ti: ffff9159ac4c0000 [Sat Jan 26 01:48:32 2019][717787.391535] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Sat Jan 26 01:48:32 2019][717787.391543] RSP: 0018:ffff9159ac4c3798 EFLAGS: 00000246 [Sat Jan 26 01:48:32 2019][717787.391543] RAX: 0000000000000000 RBX: ffff911e3329cef8 RCX: 0000000001410000 [Sat Jan 26 01:48:32 2019][717787.391544] RDX: ffff9149ff6db780 RSI: 0000000000710101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:32 2019][717787.391545] RBP: ffff9159ac4c3798 R08: ffff9129ff09b780 R09: 0000000000000000 [Sat Jan 26 01:48:32 2019][717787.391545] R10: ffff9129ff09f140 R11: ffffdbb280aad800 R12: ffff9159ac4c3738 [Sat Jan 26 01:48:32 2019][717787.391546] R13: ffff911e3329cfe0 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:32 2019][717787.391547] FS: 00007fc05d72f740(0000) GS:ffff9129ff080000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:32 2019][717787.391548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:32 2019][717787.391549] CR2: 00007fc05d188d10 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:32 2019][717787.391550] Call Trace: [Sat Jan 26 01:48:32 2019][717787.391556] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:32 2019][717787.391560] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:32 2019][717787.391575] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:32 2019][717787.391585] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:32 2019][717787.391593] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Sat Jan 26 01:48:32 2019][717787.391598] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Sat Jan 26 01:48:32 2019][717787.391603] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:32 2019][717787.391610] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Sat Jan 26 01:48:32 2019][717787.391618] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:32 2019][717787.391624] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Sat Jan 26 01:48:32 2019][717787.391631] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Sat Jan 26 01:48:32 2019][717787.391642] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:32 2019][717787.391650] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Sat Jan 26 01:48:32 2019][717787.391659] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Sat Jan 26 01:48:32 2019][717787.391676] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Sat Jan 26 01:48:32 2019][717787.391736] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.391771] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.391774] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:32 2019][717787.391776] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:32 2019][717787.391802] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.391837] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.391871] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.391879] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:32 2019][717787.391911] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.391942] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.391946] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:32 2019][717787.391948] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:32 2019][717787.391979] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.392010] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.392014] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:32 2019][717787.392016] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:32 2019][717787.392019] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:32 2019][717787.392020] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:32 2019][717787.392020] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Sat Jan 26 01:48:32 2019][717787.417314] Lustre: MGS: Received new LWP connection from 10.8.6.20@o2ib6, removing former export from same NID [Sat Jan 26 01:48:32 2019][717787.426463] NMI watchdog: BUG: soft lockup - CPU#44 stuck for 22s! [mdt00_011:7500] [Sat Jan 26 01:48:32 2019][717787.426464] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:32 2019][717787.426538] CPU: 44 PID: 7500 Comm: mdt00_011 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:32 2019][717787.426539] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:32 2019][717787.426541] task: ffff9159ef230000 ti: ffff9159f5628000 task.ti: ffff9159f5628000 [Sat Jan 26 01:48:32 2019][717787.426542] RIP: 0010:[] [] ldiskfs_inode_touch_time_cmp+0xd/0x90 [ldiskfs] [Sat Jan 26 01:48:32 2019][717787.426560] RSP: 0018:ffff9159f562b310 EFLAGS: 00000286 [Sat Jan 26 01:48:32 2019][717787.426561] RAX: 8000040400080000 RBX: ffffffffade19f22 RCX: 000000012a45e81a [Sat Jan 26 01:48:32 2019][717787.426561] RDX: ffff91205631ad10 RSI: ffff91205ebfa8e8 RDI: 0000000000000000 [Sat Jan 26 01:48:32 2019][717787.426562] RBP: ffff9159f562b360 R08: ffff9159f562b490 R09: 00000000006f34e0 [Sat Jan 26 01:48:32 2019][717787.426563] R10: 000000006ff8a901 R11: ffffdbb27ebfe280 R12: 0000000000000000 [Sat Jan 26 01:48:32 2019][717787.426563] R13: ffff911b291c4c80 R14: ffff9129f6490400 R15: ffffdbb2bfc76040 [Sat Jan 26 01:48:32 2019][717787.426565] FS: 00007f6417cd6700(0000) GS:ffff9129ff0c0000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:32 2019][717787.426566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:32 2019][717787.426566] CR2: 00007f640cefb080 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:32 2019][717787.426568] Call Trace: [Sat Jan 26 01:48:32 2019][717787.426575] [] ? merge+0x62/0xc0 [Sat Jan 26 01:48:32 2019][717787.426587] [] ? ldiskfs_init_inode_table+0x410/0x410 [ldiskfs] [Sat Jan 26 01:48:32 2019][717787.426589] [] list_sort+0x9b/0x250 [Sat Jan 26 01:48:32 2019][717787.426601] [] __ldiskfs_es_shrink+0x1ce/0x2a0 [ldiskfs] [Sat Jan 26 01:48:32 2019][717787.426611] [] ldiskfs_es_shrink+0xb4/0x130 [ldiskfs] [Sat Jan 26 01:48:32 2019][717787.426616] [] shrink_slab+0x175/0x340 [Sat Jan 26 01:48:32 2019][717787.426618] [] ? zone_watermark_ok+0x1f/0x30 [Sat Jan 26 01:48:32 2019][717787.426622] [] ? compaction_suitable+0xa3/0xb0 [Sat Jan 26 01:48:32 2019][717787.426624] [] zone_reclaim+0x1d1/0x2f0 [Sat Jan 26 01:48:32 2019][717787.426626] [] get_page_from_freelist+0x87b/0xa70 [Sat Jan 26 01:48:32 2019][717787.426657] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Sat Jan 26 01:48:32 2019][717787.426667] [] ? qsd_op_end+0x7c/0x6f0 [lquota] [Sat Jan 26 01:48:32 2019][717787.426669] [] __alloc_pages_nodemask+0x176/0x420 [Sat Jan 26 01:48:32 2019][717787.426673] [] alloc_pages_current+0x98/0x110 [Sat Jan 26 01:48:32 2019][717787.426676] [] __get_free_pages+0xe/0x40 [Sat Jan 26 01:48:32 2019][717787.426679] [] kmalloc_order_trace+0x2e/0xa0 [Sat Jan 26 01:48:32 2019][717787.426681] [] __kmalloc+0x211/0x230 [Sat Jan 26 01:48:32 2019][717787.426740] [] null_alloc_rs+0x16d/0x340 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.426781] [] sptlrpc_svc_alloc_rs+0x66/0x350 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.426797] [] ? mdt_root_squash+0x21/0x430 [mdt] [Sat Jan 26 01:48:32 2019][717787.426835] [] lustre_pack_reply_v2+0x8a/0x280 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.426872] [] lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.426908] [] lustre_pack_reply+0x11/0x20 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.426947] [] req_capsule_server_pack+0x43/0xf0 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.426962] [] mdt_getxattr+0x71c/0x12e0 [mdt] [Sat Jan 26 01:48:32 2019][717787.426974] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Sat Jan 26 01:48:32 2019][717787.427011] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.427023] [] mdt_intent_getxattr+0xc5/0x270 [mdt] [Sat Jan 26 01:48:32 2019][717787.427034] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Sat Jan 26 01:48:32 2019][717787.427046] [] ? mdt_intent_getattr+0x480/0x480 [mdt] [Sat Jan 26 01:48:32 2019][717787.427076] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.427088] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Sat Jan 26 01:48:32 2019][717787.427096] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Sat Jan 26 01:48:32 2019][717787.427129] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.427166] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.427211] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.427253] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.427292] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.427298] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:32 2019][717787.427335] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.427371] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.427376] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:32 2019][717787.427379] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:32 2019][717787.427415] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.427452] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:32 2019][717787.427456] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:32 2019][717787.427459] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:32 2019][717787.427463] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:32 2019][717787.427465] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:32 2019][717787.427466] Code: ff 8d 4a 01 89 d0 f0 0f b1 0f 39 d0 0f 84 fb fd ff ff 89 c2 eb e2 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 86 e8 fc ff ff <48> 89 e5 48 c1 e8 2b a8 01 74 15 48 8b 8a e8 fc ff ff b8 01 00 [Sat Jan 26 01:48:32 2019] [Sat Jan 26 01:48:32 2019][717790.496921] grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:33 2019][717790.578088] CPU: 9 PID: 7223 Comm: mdt_io01_012 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:33 2019][717790.590507] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:33 2019][717790.598159] task: ffff91599d3c6180 ti: ffff9159f6fb4000 task.ti: ffff9159f6fb4000 [Sat Jan 26 01:48:33 2019][717790.605726] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:33 2019][717790.615745] RSP: 0018:ffff9159f6fb7798 EFLAGS: 00000246 [Sat Jan 26 01:48:33 2019][717790.621144] RAX: 0000000000000000 RBX: ffff911e3329cad0 RCX: 0000000000490000 [Sat Jan 26 01:48:33 2019][717790.628362] RDX: ffff9139ff7db780 RSI: 0000000000e90101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:33 2019][717790.635583] RBP: ffff9159f6fb7798 R08: ffff9139ff69b780 R09: 0000000000000000 [Sat Jan 26 01:48:33 2019][717790.642803] R10: ffff9139ff69f140 R11: ffffdbb251ebd400 R12: ffff9159f6fb7738 [Sat Jan 26 01:48:33 2019][717790.650022] R13: ffff911e3329cbb8 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:33 2019][717790.657241] FS: 00007f0185f91700(0000) GS:ffff9139ff680000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:33 2019][717790.665413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:33 2019][717790.671247] CR2: 00007f63fb343000 CR3: 00000020303fc000 CR4: 00000000003407e0 [Sat Jan 26 01:48:33 2019][717790.678467] Call Trace: [Sat Jan 26 01:48:33 2019][717790.681009] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:33 2019][717790.687361] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:33 2019][717790.692942] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:33 2019][717790.699738] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:33 2019][717790.707046] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Sat Jan 26 01:48:33 2019][717790.713833] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Sat Jan 26 01:48:33 2019][717790.720796] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:33 2019][717790.727242] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Sat Jan 26 01:48:33 2019][717790.733691] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:33 2019][717790.740572] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Sat Jan 26 01:48:33 2019][717790.747019] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Sat Jan 26 01:48:33 2019][717790.754770] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:33 2019][717790.762866] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Sat Jan 26 01:48:33 2019][717790.770006] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Sat Jan 26 01:48:33 2019][717790.777672] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Sat Jan 26 01:48:33 2019][717790.784236] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Sat Jan 26 01:48:33 2019][717790.790545] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Sat Jan 26 01:48:33 2019][717790.797195] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Sat Jan 26 01:48:33 2019][717790.804155] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:33 2019][717790.809813] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:33 2019][717790.816282] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Sat Jan 26 01:48:33 2019][717790.823624] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:33 2019][717790.830616] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:33 2019][717790.838273] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:33 2019][717790.845445] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:33 2019][717790.853219] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:33 2019][717790.860090] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:33 2019][717790.866443] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:33 2019][717790.872397] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:33 2019][717790.878786] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:33 2019][717790.886264] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:33 2019][717790.891230] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:33 2019][717790.897409] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:33 2019][717790.903937] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:33 2019][717790.910115] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:33 2019][717790.973562] Lustre: 5943:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1548496146/real 0] req@ffff91594ee4ce00 x1623664975830112/t0(0) o13->fir-OST0021-osc-MDT0000@10.0.10.106@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1548496153 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Sat Jan 26 01:48:33 2019][717791.000896] Lustre: 5943:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [Sat Jan 26 01:48:33 2019][717791.010636] Lustre: fir-OST0021-osc-MDT0000: Connection to fir-OST0021 (at 10.0.10.106@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Sat Jan 26 01:48:33 2019][717791.026785] Lustre: Skipped 7 previous similar messages [Sat Jan 26 01:48:33 2019][717791.047551] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [mdt_io00_001:6210] [Sat Jan 26 01:48:33 2019][717791.055467] Modules linked in:[717791.055549] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [mdt_io01_013:8951] [Sat Jan 26 01:48:33 2019][717791.055550] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:33 2019][717791.055598] CPU: 1 PID: 8951 Comm: mdt_io01_013 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:33 2019][717791.055598] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:33 2019][717791.055600] task: ffff91599d3d30c0 ti: ffff9159b1174000 task.ti: ffff9159b1174000 [Sat Jan 26 01:48:33 2019][717791.055601] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:33 2019][717791.055605] RSP: 0018:ffff9159b1177808 EFLAGS: 00000246 [Sat Jan 26 01:48:33 2019][717791.055606] RAX: 0000000000000000 RBX: ffff912d55335320 RCX: 0000000000090000 [Sat Jan 26 01:48:33 2019][717791.055606] RDX: ffff9139ff65b780 RSI: 0000000000290101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:33 2019][717791.055607] RBP: ffff9159b1177808 R08: ffff9139ff61b780 R09: 0000000000000000 [Sat Jan 26 01:48:33 2019][717791.055608] R10: ffff9139ff61f140 R11: ffffdbb251edf600 R12: ffff9159b11777a8 [Sat Jan 26 01:48:33 2019][717791.055608] R13: ffff912d55335408 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:33 2019][717791.055610] FS: 00007f63fb2d1700(0000) GS:ffff9139ff600000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:33 2019][717791.055611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:33 2019][717791.055612] CR2: 00007f63fb343000 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:33 2019][717791.055612] Call Trace: [Sat Jan 26 01:48:33 2019][717791.055615] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:33 2019][717791.055617] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:33 2019][717791.055630] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:33 2019][717791.055642] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:33 2019][717791.055646] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:33 2019][717791.055652] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:33 2019][717791.055661] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:33 2019][717791.055662] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:33 2019][717791.055673] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:33 2019][717791.055684] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:33 2019][717791.055699] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:33 2019][717791.055742] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:33 2019][717791.055780] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Sat Jan 26 01:48:33 2019][717791.055816] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Sat Jan 26 01:48:33 2019][717791.055818] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:33 2019][717791.055820] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:33 2019][717791.055822] [] ? __enqueue_entity+0x78/0x80 [Sat Jan 26 01:48:33 2019][717791.055863] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:33 2019][717791.055904] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:33 2019][717791.055943] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:33 2019][717791.055950] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:33 2019][717791.055987] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:33 2019][717791.056024] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:33 2019][717791.056026] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:33 2019][717791.056028] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:33 2019][717791.056064] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:33 2019][717791.056101] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:33 2019][717791.056102] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:33 2019][717791.056104] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:33 2019][717791.056106] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:33 2019][717791.056107] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:33 2019][717791.056108] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:33 2019][717791.063550] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [mdt_io02_003:6779] [Sat Jan 26 01:48:33 2019][717791.063551] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:34 2019][717791.063618] CPU: 2 PID: 6779 Comm: mdt_io02_003 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:34 2019][717791.063619] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:34 2019][717791.063620] task: ffff9159f774c100 ti: ffff9159ee690000 task.ti: ffff9159ee690000 [Sat Jan 26 01:48:34 2019][717791.063621] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Sat Jan 26 01:48:34 2019][717791.063629] RSP: 0018:ffff9159ee693808 EFLAGS: 00000246 [Sat Jan 26 01:48:34 2019][717791.063630] RAX: 0000000000000000 RBX: ffff912d553063c0 RCX: 0000000000110000 [Sat Jan 26 01:48:34 2019][717791.063631] RDX: ffff9139ff61b780 RSI: 0000000000090101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:34 2019][717791.063632] RBP: ffff9159ee693808 R08: ffff9149ff61b780 R09: 0000000000000000 [Sat Jan 26 01:48:34 2019][717791.063632] R10: ffff9149ff61f140 R11: ffffdbb2aebc2c00 R12: ffff9159ee6937a8 [Sat Jan 26 01:48:34 2019][717791.063633] R13: ffff912d553064a8 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:34 2019][717791.063635] FS: 00007f2781319740(0000) GS:ffff9149ff600000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:34 2019][717791.063635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:34 2019][717791.063636] CR2: 000055fdd4177300 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:34 2019][717791.063637] Call Trace: [Sat Jan 26 01:48:34 2019][717791.063643] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:34 2019][717791.063646] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:34 2019][717791.063662] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:34 2019][717791.063674] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:34 2019][717791.063678] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:34 2019][717791.063687] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:34 2019][717791.063696] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:34 2019][717791.063697] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:34 2019][717791.063711] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:34 2019][717791.063722] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:34 2019][717791.063744] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:34 2019][717791.063808] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.063850] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.063892] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.063932] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.063941] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:34 2019][717791.063979] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.064016] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.064022] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:34 2019][717791.064025] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:34 2019][717791.064063] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.064100] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.064104] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:34 2019][717791.064106] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:34 2019][717791.064109] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:34 2019][717791.064110] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:34 2019][717791.064111] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Sat Jan 26 01:48:34 2019][717791.081552] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [mdt_io00_018:8993] [Sat Jan 26 01:48:34 2019][717791.081553] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:34 2019][717791.081623] CPU: 4 PID: 8993 Comm: mdt_io00_018 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:34 2019][717791.081624] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:34 2019][717791.081625] task: ffff9159e7e2e180 ti: ffff91599ee50000 task.ti: ffff91599ee50000 [Sat Jan 26 01:48:34 2019][717791.081626] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:34 2019][717791.081633] RSP: 0018:ffff91599ee53808 EFLAGS: 00000246 [Sat Jan 26 01:48:34 2019][717791.081634] RAX: 0000000000000000 RBX: ffff914338cc5b70 RCX: 0000000000210000 [Sat Jan 26 01:48:34 2019][717791.081635] RDX: ffff9139ff89b780 RSI: 0000000001490101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:34 2019][717791.081636] RBP: ffff91599ee53808 R08: ffff9129fee5b780 R09: 0000000000000000 [Sat Jan 26 01:48:34 2019][717791.081636] R10: ffff9129fee5f140 R11: ffffdbb2153b1600 R12: ffff91599ee537a8 [Sat Jan 26 01:48:34 2019][717791.081637] R13: ffff914338cc5c58 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:34 2019][717791.081638] FS: 00007f6417cd6700(0000) GS:ffff9129fee40000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:34 2019][717791.081639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:34 2019][717791.081640] CR2: 00007ffc4a477cf8 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:34 2019][717791.081641] Call Trace: [Sat Jan 26 01:48:34 2019][717791.081646] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:34 2019][717791.081650] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:34 2019][717791.081665] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:34 2019][717791.081677] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:34 2019][717791.081682] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:34 2019][717791.081691] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:34 2019][717791.081699] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:34 2019][717791.081701] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:34 2019][717791.081713] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:34 2019][717791.081724] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:34 2019][717791.081742] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:34 2019][717791.081801] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.081839] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.081876] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.081882] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:34 2019][717791.081885] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:34 2019][717791.081886] [] ? __enqueue_entity+0x78/0x80 [Sat Jan 26 01:48:34 2019][717791.081928] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.081971] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.082010] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.082019] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:34 2019][717791.082057] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.082094] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.082098] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:34 2019][717791.082100] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:34 2019][717791.082137] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.082174] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:34 2019][717791.082178] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:34 2019][717791.082180] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:34 2019][717791.082182] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:34 2019][717791.082184] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:34 2019][717791.082184] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:34 2019][717791.090550] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [mdt_io01_019:8957] [Sat Jan 26 01:48:34 2019][717791.090550] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:34 2019][717791.090597] CPU: 5 PID: 8957 Comm: mdt_io01_019 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:34 2019][717791.090597] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:34 2019][717791.090598] task: ffff9159efebe180 ti: ffff91599f648000 task.ti: ffff91599f648000 [Sat Jan 26 01:48:34 2019][717791.090599] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Sat Jan 26 01:48:34 2019][717791.090602] RSP: 0018:ffff91599f64b798 EFLAGS: 00000246 [Sat Jan 26 01:48:34 2019][717791.090603] RAX: 0000000000000000 RBX: ffff911e383790a0 RCX: 0000000000290000 [Sat Jan 26 01:48:34 2019][717791.090604] RDX: ffff9129fefdb780 RSI: 0000000000e10101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:34 2019][717791.090604] RBP: ffff91599f64b798 R08: ffff9139ff65b780 R09: 0000000000000000 [Sat Jan 26 01:48:34 2019][717791.090605] R10: ffff9139ff65f140 R11: ffffdbb280ed8600 R12: ffff91599f64b738 [Sat Jan 26 01:48:35 2019][717791.090606] R13: ffff911e38379188 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:35 2019][717791.090607] FS: 00007f277afbf700(0000) GS:ffff9139ff640000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:35 2019][717791.090608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:35 2019][717791.090609] CR2: 00007f2781327000 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:35 2019][717791.090609] Call Trace: [Sat Jan 26 01:48:35 2019][717791.090611] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:35 2019][717791.090613] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:35 2019][717791.090625] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:35 2019][717791.090636] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:35 2019][717791.090642] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Sat Jan 26 01:48:35 2019][717791.090647] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Sat Jan 26 01:48:35 2019][717791.090649] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:35 2019][717791.090657] [] ? qsd_op_begin+0xb1/0x4b0 [lquota] [Sat Jan 26 01:48:35 2019][717791.090665] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:35 2019][717791.090673] [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] [Sat Jan 26 01:48:35 2019][717791.090683] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:35 2019][717791.090694] [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] [Sat Jan 26 01:48:35 2019][717791.090704] [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Sat Jan 26 01:48:35 2019][717791.090719] [] mdt_obd_commitrw+0x953/0x1150 [mdt] [Sat Jan 26 01:48:35 2019][717791.090763] [] obd_commitrw+0x9c/0x370 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.090805] [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.090807] [] ? load_balance+0x178/0x9a0 [Sat Jan 26 01:48:35 2019][717791.090839] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.090880] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.090923] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.090930] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:35 2019][717791.090970] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.091007] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.091009] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:35 2019][717791.091010] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:35 2019][717791.091050] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.091089] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.091090] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:35 2019][717791.091092] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:35 2019][717791.091094] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:35 2019][717791.091095] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:35 2019][717791.091096] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Sat Jan 26 01:48:35 2019][717791.131551] NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [mdt_io02_007:7098] [Sat Jan 26 01:48:35 2019][717791.131551] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:35 2019][717791.131599] CPU: 10 PID: 7098 Comm: mdt_io02_007 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:35 2019][717791.131600] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:35 2019][717791.131601] task: ffff9159f6ade180 ti: ffff9159f0ab0000 task.ti: ffff9159f0ab0000 [Sat Jan 26 01:48:35 2019][717791.131602] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Sat Jan 26 01:48:35 2019][717791.131606] RSP: 0018:ffff9159f0ab3808 EFLAGS: 00000246 [Sat Jan 26 01:48:35 2019][717791.131607] RAX: 0000000000000000 RBX: ffff912d5536adb8 RCX: 0000000000510000 [Sat Jan 26 01:48:35 2019][717791.131607] RDX: ffff9129fef5b780 RSI: 0000000000a10101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:35 2019][717791.131608] RBP: ffff9159f0ab3808 R08: ffff9149ff69b780 R09: 0000000000000000 [Sat Jan 26 01:48:35 2019][717791.131609] R10: ffff9149ff69f140 R11: ffffdbb2fe3acc00 R12: ffff9159f0ab37a8 [Sat Jan 26 01:48:35 2019][717791.131609] R13: ffff912d5536aea0 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:35 2019][717791.131611] FS: 00007f63fb2d1700(0000) GS:ffff9149ff680000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:35 2019][717791.131612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:35 2019][717791.131613] CR2: 00007fcce4727d38 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:35 2019][717791.131613] Call Trace: [Sat Jan 26 01:48:35 2019][717791.131616] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:35 2019][717791.131618] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:35 2019][717791.131631] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:35 2019][717791.131643] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:35 2019][717791.131645] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:35 2019][717791.131651] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:35 2019][717791.131659] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:35 2019][717791.131661] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:35 2019][717791.131671] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:35 2019][717791.131682] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:35 2019][717791.131697] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:35 2019][717791.131740] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.131744] [] ? __enqueue_entity+0x78/0x80 [Sat Jan 26 01:48:35 2019][717791.131786] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.131829] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.131868] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.131875] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:35 2019][717791.131913] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.131950] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.131952] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:35 2019][717791.131953] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:35 2019][717791.131990] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.132027] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.132029] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:35 2019][717791.132030] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:35 2019][717791.132032] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:35 2019][717791.132034] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:35 2019][717791.132034] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Sat Jan 26 01:48:35 2019][717791.148553] NMI watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [mdt_io00_007:6909] [Sat Jan 26 01:48:35 2019][717791.148554] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:35 2019][717791.148620] CPU: 12 PID: 6909 Comm: mdt_io00_007 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:35 2019][717791.148620] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:35 2019][717791.148621] task: ffff9159fc2cb0c0 ti: ffff9159faaf8000 task.ti: ffff9159faaf8000 [Sat Jan 26 01:48:35 2019][717791.148623] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:35 2019][717791.148627] RSP: 0018:ffff9159faafb808 EFLAGS: 00000246 [Sat Jan 26 01:48:35 2019][717791.148628] RAX: 0000000000000000 RBX: ffff911e383798f0 RCX: 0000000000610000 [Sat Jan 26 01:48:35 2019][717791.148629] RDX: ffff9149ff8db780 RSI: 0000000001710101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:35 2019][717791.148630] RBP: ffff9159faafb808 R08: ffff9129feedb780 R09: 0000000000000000 [Sat Jan 26 01:48:35 2019][717791.148631] R10: ffff9129feedf140 R11: ffffdbb212834400 R12: ffff9159faafb7a8 [Sat Jan 26 01:48:35 2019][717791.148631] R13: ffff911e383799d8 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:35 2019][717791.148633] FS: 00007f6417cd6700(0000) GS:ffff9129feec0000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:35 2019][717791.148634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:35 2019][717791.148634] CR2: 00007fe2d3446000 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:35 2019][717791.148635] Call Trace: [Sat Jan 26 01:48:35 2019][717791.148639] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:35 2019][717791.148642] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:35 2019][717791.148657] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:35 2019][717791.148669] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:35 2019][717791.148672] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:35 2019][717791.148680] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:35 2019][717791.148688] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:35 2019][717791.148691] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:35 2019][717791.148703] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:35 2019][717791.148713] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:35 2019][717791.148730] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:35 2019][717791.148777] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:35 2019][717791.148816] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.148819] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:36 2019][717791.148821] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:36 2019][717791.148823] [] ? __enqueue_entity+0x78/0x80 [Sat Jan 26 01:48:36 2019][717791.148864] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.148906] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.148945] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.148953] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:36 2019][717791.148990] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.149028] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.149030] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:36 2019][717791.149032] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:36 2019][717791.149069] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.149105] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.149108] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:36 2019][717791.149109] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:36 2019][717791.149111] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:36 2019][717791.149113] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:36 2019][717791.149113] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:36 2019][717791.217555] NMI watchdog: BUG: soft lockup - CPU#20 stuck for 23s! [mdt_io00_023:10518] [Sat Jan 26 01:48:36 2019][717791.217555] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:36 2019][717791.217609] CPU: 20 PID: 10518 Comm: mdt_io00_023 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:36 2019][717791.217609] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:36 2019][717791.217610] task: ffff9149ca784100 ti: ffff915984200000 task.ti: ffff915984200000 [Sat Jan 26 01:48:36 2019][717791.217611] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:36 2019][717791.217615] RSP: 0018:ffff915984203808 EFLAGS: 00000246 [Sat Jan 26 01:48:36 2019][717791.217616] RAX: 0000000000000000 RBX: ffff911e383bbe58 RCX: 0000000000a10000 [Sat Jan 26 01:48:36 2019][717791.217617] RDX: ffff9149ff89b780 RSI: 0000000001510101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:36 2019][717791.217617] RBP: ffff915984203808 R08: ffff9129fef5b780 R09: 0000000000000000 [Sat Jan 26 01:48:36 2019][717791.217618] R10: ffff9129fef5f140 R11: ffffdbb205a4ee00 R12: ffff9159842037a8 [Sat Jan 26 01:48:36 2019][717791.217618] R13: ffff911e383bbf40 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:36 2019][717791.217620] FS: 00007f6417cd6700(0000) GS:ffff9129fef40000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:36 2019][717791.217620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:36 2019][717791.217621] CR2: 00007fbdf1572080 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:36 2019][717791.217622] Call Trace: [Sat Jan 26 01:48:36 2019][717791.217624] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:36 2019][717791.217627] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:36 2019][717791.217640] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:36 2019][717791.217650] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:36 2019][717791.217653] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:36 2019][717791.217660] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:36 2019][717791.217668] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:36 2019][717791.217670] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:36 2019][717791.217681] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:36 2019][717791.217692] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:36 2019][717791.217708] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:36 2019][717791.217751] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.217788] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.217823] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.217826] [] ? __mutex_lock_slowpath+0x1b8/0x1d0 [Sat Jan 26 01:48:36 2019][717791.217829] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:36 2019][717791.217868] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.217908] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.217946] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.217953] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:36 2019][717791.217989] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.218024] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.218026] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:36 2019][717791.218028] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:36 2019][717791.218064] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.218099] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.218101] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:36 2019][717791.218102] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:36 2019][717791.218104] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:36 2019][717791.218106] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:36 2019][717791.218106] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:36 2019][717791.259555] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! [mdt_io01_010:7095] [Sat Jan 26 01:48:36 2019][717791.259555] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:36 2019][717791.259598] CPU: 25 PID: 7095 Comm: mdt_io01_010 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:36 2019][717791.259599] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:36 2019][717791.259600] task: ffff9159f6adb0c0 ti: ffff9159f0a64000 task.ti: ffff9159f0a64000 [Sat Jan 26 01:48:36 2019][717791.259600] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:36 2019][717791.259603] RSP: 0018:ffff9159f0a67808 EFLAGS: 00000246 [Sat Jan 26 01:48:36 2019][717791.259604] RAX: 0000000000000000 RBX: ffff912d55310000 RCX: 0000000000c90000 [Sat Jan 26 01:48:36 2019][717791.259605] RDX: ffff9139ff85b780 RSI: 0000000001290101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:36 2019][717791.259605] RBP: ffff9159f0a67808 R08: ffff9139ff79b780 R09: 0000000000000000 [Sat Jan 26 01:48:36 2019][717791.259606] R10: ffff9139ff79f140 R11: ffffdbb2bd173200 R12: ffff9159f0a677a8 [Sat Jan 26 01:48:36 2019][717791.259606] R13: ffff912d553100e8 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:36 2019][717791.259608] FS: 00007f63fb2d1700(0000) GS:ffff9139ff780000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:36 2019][717791.259609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:36 2019][717791.259609] CR2: 00007f63fb343000 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:36 2019][717791.259610] Call Trace: [Sat Jan 26 01:48:36 2019][717791.259612] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:36 2019][717791.259614] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:36 2019][717791.259624] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:36 2019][717791.259635] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:36 2019][717791.259637] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:36 2019][717791.259643] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:36 2019][717791.259650] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:36 2019][717791.259652] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:36 2019][717791.259662] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:36 2019][717791.259672] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:36 2019][717791.259687] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:36 2019][717791.259728] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.259733] [] ? __slab_free+0x81/0x2f0 [Sat Jan 26 01:48:36 2019][717791.259768] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.259770] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:36 2019][717791.259772] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:36 2019][717791.259812] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.259851] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.259889] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.259897] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:36 2019][717791.259933] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.259969] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.259970] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:36 2019][717791.259972] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:36 2019][717791.260008] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.260043] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:36 2019][717791.260044] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:36 2019][717791.260046] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:36 2019][717791.260048] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:36 2019][717791.260049] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:36 2019][717791.260050] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:36 2019][717791.286556] NMI watchdog: BUG: soft lockup - CPU#28 stuck for 23s! [mdt_io00_019:9002] [Sat Jan 26 01:48:37 2019][717791.286557] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:37 2019][717791.286618] CPU: 28 PID: 9002 Comm: mdt_io00_019 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:37 2019][717791.286619] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:37 2019][717791.286620] task: ffff9159a7661040 ti: ffff9159f1714000 task.ti: ffff9159f1714000 [Sat Jan 26 01:48:37 2019][717791.286621] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:37 2019][717791.286624] RSP: 0018:ffff9159f1717808 EFLAGS: 00000246 [Sat Jan 26 01:48:37 2019][717791.286625] RAX: 0000000000000000 RBX: ffff912d55310428 RCX: 0000000000e10000 [Sat Jan 26 01:48:37 2019][717791.286626] RDX: ffff9139ff79b780 RSI: 0000000000c90101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:37 2019][717791.286627] RBP: ffff9159f1717808 R08: ffff9129fefdb780 R09: 0000000000000000 [Sat Jan 26 01:48:37 2019][717791.286627] R10: ffff9129fefdf140 R11: ffffdbb23fdbde00 R12: ffff9159f17177a8 [Sat Jan 26 01:48:37 2019][717791.286628] R13: ffff912d55310510 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:37 2019][717791.286629] FS: 00007f64178d5700(0000) GS:ffff9129fefc0000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:37 2019][717791.286630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:37 2019][717791.286631] CR2: 00007f641c871000 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:37 2019][717791.286631] Call Trace: [Sat Jan 26 01:48:37 2019][717791.286634] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:37 2019][717791.286637] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:37 2019][717791.286649] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:37 2019][717791.286661] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:37 2019][717791.286664] [] ? ___slab_alloc+0x209/0x4f0 [Sat Jan 26 01:48:37 2019][717791.286672] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:37 2019][717791.286680] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:37 2019][717791.286682] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:37 2019][717791.286693] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:37 2019][717791.286703] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:37 2019][717791.286719] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:37 2019][717791.286762] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.286767] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:37 2019][717791.286769] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:37 2019][717791.286771] [] ? __enqueue_entity+0x78/0x80 [Sat Jan 26 01:48:37 2019][717791.286811] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.286851] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.286889] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.286896] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:37 2019][717791.286932] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.286968] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.286970] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:37 2019][717791.286972] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:37 2019][717791.287008] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.287043] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.287045] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:37 2019][717791.287047] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:37 2019][717791.287048] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:37 2019][717791.287050] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:37 2019][717791.287051] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:37 2019][717791.329556] NMI watchdog: BUG: soft lockup - CPU#33 stuck for 23s! [mdt_io01_030:11093] [Sat Jan 26 01:48:37 2019][717791.329557] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:37 2019][717791.329603] CPU: 33 PID: 11093 Comm: mdt_io01_030 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:37 2019][717791.329603] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:37 2019][717791.329604] task: ffff9129ee3ce180 ti: ffff9159792e4000 task.ti: ffff9159792e4000 [Sat Jan 26 01:48:37 2019][717791.329605] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x128/0x200 [Sat Jan 26 01:48:37 2019][717791.329608] RSP: 0018:ffff9159792e7808 EFLAGS: 00000246 [Sat Jan 26 01:48:37 2019][717791.329609] RAX: 0000000000000000 RBX: ffff911e383794c8 RCX: 0000000001090000 [Sat Jan 26 01:48:37 2019][717791.329609] RDX: ffff9139ff75b780 RSI: 0000000000a90101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:37 2019][717791.329610] RBP: ffff9159792e7808 R08: ffff9139ff81b780 R09: 0000000000000000 [Sat Jan 26 01:48:37 2019][717791.329611] R10: ffff9139ff81f140 R11: ffffdbb300dac000 R12: ffff9159792e77a8 [Sat Jan 26 01:48:37 2019][717791.329611] R13: ffff911e383795b0 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:37 2019][717791.329612] FS: 00007f0185f70700(0000) GS:ffff9139ff800000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:37 2019][717791.329613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:37 2019][717791.329614] CR2: 0000000002109108 CR3: 00000020303fc000 CR4: 00000000003407e0 [Sat Jan 26 01:48:37 2019][717791.329615] Call Trace: [Sat Jan 26 01:48:37 2019][717791.329617] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:37 2019][717791.329618] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:37 2019][717791.329629] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:37 2019][717791.329640] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:37 2019][717791.329642] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:37 2019][717791.329647] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:37 2019][717791.329655] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:37 2019][717791.329656] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:37 2019][717791.329667] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:37 2019][717791.329677] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:37 2019][717791.329691] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:37 2019][717791.329732] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.329734] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:37 2019][717791.329735] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:37 2019][717791.329775] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.329815] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.329853] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.329859] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:37 2019][717791.329896] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.329931] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.329933] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:37 2019][717791.329935] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:37 2019][717791.329970] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.330006] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:37 2019][717791.330007] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:37 2019][717791.330009] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:37 2019][717791.330011] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:37 2019][717791.330012] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:37 2019][717791.330013] Code: 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 85 c0 <74> f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 [Sat Jan 26 01:48:37 2019][717791.338555] NMI watchdog: BUG: soft lockup - CPU#34 stuck for 23s! [mdt_io02_009:7101] [Sat Jan 26 01:48:37 2019][717791.338556] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:37 2019][717791.338600] CPU: 34 PID: 7101 Comm: mdt_io02_009 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:37 2019][717791.338601] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:37 2019][717791.338602] task: ffff9159f7c02080 ti: ffff9159a5fb4000 task.ti: ffff9159a5fb4000 [Sat Jan 26 01:48:37 2019][717791.338602] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:37 2019][717791.338605] RSP: 0018:ffff9159a5fb7808 EFLAGS: 00000246 [Sat Jan 26 01:48:38 2019][717791.338606] RAX: 0000000000000000 RBX: ffff912d553390a0 RCX: 0000000001110000 [Sat Jan 26 01:48:38 2019][717791.338607] RDX: ffff9129ff05b780 RSI: 0000000001210101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:38 2019][717791.338607] RBP: ffff9159a5fb7808 R08: ffff9149ff81b780 R09: 0000000000000000 [Sat Jan 26 01:48:38 2019][717791.338608] R10: ffff9149ff81f140 R11: ffffdbb2a7f8d000 R12: ffff9159a5fb77a8 [Sat Jan 26 01:48:38 2019][717791.338609] R13: ffff912d55339188 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:38 2019][717791.338610] FS: 00007fc2d94a5740(0000) GS:ffff9149ff800000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:38 2019][717791.338611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:38 2019][717791.338611] CR2: 00007fc2d8ec42e0 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:38 2019][717791.338612] Call Trace: [Sat Jan 26 01:48:38 2019][717791.338614] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:38 2019][717791.338616] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:38 2019][717791.338627] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:38 2019][717791.338637] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:38 2019][717791.338639] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:38 2019][717791.338645] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:38 2019][717791.338653] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:38 2019][717791.338654] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:38 2019][717791.338665] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:38 2019][717791.338675] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:38 2019][717791.338690] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:38 2019][717791.338730] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.338732] [] ? __enqueue_entity+0x78/0x80 [Sat Jan 26 01:48:38 2019][717791.338772] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.338811] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.338849] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.338856] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:38 2019][717791.338892] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.338928] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.338930] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:38 2019][717791.338931] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:38 2019][717791.338967] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.339002] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.339004] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:38 2019][717791.339005] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:38 2019][717791.339007] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:38 2019][717791.339008] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:38 2019][717791.339009] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:38 2019][717791.356557] NMI watchdog: BUG: soft lockup - CPU#36 stuck for 23s! [mdt_io00_000:6209] [Sat Jan 26 01:48:38 2019][717791.356557] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:38 2019][717791.356617] CPU: 36 PID: 6209 Comm: mdt_io00_000 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:38 2019][717791.356618] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:38 2019][717791.356618] task: ffff9149f865d140 ti: ffff91599f71c000 task.ti: ffff91599f71c000 [Sat Jan 26 01:48:38 2019][717791.356620] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Sat Jan 26 01:48:38 2019][717791.356623] RSP: 0018:ffff91599f71f808 EFLAGS: 00000246 [Sat Jan 26 01:48:38 2019][717791.356624] RAX: 0000000000000000 RBX: ffff914338cdcef8 RCX: 0000000001210000 [Sat Jan 26 01:48:38 2019][717791.356625] RDX: ffff9129fee1b780 RSI: 0000000000010101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:38 2019][717791.356625] RBP: ffff91599f71f808 R08: ffff9129ff05b780 R09: 0000000000000000 [Sat Jan 26 01:48:38 2019][717791.356626] R10: ffff9129ff05f140 R11: ffffdbb23d346e00 R12: ffff91599f71f7a8 [Sat Jan 26 01:48:38 2019][717791.356627] R13: ffff914338cdcfe0 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:38 2019][717791.356628] FS: 00007fe2d3435880(0000) GS:ffff9129ff040000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:38 2019][717791.356629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:38 2019][717791.356630] CR2: 0000000000449710 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:38 2019][717791.356630] Call Trace: [Sat Jan 26 01:48:38 2019][717791.356633] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:38 2019][717791.356636] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:38 2019][717791.356649] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:38 2019][717791.356660] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:38 2019][717791.356663] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:38 2019][717791.356670] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:38 2019][717791.356678] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:38 2019][717791.356679] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:38 2019][717791.356690] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:38 2019][717791.356700] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:38 2019][717791.356716] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:38 2019][717791.356761] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.356797] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.356833] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.356836] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:38 2019][717791.356839] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:38 2019][717791.356840] [] ? __enqueue_entity+0x78/0x80 [Sat Jan 26 01:48:38 2019][717791.356880] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.356920] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.356958] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.356966] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:38 2019][717791.357002] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.357038] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.357039] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:38 2019][717791.357042] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:38 2019][717791.357078] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.357114] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:38 2019][717791.357115] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:38 2019][717791.357117] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:38 2019][717791.357118] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:38 2019][717791.357120] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:38 2019][717791.357120] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Sat Jan 26 01:48:38 2019][717791.364556] NMI watchdog: BUG: soft lockup - CPU#37 stuck for 23s! [mdt_io01_027:9838] [Sat Jan 26 01:48:38 2019][717791.364557] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:38 2019][717791.364601] CPU: 37 PID: 9838 Comm: mdt_io01_027 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:38 2019][717791.364601] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:38 2019][717791.364602] task: ffff9149d62d5140 ti: ffff915991af4000 task.ti: ffff915991af4000 [Sat Jan 26 01:48:38 2019][717791.364603] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:38 2019][717791.364606] RSP: 0018:ffff915991af7808 EFLAGS: 00000246 [Sat Jan 26 01:48:38 2019][717791.364607] RAX: 0000000000000000 RBX: ffff912d552f5b70 RCX: 0000000001290000 [Sat Jan 26 01:48:38 2019][717791.364607] RDX: ffff9139ff8db780 RSI: 0000000001690101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:38 2019][717791.364608] RBP: ffff915991af7808 R08: ffff9139ff85b780 R09: 0000000000000000 [Sat Jan 26 01:48:38 2019][717791.364609] R10: ffff9139ff85f140 R11: ffffdbb24f335400 R12: ffff915991af77a8 [Sat Jan 26 01:48:38 2019][717791.364609] R13: ffff912d552f5c58 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:38 2019][717791.364611] FS: 00007fccefbfb840(0000) GS:ffff9139ff840000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:38 2019][717791.364611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:38 2019][717791.364612] CR2: 00007fcce673b3c0 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:38 2019][717791.364613] Call Trace: [Sat Jan 26 01:48:38 2019][717791.364615] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:38 2019][717791.364617] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:38 2019][717791.364628] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:38 2019][717791.364639] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:38 2019][717791.364643] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:38 2019][717791.364649] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:38 2019][717791.364657] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:38 2019][717791.364659] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:38 2019][717791.364669] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:38 2019][717791.364680] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:39 2019][717791.364695] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:39 2019][717791.364738] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.364776] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.364814] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.364815] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:39 2019][717791.364817] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:39 2019][717791.364859] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.364901] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.364940] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.364947] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:39 2019][717791.364985] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.365022] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.365023] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:39 2019][717791.365025] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:39 2019][717791.365062] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.365098] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.365100] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:39 2019][717791.365101] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:39 2019][717791.365103] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:39 2019][717791.365105] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:39 2019][717791.365105] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:39 2019][717791.399560] NMI watchdog: BUG: soft lockup - CPU#41 stuck for 23s! [mdt_io01_021:9001] [Sat Jan 26 01:48:39 2019][717791.399560] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:39 2019][717791.399604] CPU: 41 PID: 9001 Comm: mdt_io01_021 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:39 2019][717791.399605] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:39 2019][717791.399606] task: ffff9159a7660000 ti: ffff9159ed73c000 task.ti: ffff9159ed73c000 [Sat Jan 26 01:48:39 2019][717791.399607] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:39 2019][717791.399609] RSP: 0018:ffff9159ed73f808 EFLAGS: 00000246 [Sat Jan 26 01:48:39 2019][717791.399610] RAX: 0000000000000000 RBX: ffff912d55335b70 RCX: 0000000001490000 [Sat Jan 26 01:48:39 2019][717791.399611] RDX: ffff9149ff61b780 RSI: 0000000000110101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:39 2019][717791.399612] RBP: ffff9159ed73f808 R08: ffff9139ff89b780 R09: 0000000000000000 [Sat Jan 26 01:48:39 2019][717791.399612] R10: ffff9139ff89f140 R11: ffffdbb251ee1000 R12: ffff9159ed73f7a8 [Sat Jan 26 01:48:39 2019][717791.399613] R13: ffff912d55335c58 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:39 2019][717791.399614] FS: 00007f63fb26e700(0000) GS:ffff9139ff880000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:39 2019][717791.399615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:39 2019][717791.399616] CR2: 00007f63fb343000 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:39 2019][717791.399616] Call Trace: [Sat Jan 26 01:48:39 2019][717791.399618] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:39 2019][717791.399620] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:39 2019][717791.399631] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:39 2019][717791.399641] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:39 2019][717791.399643] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:39 2019][717791.399649] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:39 2019][717791.399657] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:39 2019][717791.399658] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:39 2019][717791.399669] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:39 2019][717791.399679] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:39 2019][717791.399693] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:39 2019][717791.399734] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.399736] [] ? __slab_free+0x81/0x2f0 [Sat Jan 26 01:48:39 2019][717791.399772] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.399774] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:39 2019][717791.399775] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:39 2019][717791.399777] [] ? __enqueue_entity+0x78/0x80 [Sat Jan 26 01:48:39 2019][717791.399817] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.399856] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.399894] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.399900] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:39 2019][717791.399936] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.399972] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.399974] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:39 2019][717791.399975] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:39 2019][717791.400011] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.400047] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.400048] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:39 2019][717791.400050] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:39 2019][717791.400052] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:39 2019][717791.400053] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:39 2019][717791.400054] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:39 2019][717791.408559] NMI watchdog: BUG: soft lockup - CPU#42 stuck for 23s! [mdt_io02_021:9200] [Sat Jan 26 01:48:39 2019][717791.408559] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:39 2019][717791.408607] CPU: 42 PID: 9200 Comm: mdt_io02_021 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:39 2019][717791.408607] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:39 2019][717791.408608] task: ffff9139ee33d140 ti: ffff9159ac6dc000 task.ti: ffff9159ac6dc000 [Sat Jan 26 01:48:39 2019][717791.408609] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x128/0x200 [Sat Jan 26 01:48:39 2019][717791.408613] RSP: 0018:ffff9159ac6df808 EFLAGS: 00000246 [Sat Jan 26 01:48:39 2019][717791.408614] RAX: 0000000000000000 RBX: ffff912d5536a990 RCX: 0000000001510000 [Sat Jan 26 01:48:39 2019][717791.408615] RDX: ffff9149ff81b780 RSI: 0000000001110101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:39 2019][717791.408615] RBP: ffff9159ac6df808 R08: ffff9149ff89b780 R09: 0000000000000000 [Sat Jan 26 01:48:39 2019][717791.408616] R10: ffff9149ff89f140 R11: ffffdbb2bdbdea00 R12: ffff9159ac6df7a8 [Sat Jan 26 01:48:39 2019][717791.408617] R13: ffff912d5536aa78 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:39 2019][717791.408618] FS: 00007f277e7c6700(0000) GS:ffff9149ff880000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:39 2019][717791.408619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:39 2019][717791.408620] CR2: 00007f0efc872000 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:39 2019][717791.408621] Call Trace: [Sat Jan 26 01:48:39 2019][717791.408623] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:39 2019][717791.408626] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:39 2019][717791.408638] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:39 2019][717791.408650] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:39 2019][717791.408652] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:39 2019][717791.408657] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:39 2019][717791.408666] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:39 2019][717791.408668] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:39 2019][717791.408678] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:39 2019][717791.408689] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:39 2019][717791.408704] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:39 2019][717791.408746] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.408749] [] ? load_balance+0x178/0x9a0 [Sat Jan 26 01:48:39 2019][717791.408752] [] ? __mutex_lock_slowpath+0x1b8/0x1d0 [Sat Jan 26 01:48:39 2019][717791.408753] [] ? __mutex_unlock_slowpath+0x70/0x90 [Sat Jan 26 01:48:39 2019][717791.408795] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.408836] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.408875] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.408882] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:39 2019][717791.408920] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.408957] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.408959] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:39 2019][717791.408961] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:39 2019][717791.408997] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.409034] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:39 2019][717791.409035] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:39 2019][717791.409037] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:39 2019][717791.409039] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:39 2019][717791.409040] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:39 2019][717791.409041] Code: 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 85 c0 <74> f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 [Sat Jan 26 01:48:40 2019][717791.434558] NMI watchdog: BUG: soft lockup - CPU#45 stuck for 22s! [mdt_io01_009:6867] [Sat Jan 26 01:48:40 2019][717791.434558] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:40 2019][717791.434600] CPU: 45 PID: 6867 Comm: mdt_io01_009 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:40 2019][717791.434600] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:40 2019][717791.434601] task: ffff9159f72dc100 ti: ffff9159a7778000 task.ti: ffff9159a7778000 [Sat Jan 26 01:48:40 2019][717791.434602] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Sat Jan 26 01:48:40 2019][717791.434605] RSP: 0018:ffff9159a777b808 EFLAGS: 00000246 [Sat Jan 26 01:48:40 2019][717791.434606] RAX: 0000000000000000 RBX: ffff912d552f5748 RCX: 0000000001690000 [Sat Jan 26 01:48:40 2019][717791.434606] RDX: ffff9129feedb780 RSI: 0000000000610101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:40 2019][717791.434607] RBP: ffff9159a777b808 R08: ffff9139ff8db780 R09: 0000000000000000 [Sat Jan 26 01:48:40 2019][717791.434608] R10: ffff9139ff8df140 R11: ffffdbb280ed4200 R12: ffff9159a777b7a8 [Sat Jan 26 01:48:40 2019][717791.434608] R13: ffff912d552f5830 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:40 2019][717791.434610] FS: 00007f14c509a740(0000) GS:ffff9139ff8c0000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:40 2019][717791.434611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:40 2019][717791.434611] CR2: 00007f14c4c85248 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:40 2019][717791.434612] Call Trace: [Sat Jan 26 01:48:40 2019][717791.434614] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:40 2019][717791.434616] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:40 2019][717791.434626] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:40 2019][717791.434637] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:40 2019][717791.434639] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:40 2019][717791.434644] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:40 2019][717791.434652] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:40 2019][717791.434654] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:40 2019][717791.434664] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:40 2019][717791.434674] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:40 2019][717791.434689] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:40 2019][717791.434729] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.434766] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.434802] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.434804] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:40 2019][717791.434805] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:40 2019][717791.434807] [] ? __enqueue_entity+0x78/0x80 [Sat Jan 26 01:48:40 2019][717791.434847] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.434886] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.434924] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.434931] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:40 2019][717791.434967] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.435003] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.435005] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:40 2019][717791.435006] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:40 2019][717791.435042] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.435077] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.435079] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:40 2019][717791.435080] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:40 2019][717791.435082] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:40 2019][717791.435084] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:40 2019][717791.435084] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Sat Jan 26 01:48:40 2019][717791.443559] NMI watchdog: BUG: soft lockup - CPU#46 stuck for 22s! [mdt_io02_005:6868] [Sat Jan 26 01:48:40 2019][717791.443560] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:40 2019][717791.443616] CPU: 46 PID: 6868 Comm: mdt_io02_005 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:40 2019][717791.443616] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:40 2019][717791.443617] task: ffff9159f72dd140 ti: ffff9159a625c000 task.ti: ffff9159a625c000 [Sat Jan 26 01:48:40 2019][717791.443619] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x128/0x200 [Sat Jan 26 01:48:40 2019][717791.443625] RSP: 0018:ffff9159a625f808 EFLAGS: 00000246 [Sat Jan 26 01:48:40 2019][717791.443626] RAX: 0000000000000000 RBX: ffff914338c8e7e8 RCX: 0000000001710000 [Sat Jan 26 01:48:40 2019][717791.443627] RDX: ffff9139ff81b780 RSI: 0000000001090101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:40 2019][717791.443628] RBP: ffff9159a625f808 R08: ffff9149ff8db780 R09: 0000000000000000 [Sat Jan 26 01:48:40 2019][717791.443629] R10: ffff9149ff8df140 R11: ffffdbb2a681e600 R12: ffff9159a625f7a8 [Sat Jan 26 01:48:40 2019][717791.443629] R13: ffff914338c8e8d0 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:40 2019][717791.443631] FS: 00007f277efc7700(0000) GS:ffff9149ff8c0000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:40 2019][717791.443632] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:40 2019][717791.443632] CR2: 00007f2781327000 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:40 2019][717791.443633] Call Trace: [Sat Jan 26 01:48:40 2019][717791.443637] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:40 2019][717791.443640] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:40 2019][717791.443653] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:40 2019][717791.443665] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:40 2019][717791.443669] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:40 2019][717791.443677] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:40 2019][717791.443685] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:40 2019][717791.443687] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:40 2019][717791.443699] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:40 2019][717791.443710] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:40 2019][717791.443730] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:40 2019][717791.443782] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.443784] [] ? load_balance+0x178/0x9a0 [Sat Jan 26 01:48:40 2019][717791.443826] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.443867] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.443906] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.443914] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:40 2019][717791.443951] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.443988] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.443990] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:40 2019][717791.443992] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:40 2019][717791.444029] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.444066] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:40 2019][717791.444068] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:40 2019][717791.444070] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:40 2019][717791.444072] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:40 2019][717791.444074] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:40 2019][717791.444074] Code: 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 85 c0 <74> f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 [Sat Jan 26 01:48:40 2019][717791.693577] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Sat Jan 26 01:48:40 2019][717791.693579] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 4 previous similar messages [Sat Jan 26 01:48:40 2019][717791.693583] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.101@o2ib7 (5): c: 7, oc: 0, rc: 8 [Sat Jan 26 01:48:40 2019][717791.693585] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 4 previous similar messages [Sat Jan 26 01:48:40 2019][717792.894587] Lustre: MGS: Received new LWP connection from 10.8.15.7@o2ib6, removing former export from same NID [Sat Jan 26 01:48:40 2019] [Sat Jan 26 01:48:40 2019][717798.467321] osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:41 2019][717798.572835] CPU: 0 PID: 6210 Comm: mdt_io00_001 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:41 2019][717798.585248] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:41 2019][717798.592904] task: ffff9149f865b0c0 ti: ffff9159fcb28000 task.ti: ffff9159fcb28000 [Sat Jan 26 01:48:41 2019][717798.600470] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:41 2019][717798.610497] RSP: 0018:ffff9159fcb2b808 EFLAGS: 00000246 [Sat Jan 26 01:48:41 2019][717798.615896] RAX: 0000000000000000 RBX: ffff912d5532cad0 RCX: 0000000000010000 [Sat Jan 26 01:48:41 2019][717798.623116] RDX: ffff9129fee5b780 RSI: 0000000000210101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:41 2019][717798.630333] RBP: ffff9159fcb2b808 R08: ffff9129fee1b780 R09: 0000000000000000 [Sat Jan 26 01:48:41 2019][717798.637555] R10: ffff9129fee1f140 R11: ffffdbb2a9c97400 R12: ffff9159fcb2b7a8 [Sat Jan 26 01:48:41 2019][717798.644774] R13: ffff912d5532cbb8 R14: 0000000000000000 R15: 1fffffffffffffff [Sat Jan 26 01:48:41 2019][717798.651994] FS: 00007fd7826e8740(0000) GS:ffff9129fee00000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:41 2019][717798.660189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:41 2019][717798.666042] CR2: 0000000000485110 CR3: 000000402c810000 CR4: 00000000003407f0 [Sat Jan 26 01:48:41 2019][717798.673261] Call Trace: [Sat Jan 26 01:48:41 2019][717798.675807] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:41 2019][717798.682168] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:41 2019][717798.687751] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:41 2019][717798.694549] [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] [Sat Jan 26 01:48:41 2019][717798.701858] [] ? ktime_get+0x52/0xe0 [Sat Jan 26 01:48:41 2019][717798.707174] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Sat Jan 26 01:48:41 2019][717798.715009] [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] [Sat Jan 26 01:48:41 2019][717798.721884] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:41 2019][717798.727638] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:41 2019][717798.735733] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:41 2019][717798.742707] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:41 2019][717798.749033] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:41 2019][717798.755558] [] ? __slab_free+0x81/0x2f0 [Sat Jan 26 01:48:41 2019][717798.761158] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Sat Jan 26 01:48:41 2019][717798.768116] [] ? update_curr+0x14c/0x1e0 [Sat Jan 26 01:48:41 2019][717798.773773] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:41 2019][717798.780245] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:41 2019][717798.787066] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:41 2019][717798.794191] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:41 2019][717798.801948] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:41 2019][717798.809113] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:41 2019][717798.816886] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:41 2019][717798.823756] [] ? wake_up_state+0x20/0x20 [Sat Jan 26 01:48:41 2019][717798.829444] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:41 2019][717798.835822] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:41 2019][717798.843301] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:41 2019][717798.848265] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:41 2019][717798.854444] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:41 2019][717798.860970] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:41 2019][717798.867148] Code: 13 48 c1 ea 0d [717798.870274] Lustre: mdt_io: This server is not able to keep up with request traffic (cpu-bound). [Sat Jan 26 01:48:41 2019][717798.870276] Lustre: Skipped 2 previous similar messages [Sat Jan 26 01:48:41 2019][717798.870279] Lustre: 9008:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=3, svcEst=31, delay=32732 [Sat Jan 26 01:48:41 2019][717798.870282] Lustre: 9008:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 2 previous similar messages [Sat Jan 26 01:48:41 2019][717798.870288] Lustre: 6870:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-21s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff9139f6ebd450 x1623623379216416/t0(0) o35->85c45b71-10aa-be9f-071b-9a9c50548db2@10.8.15.7@o2ib6:200/0 lens 392/456 e 0 to 0 dl 1548496140 ref 2 fl Interpret:/0/0 rc 0/0 [Sat Jan 26 01:48:41 2019][717798.870301] LustreError: 7215:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff9129a3c3ac50 x1623623379250400/t0(0) o4->85c45b71-10aa-be9f-071b-9a9c50548db2@10.8.15.7@o2ib6:232/0 lens 488/448 e 0 to 0 dl 1548496172 ref 1 fl Interpret:/0/0 rc 0/0 [Sat Jan 26 01:48:41 2019][717798.870314] Lustre: 8975:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 23s req@ffff9129d8ba0050 x1623671563125376/t0(0) o4->f9afcafe-b7fa-076a-0d87-fc0f78fd02d0@10.8.6.20@o2ib6:0/0 lens 3000/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 [Sat Jan 26 01:48:41 2019][717798.870319] LustreError: 5881:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff91435f519600 [Sat Jan 26 01:48:41 2019][717798.870336] Lustre: fir-MDT0002: Bulk IO write error with 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6), client will retry: rc = -110 [Sat Jan 26 01:48:41 2019][717798.870369] Lustre: MGS: Received new LWP connection from 10.0.10.104@o2ib7, removing former export from same NID [Sat Jan 26 01:48:41 2019][717798.870371] Lustre: Skipped 1 previous similar message [Sat Jan 26 01:48:41 2019][717798.870384] Lustre: MGS: Connection restored to 10.0.10.104@o2ib7 (at 10.0.10.104@o2ib7) [Sat Jan 26 01:48:41 2019][717798.870385] Lustre: Skipped 3 previous similar messages [Sat Jan 26 01:48:41 2019][717798.870448] LustreError: 7097:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.15.7@o2ib6: deadline 12:1s ago [Sat Jan 26 01:48:41 2019][717798.870448] req@ffff9129a3c3d050 x1623623379252896/t0(0) o4->85c45b71-10aa-be9f-071b-9a9c50548db2@10.8.15.7@o2ib6:220/0 lens 792/0 e 0 to 0 dl 1548496160 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [Sat Jan 26 01:48:41 2019][717798.870456] Lustre: 7097:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (12:1s); client may timeout. req@ffff9129a3c3d050 x1623623379252896/t0(0) o4->85c45b71-10aa-be9f-071b-9a9c50548db2@10.8.15.7@o2ib6:220/0 lens 792/0 e 0 to 0 dl 1548496160 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [Sat Jan 26 01:48:41 2019][717798.870541] LustreError: 5879:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff91290d1bb400 [Sat Jan 26 01:48:41 2019][717798.870632] LNetError: 5880:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) [Sat Jan 26 01:48:41 2019][717798.870960] LustreError: 5886:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9149d4724a00 [Sat Jan 26 01:48:41 2019][717798.871083] Lustre: fir-MDT0002: Client 9daa8e10-c154-ca96-d85d-3f6355b9ec3f (at 10.8.2.5@o2ib6) reconnecting [Sat Jan 26 01:48:41 2019][717799.114751] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [mdt_io00_003:6773] [Sat Jan 26 01:48:41 2019][717799.114752] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:41 2019][717799.114822] CPU: 8 PID: 6773 Comm: mdt_io00_003 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:41 2019][717799.114822] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:41 2019][717799.114824] task: ffff9159a27630c0 ti: ffff9159b55cc000 task.ti: ffff9159b55cc000 [Sat Jan 26 01:48:41 2019][717799.114825] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:41 2019][717799.114833] RSP: 0018:ffff9159b55cf8f0 EFLAGS: 00000246 [Sat Jan 26 01:48:41 2019][717799.114834] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000410000 [Sat Jan 26 01:48:41 2019][717799.114834] RDX: ffff9149ff75b780 RSI: 0000000000b10101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:41 2019][717799.114835] RBP: ffff9159b55cf8f0 R08: ffff9129fee9b780 R09: 0000000000000000 [Sat Jan 26 01:48:41 2019][717799.114836] R10: 000007da9ebd7000 R11: ffff9128f80640e0 R12: ffff9129a5eccb78 [Sat Jan 26 01:48:41 2019][717799.114836] R13: ffff9159b55cf888 R14: 0000000000000000 R15: ffff911b2907cc00 [Sat Jan 26 01:48:41 2019][717799.114838] FS: 00007fa73e72a880(0000) GS:ffff9129fee80000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:41 2019][717799.114839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:41 2019][717799.114839] CR2: 00007ffc4a456c58 CR3: 0000003013314000 CR4: 00000000003407e0 [Sat Jan 26 01:48:41 2019][717799.114840] Call Trace: [Sat Jan 26 01:48:41 2019][717799.114846] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:41 2019][717799.114849] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:41 2019][717799.114865] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:41 2019][717799.114874] [] ldiskfs_map_blocks+0x210/0x700 [ldiskfs] [Sat Jan 26 01:48:41 2019][717799.114878] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:41 2019][717799.114890] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:41 2019][717799.114901] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:41 2019][717799.114920] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:41 2019][717799.114983] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:41 2019][717799.114988] [] ? load_balance+0x178/0x9a0 [Sat Jan 26 01:48:41 2019][717799.115031] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:41 2019][717799.115073] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:41 2019][717799.115113] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:41 2019][717799.115122] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:41 2019][717799.115160] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:41 2019][717799.115198] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:41 2019][717799.115202] [] ? wake_up_state+0x20/0x20 [Sat Jan 26 01:48:41 2019][717799.115239] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:41 2019][717799.115276] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:41 2019][717799.115279] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:41 2019][717799.115281] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:41 2019][717799.115284] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:41 2019][717799.115286] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:41 2019][717799.115286] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:42 2019][717799.199753] NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [mdt_io02_002:6217] [Sat Jan 26 01:48:42 2019][717799.199753] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:42 2019][717799.199817] CPU: 18 PID: 6217 Comm: mdt_io02_002 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:42 2019][717799.199817] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:42 2019][717799.199818] task: ffff9159b55c5140 ti: ffff9159a837c000 task.ti: ffff9159a837c000 [Sat Jan 26 01:48:42 2019][717799.199819] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x128/0x200 [Sat Jan 26 01:48:42 2019][717799.199825] RSP: 0018:ffff9159a837f8f0 EFLAGS: 00000246 [Sat Jan 26 01:48:42 2019][717799.199825] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000910000 [Sat Jan 26 01:48:42 2019][717799.199826] RDX: ffff9129ff01b780 RSI: 0000000001010101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:42 2019][717799.199827] RBP: ffff9159a837f8f0 R08: ffff9149ff71b780 R09: 0000000000000000 [Sat Jan 26 01:48:42 2019][717799.199827] R10: 000007da9ebd7000 R11: ffff9128f80640e0 R12: ffff9129a5ece378 [Sat Jan 26 01:48:42 2019][717799.199828] R13: ffff9159a837f888 R14: 0000000000000000 R15: ffff913aa9a6b000 [Sat Jan 26 01:48:42 2019][717799.199829] FS: 00007fe2d3435880(0000) GS:ffff9149ff700000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:42 2019][717799.199830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:42 2019][717799.199831] CR2: 000055fdd4177300 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:42 2019][717799.199832] Call Trace: [Sat Jan 26 01:48:42 2019][717799.199835] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:42 2019][717799.199838] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:42 2019][717799.199851] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:42 2019][717799.199859] [] ldiskfs_map_blocks+0x210/0x700 [ldiskfs] [Sat Jan 26 01:48:42 2019][717799.199862] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:42 2019][717799.199876] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:42 2019][717799.199886] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:42 2019][717799.199907] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:42 2019][717799.199964] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.199968] [] ? __mutex_lock_slowpath+0x1b8/0x1d0 [Sat Jan 26 01:48:42 2019][717799.200008] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.200048] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.200086] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.200096] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:42 2019][717799.200132] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.200168] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.200172] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:42 2019][717799.200174] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:42 2019][717799.200210] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.200246] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.200248] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:42 2019][717799.200249] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:42 2019][717799.200252] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:42 2019][717799.200253] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:42 2019][717799.200254] Code: 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 85 c0 <74> f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 [Sat Jan 26 01:48:42 2019][717799.233755] NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [mdt_io02_004:6782] [Sat Jan 26 01:48:42 2019][717799.233756] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:42 2019][717799.233815] CPU: 22 PID: 6782 Comm: mdt_io02_004 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:42 2019][717799.233816] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:42 2019][717799.233817] task: ffff9159f774e180 ti: ffff91599d71c000 task.ti: ffff91599d71c000 [Sat Jan 26 01:48:42 2019][717799.233818] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:42 2019][717799.233825] RSP: 0018:ffff91599d71f8f0 EFLAGS: 00000246 [Sat Jan 26 01:48:42 2019][717799.233826] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000b10000 [Sat Jan 26 01:48:42 2019][717799.233827] RDX: ffff9149ff71b780 RSI: 0000000000910101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:42 2019][717799.233827] RBP: ffff91599d71f8f0 R08: ffff9149ff75b780 R09: 0000000000000000 [Sat Jan 26 01:48:42 2019][717799.233828] R10: 000007da9ebd7000 R11: ffff9128f80640e0 R12: ffff9129a5ece378 [Sat Jan 26 01:48:42 2019][717799.233829] R13: ffff91599d71f888 R14: ffff9149ff75ac00 R15: ffff9159f774e1e8 [Sat Jan 26 01:48:42 2019][717799.233830] FS: 00007f21dc71d740(0000) GS:ffff9149ff740000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:42 2019][717799.233831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:42 2019][717799.233831] CR2: 00007f2781327000 CR3: 0000004031fb0000 CR4: 00000000003407e0 [Sat Jan 26 01:48:42 2019][717799.233832] Call Trace: [Sat Jan 26 01:48:42 2019][717799.233837] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:42 2019][717799.233840] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:42 2019][717799.233852] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:42 2019][717799.233860] [] ldiskfs_map_blocks+0x210/0x700 [ldiskfs] [Sat Jan 26 01:48:42 2019][717799.233863] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:42 2019][717799.233875] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:42 2019][717799.233884] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:42 2019][717799.233904] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:42 2019][717799.233960] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.233963] [] ? load_balance+0x178/0x9a0 [Sat Jan 26 01:48:42 2019][717799.233997] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.234031] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.234063] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.234071] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:42 2019][717799.234102] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.234133] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.234135] [] ? default_wake_function+0x12/0x20 [Sat Jan 26 01:48:42 2019][717799.234138] [] ? __wake_up_common+0x5b/0x90 [Sat Jan 26 01:48:42 2019][717799.234168] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.234198] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:42 2019][717799.234200] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:42 2019][717799.234201] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:42 2019][717799.234204] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:42 2019][717799.234205] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:42 2019][717799.234206] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:42 2019][717799.321756] NMI watchdog: BUG: soft lockup - CPU#32 stuck for 22s! [mdt_io00_010:7100] [Sat Jan 26 01:48:42 2019][717799.321756] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Sat Jan 26 01:48:42 2019][717799.321802] CPU: 32 PID: 7100 Comm: mdt_io00_010 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Sat Jan 26 01:48:42 2019][717799.321802] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Sat Jan 26 01:48:42 2019][717799.321803] task: ffff9159f7c01040 ti: ffff9159a2be0000 task.ti: ffff9159a2be0000 [Sat Jan 26 01:48:42 2019][717799.321804] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Sat Jan 26 01:48:42 2019][717799.321807] RSP: 0018:ffff9159a2be38f0 EFLAGS: 00000246 [Sat Jan 26 01:48:42 2019][717799.321807] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000001010000 [Sat Jan 26 01:48:42 2019][717799.321808] RDX: ffff9149ff69b780 RSI: 0000000000510101 RDI: ffff9149fce5fc80 [Sat Jan 26 01:48:42 2019][717799.321809] RBP: ffff9159a2be38f0 R08: ffff9129ff01b780 R09: 0000000000000000 [Sat Jan 26 01:48:42 2019][717799.321809] R10: 000007da9ebd7000 R11: ffff9128f80640e0 R12: ffff9129a5ecd378 [Sat Jan 26 01:48:42 2019][717799.321810] R13: ffff9159a2be3888 R14: ffff9129feedac00 R15: ffff9159f7c010a8 [Sat Jan 26 01:48:42 2019][717799.321811] FS: 00007f7812319740(0000) GS:ffff9129ff000000(0000) knlGS:0000000000000000 [Sat Jan 26 01:48:42 2019][717799.321812] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Sat Jan 26 01:48:42 2019][717799.321813] CR2: 00007f7811b9d1a0 CR3: 000000402c810000 CR4: 00000000003407e0 [Sat Jan 26 01:48:43 2019][717799.321813] Call Trace: [Sat Jan 26 01:48:43 2019][717799.321815] [] queued_spin_lock_slowpath+0xb/0xf [Sat Jan 26 01:48:43 2019][717799.321817] [] _raw_spin_lock+0x20/0x30 [Sat Jan 26 01:48:43 2019][717799.321828] [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] [Sat Jan 26 01:48:43 2019][717799.321836] [] ldiskfs_map_blocks+0x210/0x700 [ldiskfs] [Sat Jan 26 01:48:43 2019][717799.321838] [] ? ktime_get_ts64+0x52/0xf0 [Sat Jan 26 01:48:43 2019][717799.321848] [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] [Sat Jan 26 01:48:43 2019][717799.321858] [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] [Sat Jan 26 01:48:43 2019][717799.321873] [] mdt_obd_preprw+0xc37/0xe20 [mdt] [Sat Jan 26 01:48:43 2019][717799.321914] [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] [Sat Jan 26 01:48:43 2019][717799.321921] [] ? __slab_free+0x81/0x2f0 [Sat Jan 26 01:48:43 2019][717799.321958] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Sat Jan 26 01:48:43 2019][717799.321961] [] ? __mutex_lock_slowpath+0x1b8/0x1d0 [Sat Jan 26 01:48:43 2019][717799.321963] [] ? account_entity_dequeue+0xae/0xd0 [Sat Jan 26 01:48:43 2019][717799.321965] [] ? __mutex_unlock_slowpath+0x70/0x90 [Sat Jan 26 01:48:43 2019][717799.322005] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Sat Jan 26 01:48:43 2019][717799.322044] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Sat Jan 26 01:48:43 2019][717799.322082] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Sat Jan 26 01:48:43 2019][717799.322088] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Sat Jan 26 01:48:43 2019][717799.322124] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Sat Jan 26 01:48:43 2019][717799.322160] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Sat Jan 26 01:48:43 2019][717799.322162] [] ? wake_up_state+0x20/0x20 [Sat Jan 26 01:48:43 2019][717799.322197] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Sat Jan 26 01:48:43 2019][717799.322233] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Sat Jan 26 01:48:43 2019][717799.322234] [] kthread+0xd1/0xe0 [Sat Jan 26 01:48:43 2019][717799.322236] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:43 2019][717799.322238] [] ret_from_fork_nospec_begin+0xe/0x21 [Sat Jan 26 01:48:43 2019][717799.322239] [] ? insert_kthread_work+0x40/0x40 [Sat Jan 26 01:48:43 2019][717799.322240] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:43 2019][717799.870787] LustreError: 8976:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff9129d8ba3c50 x1623672576978640/t0(0) o4->9daa8e10-c154-ca96-d85d-3f6355b9ec3f@10.8.2.5@o2ib6:231/0 lens 488/448 e 0 to 0 dl 1548496171 ref 1 fl Interpret:/0/0 rc 0/0 [Sat Jan 26 01:48:43 2019][717799.870790] LustreError: 8976:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 5 previous similar messages [Sat Jan 26 01:48:43 2019][717799.870811] Lustre: fir-MDT0002: Bulk IO write error with 9daa8e10-c154-ca96-d85d-3f6355b9ec3f (at 10.8.2.5@o2ib6), client will retry: rc = -110 [Sat Jan 26 01:48:43 2019][717799.870813] Lustre: Skipped 5 previous similar messages [Sat Jan 26 01:48:43 2019] [Sat Jan 26 01:48:43 2019][717800.792942] 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Sat Jan 26 01:48:43 2019][717800.812042] Lustre: 5903:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1548496132/real 1548496161] req@ffff911e32e9b600 x1623664975828400/t0(0) o13->fir-OST0024-osc-MDT0000@10.0.10.107@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1548496139 ref 1 fl Rpc:RX/0/ffffffff rc 0/-1 [Sat Jan 26 01:48:43 2019][717800.840262] Lustre: 5903:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 23 previous similar messages [Sat Jan 26 01:48:51 2019][717808.893900] Lustre: fir-MDT0000: Client f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6) reconnecting [Sat Jan 26 01:48:51 2019][717808.903989] Lustre: Skipped 9 previous similar messages [Sat Jan 26 01:49:05 2019][717823.030241] Lustre: fir-MDT0002: Connection restored to 16f622df-9b71-a16e-7fd5-e6f7aaabcc11 (at 10.8.2.5@o2ib6) [Sat Jan 26 01:49:05 2019][717823.040514] Lustre: Skipped 45 previous similar messages [Sat Jan 26 01:49:05 2019][717823.302545] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [Sat Jan 26 01:49:05 2019][717823.318880] LustreError: Skipped 3 previous similar messages [Sat Jan 26 01:49:06 2019][717824.102430] Lustre: MGS: Received new LWP connection from 10.9.0.2@o2ib4, removing former export from same NID [Sat Jan 26 01:49:06 2019][717824.112548] Lustre: Skipped 4 previous similar messages [Sat Jan 26 01:49:16 2019][717833.948805] Lustre: fir-MDT0002: Client f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6) reconnecting [Sat Jan 26 01:49:16 2019][717833.958919] Lustre: Skipped 13 previous similar messages [Sat Jan 26 01:49:20 2019][717838.311363] Lustre: MGS: Received new LWP connection from 10.0.10.107@o2ib7, removing former export from same NID [Sat Jan 26 01:49:20 2019][717838.321714] Lustre: Skipped 5 previous similar messages [Sat Jan 26 02:38:32 2019][720790.097860] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 02:51:49 2019][721587.231437] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 02:51:49 2019][721587.243090] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 03:22:17 2019][723415.588665] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 03:22:17 2019][723415.600315] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 03:26:11 2019][723649.485008] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 03:30:03 2019][723880.695217] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 03:30:03 2019][723880.706865] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 03:48:50 2019][725008.433221] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 03:48:50 2019][725008.444877] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 03:55:31 2019][725409.572963] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 04:02:15 2019][725813.031787] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 04:02:38 2019][725835.975454] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 04:02:38 2019][725835.987103] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 04:03:33 2019][725890.882740] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 04:03:33 2019][725890.894392] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 04:30:33 2019][727510.991780] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 04:33:45 2019][727703.651043] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 04:33:45 2019][727703.662700] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 04:39:23 2019][728041.611279] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 04:49:56 2019][728674.103880] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 07:47:18 2019][739316.962637] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 07:53:09 2019][739667.628584] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 08:11:43 2019][740781.614929] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 08:15:20 2019][740998.407577] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 08:15:20 2019][740998.419231] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 08:16:40 2019][741078.977769] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 08:18:54 2019][741212.578600] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 08:18:54 2019][741212.590253] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 08:21:24 2019][741362.959008] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 08:21:24 2019][741362.970661] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 08:48:24 2019][742982.632977] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 08:48:24 2019][742982.644625] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 09:03:59 2019][743917.397729] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 09:04:17 2019][743935.445591] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 09:04:17 2019][743935.457244] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 09:12:49 2019][744448.049843] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 09:12:49 2019][744448.061520] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 09:47:20 2019][746518.274681] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 09:47:39 2019][746537.762164] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 09:55:37 2019][747015.640342] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 10:18:04 2019][748362.760785] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 10:24:48 2019][748766.376936] Lustre: DEBUG MARKER: Sat Jan 26 10:25:28 2019 [Sat Jan 26 10:24:48 2019][748766.376936] [Sat Jan 26 10:31:48 2019][749187.020136] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 11:01:31 2019][750970.015455] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 11:13:51 2019][751709.614594] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 11:16:27 2019][751866.098053] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 11:16:56 2019][751895.226476] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 11:16:56 2019][751895.238141] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 11:17:01 2019][751899.903271] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 11:17:01 2019][751899.914923] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 11:17:08 2019][751906.460775] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 11:17:17 2019][751915.802864] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Sat Jan 26 11:17:17 2019][751915.812519] Lustre: Skipped 73 previous similar messages [Sat Jan 26 11:32:13 2019][752811.529229] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 11:39:16 2019][753234.343135] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 11:48:15 2019][753773.969390] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 11:50:56 2019][753934.740404] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 11:50:56 2019][753934.752061] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 12:04:36 2019][754754.871096] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:08:20 2019][754978.803167] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:08:20 2019][754978.814844] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 12:11:05 2019][755144.004337] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:11:05 2019][755144.016023] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 12:33:51 2019][756510.362948] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:33:51 2019][756510.374641] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 12:34:54 2019][756572.960935] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:34:54 2019][756572.972590] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 12:35:00 2019][756579.348876] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:39:09 2019][756827.622002] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:39:09 2019][756827.633661] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 12:39:15 2019][756834.384801] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:41:06 2019][756945.098526] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:42:42 2019][757040.924677] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:42:42 2019][757040.936337] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 12:43:34 2019][757092.679799] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Sat Jan 26 12:43:34 2019][757092.689975] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.104@o2ib7 (5): c: 0, oc: 0, rc: 8 [Sat Jan 26 12:43:35 2019][757094.398852] Lustre: 9015:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1548535448/real 1548535448] req@ffff912e55b0a100 x1623666502060672/t0(0) o104->fir-MDT0002@10.8.15.7@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1548535455 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Sat Jan 26 12:43:36 2019][757094.615867] Lustre: fir-OST002e-osc-MDT0002: Connection to fir-OST002e (at 10.0.10.107@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Sat Jan 26 12:43:36 2019][757094.632027] Lustre: Skipped 24 previous similar messages [Sat Jan 26 12:43:36 2019][757094.679854] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Sat Jan 26 12:43:36 2019][757094.690021] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 7 previous similar messages [Sat Jan 26 12:43:36 2019][757094.700188] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.105@o2ib7 (7): c: 0, oc: 0, rc: 8 [Sat Jan 26 12:43:36 2019][757094.712172] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 7 previous similar messages [Sat Jan 26 12:43:36 2019][757094.997993] Lustre: fir-MDT0002: Client cc9aeebd-5eb1-fd0a-28e5-4a7c51fd2782 (at 10.8.6.33@o2ib6) reconnecting [Sat Jan 26 12:43:36 2019][757095.008083] Lustre: Skipped 27 previous similar messages [Sat Jan 26 12:43:36 2019][757095.013534] Lustre: fir-MDT0002: Connection restored to c50cb93a-a125-4e21-ca5d-0351a87eda04 (at 10.8.6.33@o2ib6) [Sat Jan 26 12:43:36 2019][757095.023882] Lustre: Skipped 2 previous similar messages [Sat Jan 26 12:43:36 2019][757095.121100] LustreError: 6779:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff9149d5620050 x1623661982854992/t0(0) o4->cc9aeebd-5eb1-fd0a-28e5-4a7c51fd2782@10.8.6.33@o2ib6:292/0 lens 488/448 e 0 to 0 dl 1548535492 ref 1 fl Interpret:/0/0 rc 0/0 [Sat Jan 26 12:43:36 2019][757095.121210] LustreError: 5880:0:(events.c:305:request_in_callback()) event type 2, status -103, service mdt_io [Sat Jan 26 12:43:36 2019][757095.121221] LustreError: 8957:0:(pack_generic.c:590:__lustre_unpack_msg()) message length 0 too small for magic/version check [Sat Jan 26 12:43:36 2019][757095.121228] LustreError: 5880:0:(events.c:450:server_bulk_callback()) event type 3, status -103, desc ffff9139ff5acc00 [Sat Jan 26 12:43:36 2019][757095.121231] LustreError: 8957:0:(sec.c:2068:sptlrpc_svc_unwrap_request()) error unpacking request from 12345-10.8.2.5@o2ib6 x1623675189195648 [Sat Jan 26 12:43:36 2019][757095.121245] LustreError: 8952:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE req@ffff912d43325c50 x1623571402685952/t0(0) o4->9b3a0481-8c2f-a606-18ad-7a06b856368e@10.8.3.11@o2ib6:293/0 lens 488/448 e 0 to 0 dl 1548535493 ref 1 fl Interpret:/0/0 rc 0/0 [Sat Jan 26 12:43:36 2019][757095.121262] Lustre: fir-MDT0000: Bulk IO write error with 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6), client will retry: rc = -110 [Sat Jan 26 12:43:36 2019][757095.121263] Lustre: Skipped 1 previous similar message [Sat Jan 26 12:43:36 2019][757095.121379] LustreError: 5881:0:(events.c:450:server_bulk_callback()) event type 3, status -103, desc ffff9128a4956a00 [Sat Jan 26 12:43:36 2019][757095.121450] LustreError: 5881:0:(events.c:305:request_in_callback()) event type 2, status -103, service mdt_io [Sat Jan 26 12:43:36 2019][757095.121566] LustreError: 5881:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9139fb619800 [Sat Jan 26 12:43:36 2019][757095.121571] LustreError: 5880:0:(events.c:305:request_in_callback()) event type 2, status -103, service mdt_io [Sat Jan 26 12:43:36 2019][757095.121883] LustreError: 5882:0:(events.c:305:request_in_callback()) event type 2, status -103, service mdt_io [Sat Jan 26 12:44:13 2019][757131.454041] Lustre: fir-MDT0000: Client 499c50f1-e88f-d71b-3734-bc99bfed13f5 (at 10.8.9.6@o2ib6) reconnecting [Sat Jan 26 12:44:13 2019][757131.464049] Lustre: Skipped 1 previous similar message [Sat Jan 26 12:44:13 2019][757131.469317] Lustre: fir-MDT0000: Connection restored to 62dcefe3-6231-7b23-eb15-3a1ee7a91bfc (at 10.8.9.6@o2ib6) [Sat Jan 26 12:44:13 2019][757131.479597] Lustre: Skipped 35 previous similar messages [Sat Jan 26 12:44:14 2019][757133.403632] Lustre: fir-MDT0002: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Sat Jan 26 12:44:14 2019][757133.413990] Lustre: Skipped 1 previous similar message [Sat Jan 26 12:44:42 2019][757161.054561] LustreError: 6209:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(131072) req@ffff9129a3c3d050 x1623675189195664/t0(0) o4->9daa8e10-c154-ca96-d85d-3f6355b9ec3f@10.8.2.5@o2ib6:341/0 lens 488/448 e 2 to 0 dl 1548535541 ref 1 fl Interpret:/0/0 rc 0/0 [Sat Jan 26 12:44:42 2019][757161.054588] Lustre: fir-MDT0002: Bulk IO write error with f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6), client will retry: rc = -110 [Sat Jan 26 12:44:42 2019][757161.054589] Lustre: fir-MDT0000: Bulk IO write error with 28764730-adc8-1b1a-0aa7-9bb3363d4889 (at 10.9.101.59@o2ib4), client will retry: rc = -110 [Sat Jan 26 12:44:42 2019][757161.054590] Lustre: Skipped 2 previous similar messages [Sat Jan 26 12:44:42 2019][757161.054591] Lustre: Skipped 2 previous similar messages [Sat Jan 26 12:44:42 2019][757161.059332] LustreError: 9190:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE req@ffff912d43324c50 x1623571402686032/t0(0) o4->9b3a0481-8c2f-a606-18ad-7a06b856368e@10.8.3.11@o2ib6:344/0 lens 488/448 e 2 to 0 dl 1548535544 ref 1 fl Interpret:/0/0 rc 0/0 [Sat Jan 26 12:44:42 2019][757161.059334] LustreError: 9190:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 1 previous similar message [Sat Jan 26 12:44:42 2019][757161.149911] LustreError: 6209:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages [Sat Jan 26 12:46:05 2019][757243.880108] Lustre: fir-MDT0000: Client 28764730-adc8-1b1a-0aa7-9bb3363d4889 (at 10.9.101.59@o2ib4) reconnecting [Sat Jan 26 12:46:05 2019][757243.890377] Lustre: Skipped 2 previous similar messages [Sat Jan 26 12:46:05 2019][757243.895719] Lustre: fir-MDT0000: Connection restored to b45f97c4-4cf7-fe2a-965d-6f2f61bac156 (at 10.9.101.59@o2ib4) [Sat Jan 26 12:47:45 2019][757344.084840] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:47:45 2019][757344.096495] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 12:49:45 2019][757463.516070] Lustre: fir-MDT0000: Client 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) reconnecting [Sat Jan 26 12:49:45 2019][757463.526188] Lustre: fir-MDT0000: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Sat Jan 26 12:50:31 2019][757510.045803] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:50:31 2019][757510.057468] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 12:50:55 2019][757533.677855] perf: interrupt took too long (3962 > 3915), lowering kernel.perf_event_max_sample_rate to 50000 [Sat Jan 26 12:52:40 2019][757638.915971] Lustre: fir-MDT0002: Client f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6) reconnecting [Sat Jan 26 12:52:40 2019][757638.926095] Lustre: fir-MDT0002: Connection restored to 6aadd60a-4be2-83fa-2199-3d4225023e35 (at 10.8.6.20@o2ib6) [Sat Jan 26 12:54:38 2019][757757.032718] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 12:54:38 2019][757757.044496] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 12:57:45 2019][757944.010332] Lustre: fir-MDT0002: Client 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) reconnecting [Sat Jan 26 12:57:45 2019][757944.020473] Lustre: fir-MDT0002: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Sat Jan 26 13:00:30 2019][758108.908396] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 13:00:30 2019][758108.920057] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 13:04:57 2019][758376.067262] Lustre: fir-MDT0000: Client 9b3a0481-8c2f-a606-18ad-7a06b856368e (at 10.8.3.11@o2ib6) reconnecting [Sat Jan 26 13:04:57 2019][758376.067347] Lustre: fir-MDT0002: Connection restored to e421d59a-4482-eafc-ea85-c173e2c58e62 (at 10.8.3.11@o2ib6) [Sat Jan 26 13:04:57 2019][758376.087705] Lustre: Skipped 1 previous similar message [Sat Jan 26 13:10:24 2019][758702.764707] perf: interrupt took too long (4967 > 4952), lowering kernel.perf_event_max_sample_rate to 40000 [Sat Jan 26 13:18:26 2019][759185.129313] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 13:18:26 2019][759185.140965] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 13:25:24 2019][759603.278201] Lustre: fir-MDT0000: Client 9daa8e10-c154-ca96-d85d-3f6355b9ec3f (at 10.8.2.5@o2ib6) reconnecting [Sat Jan 26 13:25:24 2019][759603.288233] Lustre: fir-MDT0000: Connection restored to 16f622df-9b71-a16e-7fd5-e6f7aaabcc11 (at 10.8.2.5@o2ib6) [Sat Jan 26 13:25:24 2019][759603.298504] Lustre: Skipped 1 previous similar message [Sat Jan 26 13:46:16 2019][760854.928353] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 13:46:16 2019][760854.940045] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 13:48:30 2019][760988.555983] Lustre: fir-MDT0000: Client f9afcafe-b7fa-076a-0d87-fc0f78fd02d0 (at 10.8.6.20@o2ib6) reconnecting [Sat Jan 26 13:48:30 2019][760988.566094] Lustre: fir-MDT0000: Connection restored to 6aadd60a-4be2-83fa-2199-3d4225023e35 (at 10.8.6.20@o2ib6) [Sat Jan 26 13:52:21 2019][761219.748992] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 13:52:21 2019][761219.760641] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 13:54:07 2019][761325.887241] Lustre: fir-MDT0000: Client cc9aeebd-5eb1-fd0a-28e5-4a7c51fd2782 (at 10.8.6.33@o2ib6) reconnecting [Sat Jan 26 13:54:07 2019][761325.897349] Lustre: fir-MDT0000: Connection restored to c50cb93a-a125-4e21-ca5d-0351a87eda04 (at 10.8.6.33@o2ib6) [Sat Jan 26 13:58:02 2019][761560.670347] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 14:04:26 2019][761945.086164] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 14:04:26 2019][761945.097836] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 14:17:24 2019][762723.478558] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 14:17:24 2019][762723.490208] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 14:34:11 2019][763730.201578] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 14:34:11 2019][763730.213241] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 14:37:02 2019][763900.859835] LNetError: 5875:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Sat Jan 26 14:37:02 2019][763900.872428] LNetError: 5875:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages [Sat Jan 26 14:55:48 2019][765026.856918] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 14:55:48 2019][765026.868616] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 15:33:13 2019][767271.674139] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:02:00 2019][768999.451590] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:04:58 2019][769177.685248] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:05:18 2019][769197.518693] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:06:00 2019][769238.864124] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:06:00 2019][769238.875784] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 16:06:13 2019][769252.324662] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:06:52 2019][769291.566013] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:06:52 2019][769291.577699] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 16:07:42 2019][769340.883239] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:07:42 2019][769340.894905] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 16:10:17 2019][769496.120858] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:10:17 2019][769496.132540] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Sat Jan 26 16:19:32 2019][770050.887813] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:19:32 2019][770050.899476] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 16:27:12 2019][770510.891855] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:27:12 2019][770510.903525] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Sat Jan 26 16:37:20 2019][771119.030486] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:37:20 2019][771119.042135] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Sat Jan 26 16:50:23 2019][771902.636317] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 16:50:23 2019][771902.647967] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 17:05:58 2019][772837.318312] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 17:05:58 2019][772837.329965] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 17:17:06 2019][773505.770470] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 17:34:19 2019][774538.110661] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 17:34:19 2019][774538.122311] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 17:44:20 2019][775139.040430] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 17:44:20 2019][775139.052082] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 17:58:26 2019][775985.274693] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 17:58:26 2019][775985.286345] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Sat Jan 26 18:25:49 2019][777628.024031] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 18:25:49 2019][777628.035702] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Sat Jan 26 18:27:12 2019][777711.671219] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 18:27:12 2019][777711.682868] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Sat Jan 26 18:30:15 2019][777893.928048] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 18:30:15 2019][777893.939703] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 18:35:48 2019][778226.963587] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 18:35:48 2019][778226.975235] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Sat Jan 26 18:49:37 2019][779056.673753] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 18:49:37 2019][779056.685439] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 19:00:08 2019][779687.436246] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 19:00:08 2019][779687.447899] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Sat Jan 26 19:15:37 2019][780616.499587] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 19:15:37 2019][780616.511273] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 19:29:53 2019][781472.209805] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 19:29:53 2019][781472.221461] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 19:40:47 2019][782126.561354] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 19:40:47 2019][782126.573012] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Sat Jan 26 19:51:53 2019][782792.341809] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 19:51:53 2019][782792.353465] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Sat Jan 26 20:06:00 2019][783639.143145] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 20:06:00 2019][783639.154822] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Sat Jan 26 20:22:14 2019][784613.756483] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 20:22:14 2019][784613.768139] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Sat Jan 26 20:32:21 2019][785220.522494] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 20:42:39 2019][785838.636998] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sat Jan 26 20:42:39 2019][785838.648654] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 20:58:29 2019][786788.506107] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 20:58:29 2019][786788.517764] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 21:22:43 2019][788242.486238] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 21:22:43 2019][788242.497898] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 21:27:57 2019][788556.358338] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 21:27:57 2019][788556.369999] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 21:34:32 2019][788951.518438] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 21:34:32 2019][788951.530092] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sat Jan 26 21:52:48 2019][790048.097006] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 21:52:48 2019][790048.108686] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 21:53:51 2019][790110.411525] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 21:58:13 2019][790372.313789] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 22:02:36 2019][790635.687403] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 22:10:36 2019][791115.610848] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 22:10:36 2019][791115.622502] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 22:22:09 2019][791808.944597] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 22:22:09 2019][791808.956253] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 22:34:18 2019][792537.647506] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 22:34:18 2019][792537.659169] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 22:45:13 2019][793193.132045] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 22:45:13 2019][793193.143711] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 22:56:15 2019][793855.050853] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 22:56:15 2019][793855.062506] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 11 previous similar messages [Sat Jan 26 23:07:39 2019][794538.828840] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 23:07:39 2019][794538.840487] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sat Jan 26 23:19:19 2019][795238.877559] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 23:19:19 2019][795238.889221] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sat Jan 26 23:29:51 2019][795871.060811] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 23:29:51 2019][795871.072460] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Sat Jan 26 23:44:41 2019][796760.389641] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 23:44:41 2019][796760.401309] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 15 previous similar messages [Sat Jan 26 23:58:49 2019][797609.093061] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sat Jan 26 23:58:49 2019][797609.104711] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 10 previous similar messages [Sun Jan 27 00:09:15 2019][798234.822379] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 00:09:15 2019][798234.834039] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 17 previous similar messages [Sun Jan 27 00:32:10 2019][799610.214128] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 00:32:10 2019][799610.225782] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Sun Jan 27 00:41:23 2019][800162.854643] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 00:57:31 2019][801130.966933] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 00:57:31 2019][801130.978583] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 00:59:49 2019][801269.287703] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.9.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Sun Jan 27 01:04:25 2019][801545.259150] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 01:12:41 2019][802040.552153] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 01:12:41 2019][802040.563811] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 01:22:25 2019][802624.671423] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 01:22:25 2019][802624.683086] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 01:26:05 2019][802844.897058] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 01:26:05 2019][802844.908711] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 01:34:33 2019][803353.318001] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 01:34:33 2019][803353.329651] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Sun Jan 27 01:47:37 2019][804136.549359] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 01:47:37 2019][804136.561016] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 11 previous similar messages [Sun Jan 27 02:07:40 2019][805340.039544] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 02:07:40 2019][805340.051205] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 02:22:13 2019][806213.112757] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 02:22:13 2019][806213.124410] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages [Sun Jan 27 02:39:00 2019][807220.090677] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 02:39:00 2019][807220.102325] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Sun Jan 27 02:54:04 2019][808123.935044] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 02:54:04 2019][808123.946700] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Sun Jan 27 03:07:32 2019][808932.097080] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 03:07:32 2019][808932.108734] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 03:24:12 2019][809932.369183] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 03:24:12 2019][809932.380851] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Sun Jan 27 03:36:45 2019][810685.098016] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 03:36:45 2019][810685.109674] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 12 previous similar messages [Sun Jan 27 03:47:28 2019][811328.474676] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 03:47:28 2019][811328.486352] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 03:59:07 2019][812027.609891] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 03:59:07 2019][812027.621539] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages [Sun Jan 27 04:11:17 2019][812757.618693] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 04:11:17 2019][812757.630343] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Sun Jan 27 04:23:59 2019][813519.718530] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 04:23:59 2019][813519.730192] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Sun Jan 27 04:47:14 2019][814914.425537] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 04:47:14 2019][814914.437195] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 04:48:52 2019][815012.736202] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 04:48:52 2019][815012.747851] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 04:55:24 2019][815404.509448] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 04:55:24 2019][815404.521111] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 05:05:10 2019][815990.578839] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 05:05:10 2019][815990.590511] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 05:15:19 2019][816599.295226] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 05:15:19 2019][816599.306885] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Sun Jan 27 05:25:47 2019][817227.236409] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 05:25:47 2019][817227.248063] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Sun Jan 27 05:44:10 2019][818330.035183] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 05:44:10 2019][818330.046839] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Sun Jan 27 05:55:25 2019][819005.623285] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 05:55:25 2019][819005.634976] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 06:06:40 2019][819680.483584] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 06:06:40 2019][819680.495241] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages [Sun Jan 27 06:21:15 2019][820555.103328] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 06:21:15 2019][820555.114999] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Sun Jan 27 06:26:54 2019][820894.567345] Lustre: fir-MDT0002: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Sun Jan 27 06:26:54 2019][820894.578464] Lustre: fir-MDT0002: Connection restored to ec1f9eae-3d99-6f64-392d-cf8811f0b7c2 (at 10.0.10.3@o2ib7) [Sun Jan 27 06:27:38 2019][820938.868359] Lustre: fir-MDT0002: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Sun Jan 27 06:27:39 2019][820938.878489] Lustre: fir-MDT0002: Connection restored to ec1f9eae-3d99-6f64-392d-cf8811f0b7c2 (at 10.0.10.3@o2ib7) [Sun Jan 27 07:00:23 2019][822903.431759] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 07:00:23 2019][822903.443415] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Sun Jan 27 07:21:07 2019][824147.106101] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 07:21:07 2019][824147.117757] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 08:03:17 2019][826677.929050] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 08:03:17 2019][826677.940707] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 08:21:02 2019][827742.680295] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 08:21:02 2019][827742.691951] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 08:22:20 2019][827820.520268] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 08:22:20 2019][827820.531926] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 08:22:30 2019][827830.272794] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 08:22:30 2019][827830.284493] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 08:30:43 2019][828323.945255] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 08:30:53 2019][828333.134662] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 08:39:30 2019][828850.541255] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 08:39:30 2019][828850.552909] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 08:59:00 2019][830020.227685] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 09:01:44 2019][830184.316507] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 09:01:44 2019][830184.328156] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Sun Jan 27 09:12:55 2019][830856.078349] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 09:12:55 2019][830856.090011] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 10:01:29 2019][833770.111257] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 10:02:57 2019][833857.221367] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 10:10:04 2019][834284.390184] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 10:10:33 2019][834314.168913] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 10:10:33 2019][834314.180565] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 10:10:43 2019][834323.728400] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 10:10:43 2019][834323.740071] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 10:18:14 2019][834774.730446] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 10:18:14 2019][834774.742098] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 10:23:52 2019][835113.200925] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 10:23:52 2019][835113.212605] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 10:28:58 2019][835418.834338] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 10:28:58 2019][835418.845983] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 10:30:17 2019][835497.881373] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 10:30:17 2019][835497.893028] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 10:32:28 2019][835628.979275] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 10:44:12 2019][836332.513151] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 10:44:12 2019][836332.524810] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Sun Jan 27 11:01:53 2019][837394.188433] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 11:21:10 2019][838550.908064] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 11:21:10 2019][838550.919713] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 11:44:06 2019][839927.215015] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 11:44:06 2019][839927.226668] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages [Sun Jan 27 11:58:04 2019][840764.462095] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 11:58:04 2019][840764.473752] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 12:00:31 2019][840911.625201] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 12:17:28 2019][841928.835308] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 12:17:28 2019][841928.846973] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 12:26:02 2019][842442.891844] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 12:26:54 2019][842494.833158] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 12:27:14 2019][842514.487586] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 12:27:14 2019][842514.499238] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 12:28:22 2019][842583.188837] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 12:29:50 2019][842670.645949] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 12:29:50 2019][842670.657609] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 12:31:24 2019][842765.198477] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 12:31:24 2019][842765.210133] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 12:39:16 2019][843237.203981] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 13:01:45 2019][844586.267679] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 13:01:45 2019][844586.279332] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Sun Jan 27 13:20:13 2019][845694.228701] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 13:24:36 2019][845956.537823] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 13:24:36 2019][845956.549476] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 13:27:03 2019][846103.752278] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 13:27:03 2019][846103.763939] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 13:33:48 2019][846509.306858] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 13:36:39 2019][846680.061489] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 13:36:39 2019][846680.073353] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 14:29:09 2019][849829.627438] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 14:29:09 2019][849829.639224] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 14:33:29 2019][850090.344783] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 14:33:29 2019][850090.356436] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 14:44:52 2019][850772.859440] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 17:50:36 2019][861917.634138] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 17:50:48 2019][861929.514115] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 17:51:45 2019][861986.207358] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 17:52:16 2019][862017.684049] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 17:52:16 2019][862017.695706] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 17:59:43 2019][862464.114255] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 17:59:43 2019][862464.125917] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 18:04:35 2019][862756.229468] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 18:04:35 2019][862756.241133] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 18:05:29 2019][862810.287840] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 18:05:29 2019][862810.299495] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 18:06:45 2019][862885.904281] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 18:07:50 2019][862951.186936] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 18:07:50 2019][862951.198598] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 18:31:31 2019][864372.077141] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 18:31:31 2019][864372.088799] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Sun Jan 27 19:03:31 2019][866292.041283] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 19:10:22 2019][866703.268263] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 19:10:22 2019][866703.279913] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 19:17:34 2019][867135.185342] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 20:31:26 2019][871567.045938] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Sun Jan 27 20:32:44 2019][871645.691742] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 20:32:44 2019][871645.703393] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 20:38:10 2019][871971.917472] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 20:45:13 2019][872394.145356] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 20:45:13 2019][872394.157010] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Sun Jan 27 20:45:37 2019][872418.671068] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 20:46:01 2019][872442.067494] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 20:46:01 2019][872442.079151] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 20:48:25 2019][872586.694715] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 20:48:25 2019][872586.706372] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 20:49:50 2019][872671.218910] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 20:49:50 2019][872671.230563] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 20:51:01 2019][872742.922505] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 20:51:01 2019][872742.934186] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Sun Jan 27 20:53:36 2019][872897.088094] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 20:53:36 2019][872897.099740] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Sun Jan 27 21:00:13 2019][873294.368863] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Sun Jan 27 21:00:13 2019][873294.378520] Lustre: Skipped 2 previous similar messages [Sun Jan 27 21:01:44 2019][873385.851456] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 21:01:44 2019][873385.863106] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages [Sun Jan 27 21:11:02 2019][873943.462373] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 21:11:02 2019][873943.474025] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 19 previous similar messages [Sun Jan 27 21:21:11 2019][874553.135450] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 21:21:12 2019][874553.147100] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 19 previous similar messages [Sun Jan 27 21:29:36 2019][875057.811093] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Sun Jan 27 21:29:36 2019][875057.820750] Lustre: Skipped 2 previous similar messages [Sun Jan 27 21:29:37 2019][875058.529695] Lustre: fir-MDT0000: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Sun Jan 27 21:29:37 2019][875058.540048] Lustre: Skipped 1 previous similar message [Sun Jan 27 21:31:17 2019][875158.179033] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 21:31:17 2019][875158.190719] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 30 previous similar messages [Sun Jan 27 21:42:10 2019][875811.948207] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 21:42:10 2019][875811.959861] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 38 previous similar messages [Sun Jan 27 21:53:02 2019][876463.828953] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 21:53:02 2019][876463.840605] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages [Sun Jan 27 21:58:49 2019][876810.599885] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Sun Jan 27 22:05:48 2019][877229.592364] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 22:05:48 2019][877229.604011] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Sun Jan 27 22:21:53 2019][878194.579147] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 22:21:53 2019][878194.590864] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Sun Jan 27 22:29:21 2019][878643.070855] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Sun Jan 27 22:29:21 2019][878643.080520] Lustre: Skipped 2 previous similar messages [Sun Jan 27 22:33:09 2019][878870.519728] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 22:33:09 2019][878870.531378] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages [Sun Jan 27 22:44:02 2019][879523.706900] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Sun Jan 27 22:44:02 2019][879523.716565] Lustre: Skipped 1 previous similar message [Sun Jan 27 22:44:05 2019][879526.259972] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 22:44:05 2019][879526.271620] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages [Sun Jan 27 22:52:17 2019][880018.660689] Lustre: fir-MDT0000: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Sun Jan 27 22:52:17 2019][880018.660714] Lustre: fir-MDT0002: Connection restored to ec1f9eae-3d99-6f64-392d-cf8811f0b7c2 (at 10.0.10.3@o2ib7) [Sun Jan 27 22:52:17 2019][880018.660716] Lustre: Skipped 2 previous similar messages [Sun Jan 27 22:52:17 2019][880018.686483] Lustre: Skipped 1 previous similar message [Sun Jan 27 22:54:35 2019][880156.586796] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 22:54:35 2019][880156.598446] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 21 previous similar messages [Sun Jan 27 23:04:35 2019][880756.718385] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Sun Jan 27 23:04:35 2019][880756.730040] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 24 previous similar messages [Sun Jan 27 23:12:59 2019][881260.529264] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Sun Jan 27 23:12:59 2019][881260.538925] Lustre: Skipped 1 previous similar message [Sun Jan 27 23:16:18 2019][881459.331725] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Sun Jan 27 23:16:18 2019][881459.343463] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 17 previous similar messages [Sun Jan 27 23:41:26 2019][882967.745253] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Sun Jan 27 23:41:26 2019][882967.754910] Lustre: Skipped 2 previous similar messages [Sun Jan 27 23:55:54 2019][883835.550005] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Sun Jan 27 23:55:54 2019][883835.559663] Lustre: Skipped 2 previous similar messages [Mon Jan 28 00:20:01 2019][885282.403083] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Mon Jan 28 00:20:01 2019][885282.414735] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Mon Jan 28 00:22:04 2019][885406.011941] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 00:22:04 2019][885406.023589] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Mon Jan 28 00:24:04 2019][885526.014569] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 00:24:04 2019][885526.024233] Lustre: Skipped 2 previous similar messages [Mon Jan 28 00:29:17 2019][885838.424739] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Mon Jan 28 00:29:17 2019][885838.436392] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Mon Jan 28 00:35:43 2019][886225.156538] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Mon Jan 28 00:35:43 2019][886225.168192] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Mon Jan 28 00:45:56 2019][886837.803328] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Mon Jan 28 00:45:56 2019][886837.815000] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Mon Jan 28 00:52:03 2019][887205.123325] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 00:52:03 2019][887205.132987] Lustre: Skipped 2 previous similar messages [Mon Jan 28 00:58:30 2019][887592.208454] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 00:58:30 2019][887592.220100] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 25 previous similar messages [Mon Jan 28 01:06:27 2019][888069.086103] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 01:06:27 2019][888069.095788] Lustre: Skipped 2 previous similar messages [Mon Jan 28 01:10:22 2019][888304.157445] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 01:10:22 2019][888304.169091] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 15 previous similar messages [Mon Jan 28 01:24:22 2019][889143.956563] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 01:24:22 2019][889143.968217] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages [Mon Jan 28 01:33:52 2019][889713.787821] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 01:33:52 2019][889713.797473] Lustre: Skipped 2 previous similar messages [Mon Jan 28 01:35:20 2019][889801.595874] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 01:35:20 2019][889801.607529] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Mon Jan 28 01:43:54 2019][890316.385260] LNetError: 5876:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Mon Jan 28 01:48:04 2019][890565.897595] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 01:48:04 2019][890565.907253] Lustre: Skipped 2 previous similar messages [Mon Jan 28 01:48:53 2019][890615.255960] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 01:48:53 2019][890615.267628] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages [Mon Jan 28 02:01:38 2019][891379.912600] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Mon Jan 28 02:12:22 2019][892024.319788] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 02:12:22 2019][892024.331436] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Mon Jan 28 02:15:31 2019][892213.177374] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 02:15:31 2019][892213.187037] Lustre: Skipped 2 previous similar messages [Mon Jan 28 02:15:51 2019][892233.145694] LNetError: 5876:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Mon Jan 28 02:24:55 2019][892777.496831] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 02:24:55 2019][892777.508504] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Mon Jan 28 02:43:16 2019][893878.397758] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 02:43:16 2019][893878.407421] Lustre: Skipped 2 previous similar messages [Mon Jan 28 02:46:37 2019][894078.806310] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 02:56:58 2019][894699.838158] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 02:56:58 2019][894699.849815] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Mon Jan 28 03:01:25 2019][894967.271434] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 03:01:25 2019][894967.283086] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Mon Jan 28 03:06:49 2019][895290.723644] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 03:06:49 2019][895290.735290] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Mon Jan 28 03:11:37 2019][895579.011848] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 03:11:37 2019][895579.021504] Lustre: Skipped 2 previous similar messages [Mon Jan 28 03:16:57 2019][895899.221357] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 03:16:57 2019][895899.233009] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Mon Jan 28 03:28:35 2019][896596.727439] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 03:28:35 2019][896596.739102] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages [Mon Jan 28 03:40:12 2019][897294.018966] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 03:40:12 2019][897294.028621] Lustre: Skipped 2 previous similar messages [Mon Jan 28 03:41:37 2019][897379.238262] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 03:41:37 2019][897379.249913] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Mon Jan 28 03:53:11 2019][898072.883664] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Mon Jan 28 04:03:43 2019][898704.917947] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 04:03:43 2019][898704.929596] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Mon Jan 28 04:08:49 2019][899010.702599] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 04:08:49 2019][899010.712253] Lustre: Skipped 2 previous similar messages [Mon Jan 28 04:20:12 2019][899694.488802] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 04:33:25 2019][900486.941479] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Mon Jan 28 04:33:25 2019][900486.953136] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Mon Jan 28 04:37:05 2019][900707.308651] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 04:37:05 2019][900707.318310] Lustre: Skipped 2 previous similar messages [Mon Jan 28 04:43:06 2019][901068.655664] LNetError: 5877:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Mon Jan 28 04:44:17 2019][901139.653231] Lustre: fir-MDT0000: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:44:17 2019][901139.663375] Lustre: fir-MDT0000: Connection restored to ec1f9eae-3d99-6f64-392d-cf8811f0b7c2 (at 10.0.10.3@o2ib7) [Mon Jan 28 04:44:17 2019][901139.673783] Lustre: Skipped 2 previous similar messages [Mon Jan 28 04:44:57 2019][901178.740429] LNetError: 5876:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Mon Jan 28 04:45:03 2019][901185.336061] Lustre: fir-MDT0002: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:45:03 2019][901185.336070] Lustre: MGS: Received new LWP connection from 10.0.10.3@o2ib7, removing former export from same NID [Mon Jan 28 04:45:03 2019][901185.336073] Lustre: Skipped 3 previous similar messages [Mon Jan 28 04:45:03 2019][901185.336084] Lustre: fir-MDT0000: Connection restored to ec1f9eae-3d99-6f64-392d-cf8811f0b7c2 (at 10.0.10.3@o2ib7) [Mon Jan 28 04:45:03 2019][901185.371988] Lustre: Skipped 1 previous similar message [Mon Jan 28 04:45:38 2019][901220.386960] Lustre: MGS: Received new LWP connection from 10.0.10.3@o2ib7, removing former export from same NID [Mon Jan 28 04:45:38 2019][901220.386970] Lustre: fir-MDT0002: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:45:38 2019][901220.386995] Lustre: fir-MDT0002: Connection restored to ec1f9eae-3d99-6f64-392d-cf8811f0b7c2 (at 10.0.10.3@o2ib7) [Mon Jan 28 04:45:38 2019][901220.386996] Lustre: Skipped 2 previous similar messages [Mon Jan 28 04:46:25 2019][901267.654263] LNet: 5877:0:(o2iblnd_cb.c:408:kiblnd_handle_rx()) PUT_NACK from 10.0.10.3@o2ib7 [Mon Jan 28 04:46:44 2019][901286.244245] Lustre: fir-MDT0002: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:46:44 2019][901286.244258] Lustre: MGS: Received new LWP connection from 10.0.10.3@o2ib7, removing former export from same NID [Mon Jan 28 04:46:44 2019][901286.244281] Lustre: MGS: Connection restored to ec1f9eae-3d99-6f64-392d-cf8811f0b7c2 (at 10.0.10.3@o2ib7) [Mon Jan 28 04:46:44 2019][901286.244283] Lustre: Skipped 1 previous similar message [Mon Jan 28 04:46:44 2019][901286.244337] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.0.10.3@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Mon Jan 28 04:46:44 2019][901286.296697] Lustre: Skipped 1 previous similar message [Mon Jan 28 04:47:09 2019][901311.461015] Lustre: MGS: Received new LWP connection from 10.0.10.3@o2ib7, removing former export from same NID [Mon Jan 28 04:47:09 2019][901311.461022] Lustre: fir-MDT0000: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:47:09 2019][901311.461024] Lustre: Skipped 1 previous similar message [Mon Jan 28 04:47:09 2019][901311.461045] Lustre: fir-MDT0000: Connection restored to ec1f9eae-3d99-6f64-392d-cf8811f0b7c2 (at 10.0.10.3@o2ib7) [Mon Jan 28 04:47:09 2019][901311.461046] Lustre: Skipped 2 previous similar messages [Mon Jan 28 04:47:46 2019][901348.277728] LNetError: 5875:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Mon Jan 28 04:48:00 2019][901362.518510] Lustre: MGS: Received new LWP connection from 10.0.10.3@o2ib7, removing former export from same NID [Mon Jan 28 04:48:00 2019][901362.518533] Lustre: fir-MDT0002: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:48:00 2019][901362.518562] Lustre: fir-MDT0002: Connection restored to ec1f9eae-3d99-6f64-392d-cf8811f0b7c2 (at 10.0.10.3@o2ib7) [Mon Jan 28 04:48:00 2019][901362.518564] Lustre: Skipped 2 previous similar messages [Mon Jan 28 04:48:25 2019][901387.686920] Lustre: fir-MDT0002: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:48:25 2019][901387.686963] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.0.10.3@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Mon Jan 28 04:49:45 2019][901466.773047] LNetError: 5877:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Mon Jan 28 04:50:04 2019][901486.712101] Lustre: MGS: Received new LWP connection from 10.0.10.3@o2ib7, removing former export from same NID [Mon Jan 28 04:50:04 2019][901486.712142] Lustre: fir-MDT0002: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:50:04 2019][901486.712172] Lustre: fir-MDT0002: Connection restored to ec1f9eae-3d99-6f64-392d-cf8811f0b7c2 (at 10.0.10.3@o2ib7) [Mon Jan 28 04:50:05 2019][901486.712175] Lustre: Skipped 3 previous similar messages [Mon Jan 28 04:50:05 2019][901486.748015] Lustre: Skipped 1 previous similar message [Mon Jan 28 04:51:03 2019][901545.138092] LNetError: 5877:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Mon Jan 28 04:51:18 2019][901560.066709] LNetError: 5875:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Mon Jan 28 04:51:36 2019][901578.369698] LNet: 5877:0:(o2iblnd_cb.c:408:kiblnd_handle_rx()) PUT_NACK from 10.0.10.3@o2ib7 [Mon Jan 28 04:51:38 2019][901580.640812] LNet: 5877:0:(o2iblnd_cb.c:408:kiblnd_handle_rx()) PUT_NACK from 10.0.10.3@o2ib7 [Mon Jan 28 04:51:40 2019][901582.094211] Lustre: fir-MDT0000: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:51:40 2019][901582.094212] Lustre: fir-MDT0002: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 04:51:40 2019][901582.094215] Lustre: Skipped 3 previous similar messages [Mon Jan 28 04:51:40 2019][901582.094219] Lustre: MGS: Received new LWP connection from 10.0.10.3@o2ib7, removing former export from same NID [Mon Jan 28 04:51:40 2019][901582.094221] Lustre: Skipped 1 previous similar message [Mon Jan 28 04:51:40 2019][901582.094240] Lustre: fir-MDT0002: Connection restored to ec1f9eae-3d99-6f64-392d-cf8811f0b7c2 (at 10.0.10.3@o2ib7) [Mon Jan 28 04:51:40 2019][901582.094241] Lustre: Skipped 5 previous similar messages [Mon Jan 28 04:52:31 2019][901633.071605] LNetError: 5877:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Mon Jan 28 04:55:22 2019][901804.298487] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 04:55:22 2019][901804.310150] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages [Mon Jan 28 04:56:57 2019][901899.045984] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 04:56:57 2019][901899.057656] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Mon Jan 28 04:59:46 2019][902068.045810] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 04:59:46 2019][902068.057487] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Mon Jan 28 05:05:37 2019][902419.393589] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 05:05:37 2019][902419.403244] Lustre: Skipped 10 previous similar messages [Mon Jan 28 05:33:40 2019][904102.461174] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 05:33:40 2019][904102.470827] Lustre: Skipped 2 previous similar messages [Mon Jan 28 05:41:27 2019][904568.899941] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 05:41:27 2019][904568.911609] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages [Mon Jan 28 05:53:57 2019][905318.948858] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 05:53:57 2019][905318.960519] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Mon Jan 28 05:54:05 2019][905327.011119] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 05:54:05 2019][905327.022771] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Mon Jan 28 05:54:42 2019][905363.973644] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 05:54:42 2019][905363.985381] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Mon Jan 28 05:57:48 2019][905550.563267] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 06:00:57 2019][905738.877080] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 06:00:57 2019][905738.888739] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Mon Jan 28 06:01:22 2019][905764.039686] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 06:01:22 2019][905764.049346] Lustre: Skipped 2 previous similar messages [Mon Jan 28 06:02:38 2019][905840.282014] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 06:02:38 2019][905840.293661] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Mon Jan 28 06:05:30 2019][906012.455371] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 06:05:30 2019][906012.467046] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Mon Jan 28 06:11:58 2019][906400.513644] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 06:11:58 2019][906400.525318] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages [Mon Jan 28 06:23:46 2019][907108.294208] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 06:23:46 2019][907108.305864] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages [Mon Jan 28 06:30:06 2019][907487.900457] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 06:30:06 2019][907487.910118] Lustre: Skipped 2 previous similar messages [Mon Jan 28 06:34:23 2019][907745.773496] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 06:34:23 2019][907745.785153] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Mon Jan 28 06:44:55 2019][908377.349316] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 06:44:55 2019][908377.358968] Lustre: Skipped 2 previous similar messages [Mon Jan 28 06:49:01 2019][908623.809734] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 06:49:01 2019][908623.821393] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 10 previous similar messages [Mon Jan 28 06:59:28 2019][909250.502967] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 06:59:28 2019][909250.512626] Lustre: Skipped 2 previous similar messages [Mon Jan 28 06:59:59 2019][909281.814507] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 06:59:59 2019][909281.826152] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Mon Jan 28 07:12:46 2019][910048.557930] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 07:12:46 2019][910048.569585] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages [Mon Jan 28 07:24:52 2019][910774.003111] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 07:24:52 2019][910774.014759] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 11 previous similar messages [Mon Jan 28 07:27:24 2019][910926.901979] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 07:27:24 2019][910926.911631] Lustre: Skipped 2 previous similar messages [Mon Jan 28 07:35:09 2019][911391.055816] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 07:35:09 2019][911391.067465] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages [Mon Jan 28 07:41:40 2019][911782.387760] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 07:41:40 2019][911782.397419] Lustre: Skipped 2 previous similar messages [Mon Jan 28 07:45:41 2019][912023.018722] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 07:45:41 2019][912023.030386] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Mon Jan 28 07:56:32 2019][912674.192256] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 07:56:32 2019][912674.203903] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Mon Jan 28 08:10:00 2019][913482.890705] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 08:10:00 2019][913482.900362] Lustre: Skipped 2 previous similar messages [Mon Jan 28 08:22:14 2019][914216.855683] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 08:22:14 2019][914216.867340] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages [Mon Jan 28 08:27:01 2019][914503.533065] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 08:27:01 2019][914503.544718] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Mon Jan 28 08:32:19 2019][914821.566388] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 08:32:19 2019][914821.578044] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Mon Jan 28 08:38:26 2019][915188.832054] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 08:38:26 2019][915188.841717] Lustre: Skipped 2 previous similar messages [Mon Jan 28 08:40:09 2019][915291.624197] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Mon Jan 28 08:40:09 2019][915291.635851] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Mon Jan 28 08:51:22 2019][915964.957178] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 08:51:22 2019][915964.968839] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Mon Jan 28 09:05:31 2019][916813.394942] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 09:05:31 2019][916813.406595] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 12 previous similar messages [Mon Jan 28 09:05:57 2019][916839.174249] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 09:05:57 2019][916839.183907] Lustre: Skipped 2 previous similar messages [Mon Jan 28 09:17:56 2019][917558.352865] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 09:17:56 2019][917558.364516] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14 previous similar messages [Mon Jan 28 09:33:58 2019][918520.430755] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 09:33:58 2019][918520.440410] Lustre: Skipped 2 previous similar messages [Mon Jan 28 09:43:46 2019][919108.381773] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 09:43:46 2019][919108.393427] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14 previous similar messages [Mon Jan 28 09:46:04 2019][919246.915582] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 09:46:04 2019][919246.927278] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Mon Jan 28 09:47:38 2019][919340.951992] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 09:47:38 2019][919340.961653] Lustre: Skipped 2 previous similar messages [Mon Jan 28 09:51:29 2019][919571.807068] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 09:51:29 2019][919571.818716] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Mon Jan 28 09:59:16 2019][920038.677151] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Mon Jan 28 09:59:16 2019][920038.688807] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Mon Jan 28 10:15:00 2019][920982.203658] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 10:15:00 2019][920982.213322] Lustre: Skipped 2 previous similar messages [Mon Jan 28 10:17:00 2019][921103.148382] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 10:17:00 2019][921103.160033] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Mon Jan 28 10:33:42 2019][922105.072706] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 10:43:58 2019][922720.742320] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 10:43:58 2019][922720.752001] Lustre: Skipped 2 previous similar messages [Mon Jan 28 10:48:42 2019][923004.521443] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 10:48:42 2019][923004.533098] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Mon Jan 28 10:59:16 2019][923638.846513] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 11:12:41 2019][924443.578544] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 11:16:01 2019][924643.773332] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 11:16:01 2019][924643.782994] Lustre: Skipped 2 previous similar messages [Mon Jan 28 11:23:57 2019][925119.587660] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Mon Jan 28 11:23:57 2019][925119.599306] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Mon Jan 28 11:34:58 2019][925781.009915] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 11:34:58 2019][925781.021567] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 51 previous similar messages [Mon Jan 28 11:45:11 2019][926394.064452] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Mon Jan 28 11:45:11 2019][926394.076101] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 15 previous similar messages [Mon Jan 28 11:46:32 2019][926475.041676] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 11:46:32 2019][926475.051327] Lustre: Skipped 2 previous similar messages [Mon Jan 28 11:55:27 2019][927009.677821] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 11:55:27 2019][927009.689473] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 31 previous similar messages [Mon Jan 28 12:07:54 2019][927756.489958] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 12:07:54 2019][927756.501632] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14 previous similar messages [Mon Jan 28 12:15:04 2019][928186.680133] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 12:15:04 2019][928186.689792] Lustre: Skipped 2 previous similar messages [Mon Jan 28 12:21:32 2019][928575.070394] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Mon Jan 28 12:21:32 2019][928575.082042] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Mon Jan 28 12:33:30 2019][929293.319124] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 12:33:30 2019][929293.330778] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 28 previous similar messages [Mon Jan 28 12:43:21 2019][929884.050402] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 12:43:21 2019][929884.060066] Lustre: Skipped 2 previous similar messages [Mon Jan 28 12:43:42 2019][929904.671981] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 12:43:42 2019][929904.683649] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 23 previous similar messages [Mon Jan 28 12:53:54 2019][930516.965966] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 12:53:54 2019][930516.977624] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 16 previous similar messages [Mon Jan 28 13:05:43 2019][931226.084847] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 13:05:43 2019][931226.096521] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 19 previous similar messages [Mon Jan 28 13:11:55 2019][931598.126567] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 13:11:55 2019][931598.136221] Lustre: Skipped 2 previous similar messages [Mon Jan 28 13:15:55 2019][931838.093870] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 13:15:55 2019][931838.105523] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Mon Jan 28 13:35:51 2019][933033.909427] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 13:35:51 2019][933033.921080] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Mon Jan 28 13:40:19 2019][933302.357773] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 13:40:19 2019][933302.367426] Lustre: Skipped 2 previous similar messages [Mon Jan 28 13:51:48 2019][933990.518885] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 13:51:48 2019][933990.530538] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Mon Jan 28 14:09:01 2019][935024.168498] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 14:09:01 2019][935024.180152] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Mon Jan 28 14:09:52 2019][935074.832239] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 14:09:52 2019][935074.841925] Lustre: Skipped 2 previous similar messages [Mon Jan 28 14:22:00 2019][935802.935375] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 14:22:00 2019][935802.947027] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages [Mon Jan 28 14:24:39 2019][935962.002758] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 14:24:39 2019][935962.012411] Lustre: Skipped 2 previous similar messages [Mon Jan 28 14:32:34 2019][936436.565991] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Mon Jan 28 14:32:34 2019][936436.577660] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Mon Jan 28 14:51:24 2019][937566.834787] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Mon Jan 28 14:51:24 2019][937566.846438] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Mon Jan 28 14:53:51 2019][937713.857348] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Mon Jan 28 14:53:51 2019][937713.867003] Lustre: Skipped 2 previous similar messages [Mon Jan 28 15:02:43 2019][938246.225716] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [Mon Jan 28 15:02:43 2019][938246.235715] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.3@o2ib7 (5): c: 0, oc: 0, rc: 8 [Mon Jan 28 15:02:45 2019][938248.225767] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.3@o2ib7: 2 seconds [Mon Jan 28 15:02:46 2019][938249.225792] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.3@o2ib7: 3 seconds [Mon Jan 28 15:02:46 2019][938249.235792] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 7 previous similar messages [Mon Jan 28 15:02:47 2019][938250.226816] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.3@o2ib7: 4 seconds [Mon Jan 28 15:02:47 2019][938250.236815] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 6 previous similar messages [Mon Jan 28 15:03:46 2019][938309.465526] LNetError: 5878:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Mon Jan 28 15:03:46 2019][938309.477825] LNetError: 5878:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message [Mon Jan 28 15:04:00 2019][938323.406657] Lustre: 8948:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1548716673/real 0] req@ffff913614026900 x1623674265378432/t0(0) o104->fir-MDT0000@10.0.10.3@o2ib7:15/16 lens 296/224 e 0 to 1 dl 1548716680 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Mon Jan 28 15:04:00 2019][938323.433045] Lustre: 8948:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 34 previous similar messages [Mon Jan 28 15:04:52 2019][938374.738943] Lustre: 8948:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1548716724/real 1548716724] req@ffff913614026900 x1623674265378432/t0(0) o104->fir-MDT0000@10.0.10.3@o2ib7:15/16 lens 296/224 e 0 to 1 dl 1548716731 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 [Mon Jan 28 15:05:08 2019][938391.315638] Lustre: fir-MDT0000: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 15:05:08 2019][938391.315709] Lustre: MGS: Received new LWP connection from 10.0.10.3@o2ib7, removing former export from same NID [Mon Jan 28 15:05:08 2019][938391.315711] Lustre: Skipped 4 previous similar messages [Mon Jan 28 15:05:08 2019][938391.315721] Lustre: fir-MDT0002: Connection restored to ec1f9eae-3d99-6f64-392d-cf8811f0b7c2 (at 10.0.10.3@o2ib7) [Mon Jan 28 15:05:08 2019][938391.315722] Lustre: Skipped 2 previous similar messages [Mon Jan 28 15:05:08 2019][938391.315796] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.0.10.3@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Mon Jan 28 15:05:08 2019][938391.315798] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.0.10.3@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Mon Jan 28 15:05:08 2019][938391.315798] LustreError: Skipped 1 previous similar message [Mon Jan 28 15:05:08 2019][938391.315799] LustreError: Skipped 1 previous similar message [Mon Jan 28 15:05:08 2019][938391.402802] Lustre: Skipped 6 previous similar messages [Mon Jan 28 15:05:44 2019][938427.504097] Lustre: fir-MDT0000: Client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) reconnecting [Mon Jan 28 15:05:44 2019][938427.504124] Lustre: fir-MDT0002: Connection restored to ec1f9eae-3d99-6f64-392d-cf8811f0b7c2 (at 10.0.10.3@o2ib7) [Mon Jan 28 15:05:44 2019][938427.504126] Lustre: Skipped 2 previous similar messages [Mon Jan 28 15:05:44 2019][938427.529841] Lustre: Skipped 1 previous similar message [Mon Jan 28 15:06:32 2019][938475.231461] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [Mon Jan 28 15:06:32 2019][938475.241467] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.3@o2ib7 (5): c: 0, oc: 1, rc: 8 [Mon Jan 28 15:06:33 2019][938476.231491] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.3@o2ib7: 1 seconds [Mon Jan 28 15:06:33 2019][938476.241489] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 10 previous similar messages [Mon Jan 28 15:06:38 2019][938481.231615] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.3@o2ib7: 0 seconds [Mon Jan 28 15:06:38 2019][938481.241618] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 21 previous similar messages [Mon Jan 28 15:06:47 2019][938490.231844] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.3@o2ib7: 3 seconds [Mon Jan 28 15:06:47 2019][938490.241841] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 49 previous similar messages [Mon Jan 28 15:07:05 2019][938508.232293] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.3@o2ib7: 10 seconds [Mon Jan 28 15:07:05 2019][938508.242377] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 94 previous similar messages [Mon Jan 28 15:07:39 2019][938542.233143] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.3@o2ib7: 16 seconds [Mon Jan 28 15:07:39 2019][938542.243225] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 201 previous similar messages [Mon Jan 28 15:08:44 2019][938607.234777] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.3@o2ib7: 20 seconds [Mon Jan 28 15:08:44 2019][938607.244865] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 401 previous similar messages [Mon Jan 28 15:08:56 2019][938618.783880] Lustre: MGS: haven't heard from client ec1f9eae-3d99-6f64-392d-cf8811f0b7c2 (at 10.0.10.3@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9129d9dbd800, cur 1548716975 expire 1548716825 last 1548716748 [Mon Jan 28 15:09:31 2019][938654.052989] Lustre: fir-MDT0000: haven't heard from client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9129bffb4c00, cur 1548717011 expire 1548716861 last 1548716784 [Mon Jan 28 15:10:12 2019][938695.057669] Lustre: fir-MDT0002: haven't heard from client c680654c-e7b5-e476-0d23-f02ec53e73ba (at 10.0.10.3@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9129cfca1c00, cur 1548717052 expire 1548716902 last 1548716825 [Mon Jan 28 15:10:53 2019][938736.238016] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.3@o2ib7: 19 seconds [Mon Jan 28 15:10:53 2019][938736.248098] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 761 previous similar messages [Mon Jan 28 15:15:11 2019][938994.244494] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.3@o2ib7: 6 seconds [Mon Jan 28 15:15:11 2019][938994.254488] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 1519 previous similar messages [Mon Jan 28 15:23:43 2019][939506.258357] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.3@o2ib7: 5 seconds [Mon Jan 28 15:23:43 2019][939506.268354] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3017 previous similar messages [Mon Jan 28 15:33:44 2019][940107.272438] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.3@o2ib7: 0 seconds [Mon Jan 28 15:33:44 2019][940107.282434] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3623 previous similar messages [Mon Jan 28 15:34:35 2019][940157.805254] Lustre: MGS: Connection restored to ec1f9eae-3d99-6f64-392d-cf8811f0b7c2 (at 10.0.10.3@o2ib7) [Mon Jan 28 15:34:35 2019][940157.814907] Lustre: Skipped 1 previous similar message [Mon Jan 28 18:40:36 2019][951319.817320] Lustre: MGS: Connection restored to 2ef59ace-0507-3d28-3cc5-ca9a29d2437b (at 10.8.2.9@o2ib6) [Mon Jan 28 18:40:36 2019][951319.826888] Lustre: Skipped 2 previous similar messages [Mon Jan 28 18:42:42 2019][951445.394757] Lustre: MGS: Connection restored to 2705c53f-bfde-167d-1dab-1ff569ebcbb3 (at 10.8.6.31@o2ib6) [Mon Jan 28 18:42:42 2019][951445.404415] Lustre: Skipped 2 previous similar messages [Mon Jan 28 18:42:43 2019][951446.723000] Lustre: fir-MDT0002: Connection restored to 2705c53f-bfde-167d-1dab-1ff569ebcbb3 (at 10.8.6.31@o2ib6) [Mon Jan 28 18:42:43 2019][951446.733350] Lustre: Skipped 1 previous similar message [Mon Jan 28 21:25:37 2019][961220.624726] Lustre: fir-MDT0000: haven't heard from client 9d0731b3-b468-e234-aad6-42e89a84a6d4 (at 10.9.101.60@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9129bffb4000, cur 1548739577 expire 1548739427 last 1548739350 [Mon Jan 28 21:25:39 2019][961222.688752] Lustre: MGS: haven't heard from client 74841472-5fb7-9c47-503a-c07169a69783 (at 10.9.101.60@o2ib4) in 229 seconds. I think it's dead, and I am evicting it. exp ffff9129d9dba400, cur 1548739579 expire 1548739429 last 1548739350 [Mon Jan 28 21:25:39 2019][961222.709938] Lustre: Skipped 1 previous similar message [Mon Jan 28 22:37:13 2019][965516.289049] Lustre: MGS: Connection restored to 74841472-5fb7-9c47-503a-c07169a69783 (at 10.9.101.60@o2ib4) [Mon Jan 28 22:37:14 2019][965517.741255] Lustre: fir-MDT0002: Connection restored to 74841472-5fb7-9c47-503a-c07169a69783 (at 10.9.101.60@o2ib4) [Mon Jan 28 22:37:14 2019][965517.751785] Lustre: Skipped 1 previous similar message [Tue Jan 29 11:22:56 2019][1011460.424640] Lustre: fir-MDT0000: Client 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) reconnecting [Tue Jan 29 11:22:56 2019][1011460.428620] Lustre: MGS: Received new LWP connection from 10.9.0.1@o2ib4, removing former export from same NID [Tue Jan 29 11:22:56 2019][1011460.428642] Lustre: MGS: Connection restored to 6caeea3e-5b74-9a8d-d8ac-3bf9f06c410e (at 10.9.0.1@o2ib4) [Tue Jan 29 11:22:56 2019][1011460.454571] Lustre: Skipped 1 previous similar message [Tue Jan 29 11:23:18 2019][1011482.389562] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.0.1@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Tue Jan 29 11:23:18 2019][1011482.406842] LustreError: Skipped 1 previous similar message [Tue Jan 29 11:23:43 2019][1011507.586782] Lustre: MGS: Received new LWP connection from 10.9.0.1@o2ib4, removing former export from same NID [Tue Jan 29 11:23:43 2019][1011507.586808] Lustre: fir-MDT0000: Client 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) reconnecting [Tue Jan 29 11:23:43 2019][1011507.586829] Lustre: fir-MDT0000: Connection restored to 6caeea3e-5b74-9a8d-d8ac-3bf9f06c410e (at 10.9.0.1@o2ib4) [Tue Jan 29 11:23:43 2019][1011507.586830] Lustre: Skipped 2 previous similar messages [Tue Jan 29 11:24:36 2019][1011560.839804] Lustre: fir-MDT0000: Client 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) reconnecting [Tue Jan 29 11:24:36 2019][1011560.849896] Lustre: Skipped 1 previous similar message [Tue Jan 29 11:24:36 2019][1011560.855245] Lustre: fir-MDT0000: Connection restored to 6caeea3e-5b74-9a8d-d8ac-3bf9f06c410e (at 10.9.0.1@o2ib4) [Tue Jan 29 11:24:36 2019][1011560.865606] Lustre: Skipped 2 previous similar messages [Tue Jan 29 11:25:13 2019][1011598.056397] Lustre: fir-MDT0000: Client 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) reconnecting [Tue Jan 29 11:25:13 2019][1011598.066531] Lustre: fir-MDT0000: Connection restored to 6caeea3e-5b74-9a8d-d8ac-3bf9f06c410e (at 10.9.0.1@o2ib4) [Tue Jan 29 11:49:42 2019][1013066.450716] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 11:49:42 2019][1013066.462449] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Tue Jan 29 11:57:29 2019][1013533.819609] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 12:03:02 2019][1013866.318269] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 12:03:02 2019][1013866.330011] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Tue Jan 29 12:09:41 2019][1014265.377049] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Jan 29 12:09:41 2019][1014265.388789] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Tue Jan 29 15:43:50 2019][1027115.016241] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Tue Jan 29 16:47:54 2019][1030959.066727] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 16:47:54 2019][1030959.078469] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 12 previous similar messages [Tue Jan 29 16:52:45 2019][1031250.339548] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Jan 29 16:52:45 2019][1031250.351293] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Jan 29 17:08:49 2019][1032214.385564] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 17:08:49 2019][1032214.397320] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Tue Jan 29 17:09:12 2019][1032236.865796] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 17:09:12 2019][1032236.877539] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Jan 29 17:10:37 2019][1032321.886093] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 17:10:37 2019][1032321.897830] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Tue Jan 29 17:18:16 2019][1032780.823809] Lustre: MGS: Connection restored to c3ea57d5-46ff-5c6e-1800-776bdd796fdc (at 10.8.20.15@o2ib6) [Tue Jan 29 17:18:16 2019][1032780.833648] Lustre: Skipped 2 previous similar messages [Tue Jan 29 17:20:05 2019][1032890.380407] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 17:23:32 2019][1033097.571569] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 17:23:32 2019][1033097.583333] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Jan 29 17:34:01 2019][1033726.629651] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 17:34:01 2019][1033726.641396] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Tue Jan 29 17:38:34 2019][1033999.445668] Lustre: fir-MDT0000: haven't heard from client 8ae4cc68-204d-d2bb-4908-94f49692744b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91369829a800, cur 1548812354 expire 1548812204 last 1548812127 [Tue Jan 29 17:38:37 2019][1034001.903956] Lustre: MGS: haven't heard from client c3ea57d5-46ff-5c6e-1800-776bdd796fdc (at 10.8.20.15@o2ib6) in 229 seconds. I think it's dead, and I am evicting it. exp ffff914096ffac00, cur 1548812356 expire 1548812206 last 1548812127 [Tue Jan 29 17:38:37 2019][1034001.925134] Lustre: Skipped 1 previous similar message [Tue Jan 29 17:45:23 2019][1034408.195540] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 17:45:23 2019][1034408.207287] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Tue Jan 29 17:55:52 2019][1035037.030814] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 18:09:47 2019][1035872.247309] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 18:09:47 2019][1035872.259042] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Tue Jan 29 18:22:33 2019][1036638.555543] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 18:22:33 2019][1036638.567287] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Tue Jan 29 18:34:53 2019][1037378.508029] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 18:34:53 2019][1037378.519768] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 20 previous similar messages [Tue Jan 29 18:48:23 2019][1038188.626764] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 18:48:23 2019][1038188.638504] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 17 previous similar messages [Tue Jan 29 18:59:14 2019][1038839.462494] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 18:59:14 2019][1038839.474235] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages [Tue Jan 29 19:10:16 2019][1039501.223997] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 19:10:16 2019][1039501.235739] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 10 previous similar messages [Tue Jan 29 19:21:29 2019][1040174.706936] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 19:21:29 2019][1040174.718678] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 19 previous similar messages [Tue Jan 29 19:31:44 2019][1040789.689496] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 19:31:44 2019][1040789.701249] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages [Tue Jan 29 19:43:07 2019][1041472.824694] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Jan 29 19:43:07 2019][1041472.836453] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 11 previous similar messages [Tue Jan 29 19:53:11 2019][1042076.427398] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 19:53:11 2019][1042076.439136] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 30 previous similar messages [Tue Jan 29 20:03:37 2019][1042702.119478] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Jan 29 20:03:37 2019][1042702.131304] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Tue Jan 29 20:15:04 2019][1043389.878363] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 20:15:04 2019][1043389.890108] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages [Tue Jan 29 20:28:43 2019][1044208.573802] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 20:28:43 2019][1044208.585538] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 24 previous similar messages [Tue Jan 29 20:39:56 2019][1044881.879146] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 20:39:56 2019][1044881.890900] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 15 previous similar messages [Tue Jan 29 20:51:50 2019][1045595.840408] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Jan 29 20:51:50 2019][1045595.852149] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 24 previous similar messages [Tue Jan 29 21:04:42 2019][1046367.692628] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Jan 29 21:04:42 2019][1046367.704370] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14 previous similar messages [Tue Jan 29 21:16:58 2019][1047103.045407] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Jan 29 21:16:58 2019][1047103.057144] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Tue Jan 29 21:27:44 2019][1047750.003036] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 21:27:44 2019][1047750.014779] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Tue Jan 29 21:40:00 2019][1048485.384320] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Jan 29 21:40:00 2019][1048485.396066] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 17 previous similar messages [Tue Jan 29 21:54:52 2019][1049377.338752] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 21:54:52 2019][1049377.350493] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 17 previous similar messages [Tue Jan 29 22:08:18 2019][1050183.966616] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 22:08:18 2019][1050183.978363] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 10 previous similar messages [Tue Jan 29 22:19:02 2019][1050827.956137] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 22:19:02 2019][1050827.967880] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 16 previous similar messages [Tue Jan 29 22:29:32 2019][1051457.952809] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 22:29:32 2019][1051457.964548] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 27 previous similar messages [Tue Jan 29 22:39:46 2019][1052071.713627] Lustre: 38457:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Jan 29 22:39:46 2019][1052071.725455] Lustre: 38457:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 22 previous similar messages [Tue Jan 29 22:50:52 2019][1052738.136963] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Jan 29 22:50:52 2019][1052738.148937] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages [Tue Jan 29 23:01:02 2019][1053348.173977] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Jan 29 23:01:02 2019][1053348.185718] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Tue Jan 29 23:15:51 2019][1054236.508853] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 23:15:51 2019][1054236.520586] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Tue Jan 29 23:27:39 2019][1054944.977263] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Jan 29 23:27:39 2019][1054944.989003] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 12 previous similar messages [Tue Jan 29 23:37:54 2019][1055559.479510] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Jan 29 23:37:54 2019][1055559.491251] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14 previous similar messages [Tue Jan 29 23:48:38 2019][1056204.028434] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Jan 29 23:48:38 2019][1056204.040167] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages [Tue Jan 29 23:59:01 2019][1056826.842161] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Jan 29 23:59:01 2019][1056826.853907] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14 previous similar messages [Wed Jan 30 00:09:06 2019][1057431.864078] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 00:09:06 2019][1057431.875813] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 22 previous similar messages [Wed Jan 30 00:19:54 2019][1058080.302071] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 00:19:54 2019][1058080.313811] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 22 previous similar messages [Wed Jan 30 00:29:57 2019][1058682.602970] Lustre: 38457:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 00:29:57 2019][1058682.614801] Lustre: 38457:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages [Wed Jan 30 00:40:31 2019][1059316.544061] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 00:40:31 2019][1059316.555811] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 16 previous similar messages [Wed Jan 30 00:50:59 2019][1059945.122689] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 00:50:59 2019][1059945.134430] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 24 previous similar messages [Wed Jan 30 01:03:20 2019][1060685.663061] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 01:03:20 2019][1060685.674800] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 31 previous similar messages [Wed Jan 30 01:14:26 2019][1061351.939985] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 01:14:26 2019][1061351.951724] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14 previous similar messages [Wed Jan 30 01:24:45 2019][1061971.119004] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 01:24:45 2019][1061971.130749] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages [Wed Jan 30 01:36:23 2019][1062669.207968] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 01:36:23 2019][1062669.219718] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14 previous similar messages [Wed Jan 30 01:49:22 2019][1063448.366157] Lustre: 38457:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 01:49:22 2019][1063448.378023] Lustre: 38457:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 11 previous similar messages [Wed Jan 30 01:59:25 2019][1064051.424446] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 01:59:25 2019][1064051.436180] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages [Wed Jan 30 02:12:08 2019][1064814.408839] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 02:12:08 2019][1064814.420577] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Wed Jan 30 02:22:59 2019][1065464.656664] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 02:22:59 2019][1065464.668403] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Wed Jan 30 02:33:26 2019][1066092.372420] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 02:33:26 2019][1066092.384154] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 20 previous similar messages [Wed Jan 30 02:45:08 2019][1066793.629607] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 02:45:08 2019][1066793.641341] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages [Wed Jan 30 02:57:23 2019][1067528.554774] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 02:57:23 2019][1067528.566519] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 16 previous similar messages [Wed Jan 30 03:07:24 2019][1068129.975885] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 03:07:24 2019][1068129.987626] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14 previous similar messages [Wed Jan 30 03:18:57 2019][1068823.455202] Lustre: 38457:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 03:18:57 2019][1068823.467029] Lustre: 38457:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14 previous similar messages [Wed Jan 30 03:33:48 2019][1069714.519798] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 03:33:48 2019][1069714.531530] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages [Wed Jan 30 03:44:03 2019][1070328.754421] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 03:44:03 2019][1070328.766167] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 12 previous similar messages [Wed Jan 30 03:54:09 2019][1070934.891257] Lustre: 53421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 03:54:09 2019][1070934.903088] Lustre: 53421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 10 previous similar messages [Wed Jan 30 04:09:22 2019][1071848.322311] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 04:09:22 2019][1071848.334058] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 22 previous similar messages [Wed Jan 30 04:19:31 2019][1072456.863559] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 04:19:31 2019][1072456.875317] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 12 previous similar messages [Wed Jan 30 04:30:16 2019][1073102.284114] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 04:30:16 2019][1073102.295847] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 22 previous similar messages [Wed Jan 30 04:40:32 2019][1073718.100045] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 04:40:32 2019][1073718.111791] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 16 previous similar messages [Wed Jan 30 04:51:37 2019][1074383.301458] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 04:51:37 2019][1074383.313309] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Wed Jan 30 05:02:31 2019][1075037.678163] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 05:02:31 2019][1075037.689897] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 30 previous similar messages [Wed Jan 30 05:12:49 2019][1075655.126299] Lustre: 38457:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 05:12:49 2019][1075655.138126] Lustre: 38457:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 21 previous similar messages [Wed Jan 30 05:25:02 2019][1076387.830887] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 05:25:02 2019][1076387.842624] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages [Wed Jan 30 05:35:11 2019][1076997.243171] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 05:35:11 2019][1076997.254924] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 46 previous similar messages [Wed Jan 30 05:47:31 2019][1077737.029065] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 05:47:31 2019][1077737.040814] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 42 previous similar messages [Wed Jan 30 05:58:51 2019][1078417.616670] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 05:58:51 2019][1078417.628408] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14 previous similar messages [Wed Jan 30 06:09:27 2019][1079053.771213] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 06:09:28 2019][1079053.782972] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages [Wed Jan 30 06:19:48 2019][1079674.144153] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 06:19:48 2019][1079674.155898] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 29 previous similar messages [Wed Jan 30 06:26:51 2019][1080096.882127] LustreError: 9649:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016bd:0x7669:0x0] [Wed Jan 30 06:26:51 2019][1080097.128129] LustreError: 22161:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016bd:0x768b:0x0] [Wed Jan 30 06:31:20 2019][1080365.931437] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 06:31:20 2019][1080365.943174] Lustre: 6615:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 15 previous similar messages [Wed Jan 30 06:41:46 2019][1080992.828103] Lustre: 53421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 06:41:46 2019][1080992.839927] Lustre: 53421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 17 previous similar messages [Wed Jan 30 06:55:03 2019][1081789.550979] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 06:55:03 2019][1081789.562740] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 12 previous similar messages [Wed Jan 30 07:08:05 2019][1082571.802611] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 07:08:05 2019][1082571.814349] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Wed Jan 30 07:11:06 2019][1082752.624598] LustreError: 20861:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016bf:0x530:0x0] [Wed Jan 30 07:12:33 2019][1082838.922412] LustreError: 9007:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016b9:0x14103:0x0] [Wed Jan 30 07:18:28 2019][1083194.829480] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 07:18:28 2019][1083194.841223] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Wed Jan 30 07:51:05 2019][1085151.125222] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 07:51:05 2019][1085151.136963] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Wed Jan 30 07:53:20 2019][1085286.116769] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 07:53:20 2019][1085286.128603] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Wed Jan 30 07:57:41 2019][1085547.720938] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 08:05:13 2019][1085999.912225] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 08:05:13 2019][1085999.923962] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 15 previous similar messages [Wed Jan 30 08:16:18 2019][1086664.180894] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 08:16:18 2019][1086664.192649] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 24 previous similar messages [Wed Jan 30 08:27:51 2019][1087357.044594] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 08:27:51 2019][1087357.056342] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages [Wed Jan 30 08:31:45 2019][1087591.316933] LustreError: 20864:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016cc:0x1c440:0x0] [Wed Jan 30 08:31:46 2019][1087592.152968] LustreError: 20150:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016cc:0x1c761:0x0] [Wed Jan 30 08:35:47 2019][1087833.499961] LustreError: 7223:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016cc:0x1f4ea:0x0] [Wed Jan 30 08:36:33 2019][1087879.912039] LustreError: 8958:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016cc:0x1fd27:0x0] [Wed Jan 30 08:38:52 2019][1088018.936454] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 08:38:53 2019][1088018.948265] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 16 previous similar messages [Wed Jan 30 08:40:59 2019][1088145.808948] LustreError: 9007:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016cd:0xdd2:0x0] [Wed Jan 30 08:49:04 2019][1088630.496179] Lustre: 53421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 08:49:04 2019][1088630.508009] Lustre: 53421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 10 previous similar messages [Wed Jan 30 08:50:30 2019][1088716.608280] LustreError: 6805:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016cb:0x19e5e:0x0] [Wed Jan 30 09:02:42 2019][1089448.864780] Lustre: 53421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 09:02:42 2019][1089448.876606] Lustre: 53421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 16 previous similar messages [Wed Jan 30 09:14:36 2019][1090162.824587] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 09:14:36 2019][1090162.836327] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 09:25:04 2019][1090790.715916] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 09:25:04 2019][1090790.727654] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 12 previous similar messages [Wed Jan 30 09:39:13 2019][1091639.673426] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 09:39:13 2019][1091639.685160] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages [Wed Jan 30 09:51:05 2019][1092351.383929] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 09:51:05 2019][1092351.395684] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages [Wed Jan 30 10:01:44 2019][1092990.222798] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 10:01:44 2019][1092990.234541] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages [Wed Jan 30 10:07:24 2019][1093330.825897] LustreError: 8958:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016d6:0xf2f7:0x0] [Wed Jan 30 10:08:37 2019][1093403.270465] Lustre: DEBUG MARKER: Wed Jan 30 10:09:16 2019 [Wed Jan 30 10:08:37 2019][1093403.270465] [Wed Jan 30 10:12:43 2019][1093649.333614] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 10:12:43 2019][1093649.345354] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 10:14:32 2019][1093758.278399] LustreError: 6783:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016d6:0x182c3:0x0] [Wed Jan 30 10:16:57 2019][1093903.815019] LustreError: 6211:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016d6:0x1b222:0x0] [Wed Jan 30 10:17:37 2019][1093943.968666] LustreError: 9007:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016db:0x98c3:0x0] [Wed Jan 30 10:17:47 2019][1093954.029976] LustreError: 9649:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016db:0x9cea:0x0] [Wed Jan 30 10:20:40 2019][1094126.108674] Lustre: DEBUG MARKER: Wed Jan 30 10:21:19 2019 [Wed Jan 30 10:20:40 2019][1094126.108674] [Wed Jan 30 10:23:15 2019][1094281.457332] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 10:23:15 2019][1094281.469076] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Wed Jan 30 10:34:23 2019][1094949.413299] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 10:34:23 2019][1094949.425041] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages [Wed Jan 30 10:49:58 2019][1095884.794936] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 10:49:58 2019][1095884.806673] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Wed Jan 30 11:01:45 2019][1096591.954416] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 11:01:45 2019][1096591.966154] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages [Wed Jan 30 11:15:21 2019][1097407.793757] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 11:15:21 2019][1097407.805497] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 11:23:00 2019][1097866.758042] LNetError: 5877:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Wed Jan 30 11:26:52 2019][1098098.981341] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 11:26:52 2019][1098098.993101] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 12 previous similar messages [Wed Jan 30 11:38:06 2019][1098772.673762] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 11:38:06 2019][1098772.685497] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Wed Jan 30 11:53:44 2019][1099710.487034] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 11:53:44 2019][1099710.498772] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages [Wed Jan 30 12:04:30 2019][1100356.847173] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 12:04:30 2019][1100356.858912] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 12:06:07 2019][1100454.113919] Lustre: fir-MDT0002: haven't heard from client 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91493821d000, cur 1548878807 expire 1548878657 last 1548878580 [Wed Jan 30 12:06:09 2019][1100456.113251] Lustre: fir-MDT0000: haven't heard from client 85c45b71-10aa-be9f-071b-9a9c50548db2 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9128dd6bd000, cur 1548878809 expire 1548878659 last 1548878582 [Wed Jan 30 12:21:50 2019][1101397.184170] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 12:41:42 2019][1102588.882708] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 12:41:42 2019][1102588.894447] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages [Wed Jan 30 12:53:26 2019][1103292.345344] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 12:53:26 2019][1103292.357080] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 10 previous similar messages [Wed Jan 30 13:04:53 2019][1103979.791203] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 13:04:53 2019][1103979.802940] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 16 previous similar messages [Wed Jan 30 13:16:47 2019][1104693.737556] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 13:16:47 2019][1104693.749307] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 19 previous similar messages [Wed Jan 30 13:33:56 2019][1105723.354744] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 13:33:57 2019][1105723.366482] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages [Wed Jan 30 13:46:15 2019][1106461.632208] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 13:46:15 2019][1106461.643945] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Wed Jan 30 14:04:22 2019][1107549.153458] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 14:04:22 2019][1107549.165198] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Wed Jan 30 14:28:49 2019][1109015.990077] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 14:28:49 2019][1109016.001819] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Wed Jan 30 14:44:46 2019][1109973.340725] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 14:49:04 2019][1110230.594330] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 14:56:43 2019][1110690.444816] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 15:02:55 2019][1111061.886635] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 15:02:55 2019][1111061.898374] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 15:05:14 2019][1111200.963418] Lustre: 38457:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 15:05:14 2019][1111200.975241] Lustre: 38457:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 15:13:48 2019][1111715.489661] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 15:34:10 2019][1112937.497416] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 15:49:39 2019][1113866.388460] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 16:34:51 2019][1116578.506172] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 16:39:21 2019][1116848.164416] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 16:40:04 2019][1116891.098085] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 16:40:21 2019][1116907.867978] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 16:52:50 2019][1117657.091477] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 16:52:50 2019][1117657.103216] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 16:58:09 2019][1117975.848930] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 16:59:21 2019][1118048.029438] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 16:59:21 2019][1118048.041174] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 17:17:34 2019][1119140.767913] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 17:17:50 2019][1119156.989105] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 17:17:50 2019][1119157.000845] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 17:27:12 2019][1119719.714175] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 17:27:13 2019][1119719.725919] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Wed Jan 30 17:34:22 2019][1120149.205862] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 17:44:15 2019][1120742.306250] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 17:44:42 2019][1120769.290462] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 17:44:42 2019][1120769.302202] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 17:52:34 2019][1121241.119778] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 17:52:34 2019][1121241.131514] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 17:53:32 2019][1121299.320496] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 18:03:01 2019][1121868.263821] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 18:03:01 2019][1121868.275560] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Wed Jan 30 18:11:16 2019][1122363.372931] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 18:21:16 2019][1122962.941641] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 18:21:16 2019][1122962.953400] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 18:36:48 2019][1123894.798430] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 18:36:48 2019][1123894.810169] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 18:47:28 2019][1124535.120134] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 18:47:28 2019][1124535.131879] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Wed Jan 30 19:03:25 2019][1125492.169606] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 19:03:25 2019][1125492.181349] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Wed Jan 30 19:15:07 2019][1126194.066277] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 19:15:07 2019][1126194.078010] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 19:28:16 2019][1126983.464733] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 19:28:16 2019][1126983.476490] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages [Wed Jan 30 19:38:35 2019][1127602.133108] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 19:38:35 2019][1127602.144842] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 41 previous similar messages [Wed Jan 30 19:49:40 2019][1128267.771276] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 19:49:40 2019][1128267.783010] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 33 previous similar messages [Wed Jan 30 20:04:08 2019][1129135.696629] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 20:04:08 2019][1129135.708362] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages [Wed Jan 30 20:32:02 2019][1130809.225215] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 20:32:02 2019][1130809.236964] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 20:35:20 2019][1131007.197920] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 20:35:20 2019][1131007.209662] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages [Wed Jan 30 20:43:17 2019][1131484.598403] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 20:43:17 2019][1131484.610160] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Wed Jan 30 20:49:56 2019][1131883.916633] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 20:49:56 2019][1131883.928371] Lustre: 8999:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Wed Jan 30 21:14:30 2019][1133358.038658] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 21:14:31 2019][1133358.050484] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Wed Jan 30 21:16:25 2019][1133472.353523] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Jan 30 21:16:25 2019][1133472.365259] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Wed Jan 30 21:31:41 2019][1134388.039933] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 21:34:55 2019][1134582.198080] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 21:40:40 2019][1134927.280131] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 21:40:40 2019][1134927.291865] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Wed Jan 30 21:45:24 2019][1135211.760270] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 21:50:26 2019][1135513.925863] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 21:50:26 2019][1135513.937604] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Wed Jan 30 21:56:43 2019][1135890.153968] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 22:05:30 2019][1136417.721619] LustreError: 15082:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0000: BRW to missing obj [0x2000016e8:0x15c7:0x0] [Wed Jan 30 22:09:05 2019][1136632.193608] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 22:09:05 2019][1136632.205347] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Wed Jan 30 22:24:05 2019][1137532.445848] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 22:24:05 2019][1137532.457586] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages [Wed Jan 30 22:35:02 2019][1138189.998032] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 22:35:02 2019][1138190.009790] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 24 previous similar messages [Wed Jan 30 22:49:15 2019][1139042.530347] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 22:49:15 2019][1139042.542086] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Wed Jan 30 23:06:01 2019][1140048.839325] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 23:06:01 2019][1140048.851064] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Wed Jan 30 23:18:47 2019][1140815.064129] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 23:18:47 2019][1140815.075868] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Wed Jan 30 23:35:12 2019][1141799.423390] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 23:35:12 2019][1141799.435131] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Wed Jan 30 23:45:59 2019][1142446.598422] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Jan 30 23:45:59 2019][1142446.610162] Lustre: 8994:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Thu Jan 31 00:02:29 2019][1143436.784949] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 00:02:29 2019][1143436.796687] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Thu Jan 31 00:56:55 2019][1146702.365367] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 00:56:55 2019][1146702.377107] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Thu Jan 31 01:19:36 2019][1148064.032297] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 01:22:48 2019][1148255.404670] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 01:33:20 2019][1148887.527800] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 01:37:16 2019][1149123.668865] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 01:37:16 2019][1149123.680600] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 01:37:42 2019][1149150.158192] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 01:39:54 2019][1149282.232709] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 01:48:29 2019][1149796.559618] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 01:48:29 2019][1149796.571357] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 02:00:25 2019][1150512.761297] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 02:27:31 2019][1152139.116335] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 02:27:31 2019][1152139.128076] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 02:28:59 2019][1152226.648297] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 02:28:59 2019][1152226.660053] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 02:35:58 2019][1152645.489353] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 02:35:58 2019][1152645.501095] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 02:52:34 2019][1153641.631673] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 03:08:13 2019][1154581.341557] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 03:18:14 2019][1155182.030890] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 03:50:35 2019][1157122.968582] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 03:50:35 2019][1157122.980323] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Thu Jan 31 03:53:08 2019][1157276.132506] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 04:13:28 2019][1158496.531916] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 04:13:28 2019][1158496.543653] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 05:15:38 2019][1162226.593330] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 05:29:31 2019][1163059.444920] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 05:29:31 2019][1163059.456656] Lustre: 6795:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Thu Jan 31 05:30:02 2019][1163090.265701] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 05:30:02 2019][1163090.277433] Lustre: 9027:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 05:30:16 2019][1163104.488195] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 05:40:10 2019][1163698.022791] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 05:40:10 2019][1163698.034531] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Thu Jan 31 05:41:04 2019][1163751.753168] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 05:41:04 2019][1163751.764901] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Thu Jan 31 05:41:27 2019][1163775.377006] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 05:41:27 2019][1163775.388743] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 05:42:11 2019][1163818.849040] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 05:49:27 2019][1164254.839940] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 05:52:44 2019][1164452.303162] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 05:52:44 2019][1164452.314916] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 06:10:55 2019][1165543.666103] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 06:42:38 2019][1167446.516971] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 07:34:54 2019][1170582.402164] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 07:38:07 2019][1170775.079628] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 07:38:07 2019][1170775.091361] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Thu Jan 31 07:50:50 2019][1171538.155542] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 07:50:50 2019][1171538.167278] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 07:56:23 2019][1171871.414512] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 07:56:23 2019][1171871.426250] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 07:58:41 2019][1172009.524840] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 07:59:17 2019][1172044.891228] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 07:59:17 2019][1172044.902966] Lustre: 6169:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Thu Jan 31 08:01:25 2019][1172173.169704] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 08:01:25 2019][1172173.181436] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 08:03:10 2019][1172277.923331] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 08:03:10 2019][1172277.935178] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 08:13:24 2019][1172892.361203] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 08:15:36 2019][1173024.130852] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 08:15:36 2019][1173024.142590] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 08:17:03 2019][1173111.869270] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 08:17:03 2019][1173111.881003] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Thu Jan 31 08:21:05 2019][1173353.883285] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 08:21:05 2019][1173353.895015] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 10 previous similar messages [Thu Jan 31 08:25:36 2019][1173624.593004] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 08:25:36 2019][1173624.604743] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 17 previous similar messages [Thu Jan 31 09:17:47 2019][1176755.628625] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 09:17:47 2019][1176755.640362] Lustre: 6613:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 29 previous similar messages [Thu Jan 31 09:37:24 2019][1177932.795828] Lustre: DEBUG MARKER: Thu Jan 31 09:38:03 2019 [Thu Jan 31 09:37:24 2019][1177932.795828] [Thu Jan 31 10:42:21 2019][1181829.426710] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 10:42:21 2019][1181829.438469] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 12:15:06 2019][1187394.457954] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 12:30:05 2019][1188293.854795] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 12:31:48 2019][1188396.768090] Lustre: 32929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 13:13:38 2019][1190907.117695] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 14:06:12 2019][1194060.643565] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 17:45:00 2019][1207189.157186] Lustre: 9028:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 18:15:48 2019][1209037.093606] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 18:15:48 2019][1209037.105342] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 18:23:25 2019][1209494.581666] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 18:24:46 2019][1209574.844921] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 18:32:17 2019][1210026.448994] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 18:34:11 2019][1210140.633017] Lustre: 38457:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 18:34:11 2019][1210140.644858] Lustre: 38457:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Thu Jan 31 18:39:56 2019][1210485.549652] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 18:43:34 2019][1210703.039877] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 18:44:18 2019][1210747.404174] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 19:00:06 2019][1211695.575854] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 19:00:06 2019][1211695.587589] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Thu Jan 31 19:25:34 2019][1213223.254772] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 19:25:34 2019][1213223.266507] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Jan 31 19:26:32 2019][1213281.311109] Lustre: 9033:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 19:33:03 2019][1213672.734715] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 19:34:36 2019][1213765.176590] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 19:35:35 2019][1213824.122988] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 19:50:03 2019][1214692.020746] Lustre: 38457:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 19:51:30 2019][1214779.729428] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 20:13:10 2019][1216079.569959] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Thu Jan 31 20:24:59 2019][1216788.594057] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Fri Feb 1 15:04:09 2019][1283940.127933] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Fri Feb 1 15:04:09 2019][1283940.137689] Lustre: Skipped 2 previous similar messages [Fri Feb 1 15:05:48 2019][1284038.636095] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Fri Feb 1 15:05:51 2019][1284041.540751] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Fri Feb 1 15:06:05 2019][1284056.036909] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Fri Feb 1 15:06:27 2019][1284078.332096] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Fri Feb 1 15:06:27 2019][1284078.341829] Lustre: Skipped 2 previous similar messages [Fri Feb 1 15:07:57 2019][1284168.167134] Lustre: MGS: haven't heard from client aed5f0f7-733c-f963-d6e5-a4993de0be55 (at 10.8.6.34@o2ib6) in 228 seconds. I think it's dead, and I am evicting it. exp ffff9129a969b400, cur 1549062516 expire 1549062366 last 1549062288 [Fri Feb 1 15:07:57 2019][1284168.188234] Lustre: Skipped 1 previous similar message [Fri Feb 1 15:10:15 2019][1284305.939067] Lustre: MGS: haven't heard from client e1765530-8352-d3b9-d806-e6c10fc2cff6 (at 10.8.6.34@o2ib6) in 228 seconds. I think it's dead, and I am evicting it. exp ffff9123459b4400, cur 1549062654 expire 1549062504 last 1549062426 [Fri Feb 1 15:12:20 2019][1284430.681287] Lustre: MGS: Connection restored to cc848e9b-ffd5-ec4e-a847-e91c935dcca0 (at 10.8.6.34@o2ib6) [Fri Feb 1 15:12:20 2019][1284430.691031] Lustre: Skipped 1 previous similar message [Mon Feb 4 07:58:54 2019][1517629.781949] Lustre: fir-MDT0002: haven't heard from client 3577bb83-18fa-14e9-4b08-b3335e8f2568 (at 10.9.0.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9129cfca5c00, cur 1549295972 expire 1549295822 last 1549295745 [Mon Feb 4 08:03:19 2019][1517895.435804] Lustre: MGS: Connection restored to 6caeea3e-5b74-9a8d-d8ac-3bf9f06c410e (at 10.9.0.1@o2ib4) [Mon Feb 4 08:03:19 2019][1517895.445457] Lustre: Skipped 2 previous similar messages [Mon Feb 4 09:26:07 2019][1522863.002064] Lustre: fir-MDT0000: haven't heard from client a0fd8789-8ae9-fb75-85b8-13d1c4cef8d2 (at 10.9.101.60@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9159f5376000, cur 1549301205 expire 1549301055 last 1549300978 [Mon Feb 4 09:26:07 2019][1522863.024029] Lustre: Skipped 2 previous similar messages [Mon Feb 4 09:26:08 2019][1522864.290902] Lustre: MGS: haven't heard from client 9a32f361-a9e7-0091-ce36-b7ab7cd443d4 (at 10.9.101.60@o2ib4) in 228 seconds. I think it's dead, and I am evicting it. exp ffff912981e75000, cur 1549301206 expire 1549301056 last 1549300978 [Mon Feb 4 09:26:08 2019][1522864.312175] Lustre: Skipped 1 previous similar message [Mon Feb 4 15:24:05 2019][1544342.147672] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 15:24:05 2019][1544342.157414] Lustre: Skipped 2 previous similar messages [Mon Feb 4 15:29:40 2019][1544676.876794] Lustre: MGS: Connection restored to e421d59a-4482-eafc-ea85-c173e2c58e62 (at 10.8.3.11@o2ib6) [Mon Feb 4 15:29:40 2019][1544676.886537] Lustre: Skipped 2 previous similar messages [Mon Feb 4 16:03:51 2019][1546727.575218] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 16:03:51 2019][1546727.584961] Lustre: Skipped 2 previous similar messages [Mon Feb 4 16:32:11 2019][1548427.733720] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 16:32:11 2019][1548427.743455] Lustre: Skipped 2 previous similar messages [Mon Feb 4 17:01:21 2019][1550177.836121] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 17:01:21 2019][1550177.845866] Lustre: Skipped 2 previous similar messages [Mon Feb 4 17:29:37 2019][1551874.040311] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 17:29:37 2019][1551874.050065] Lustre: Skipped 2 previous similar messages [Mon Feb 4 17:59:43 2019][1553680.055911] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 17:59:43 2019][1553680.065654] Lustre: Skipped 2 previous similar messages [Mon Feb 4 18:30:30 2019][1555527.116133] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 18:30:30 2019][1555527.125878] Lustre: Skipped 2 previous similar messages [Mon Feb 4 18:59:17 2019][1557254.240316] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 18:59:17 2019][1557254.250053] Lustre: Skipped 2 previous similar messages [Mon Feb 4 19:27:26 2019][1558943.218167] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 19:27:26 2019][1558943.227910] Lustre: Skipped 2 previous similar messages [Mon Feb 4 19:56:37 2019][1560693.842067] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 19:56:37 2019][1560693.851819] Lustre: Skipped 2 previous similar messages [Mon Feb 4 20:24:51 2019][1562388.432285] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 20:24:51 2019][1562388.442033] Lustre: Skipped 2 previous similar messages [Mon Feb 4 20:38:46 2019][1563223.612013] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 20:38:46 2019][1563223.621748] Lustre: Skipped 2 previous similar messages [Mon Feb 4 21:07:19 2019][1564936.684544] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 21:07:19 2019][1564936.694284] Lustre: Skipped 2 previous similar messages [Mon Feb 4 21:35:58 2019][1566655.826329] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 21:35:58 2019][1566655.836082] Lustre: Skipped 2 previous similar messages [Mon Feb 4 22:03:54 2019][1568331.794368] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 22:03:54 2019][1568331.804118] Lustre: Skipped 2 previous similar messages [Mon Feb 4 22:32:29 2019][1570046.470354] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 22:32:29 2019][1570046.480094] Lustre: Skipped 2 previous similar messages [Mon Feb 4 23:01:42 2019][1571799.025401] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 23:01:42 2019][1571799.035142] Lustre: Skipped 2 previous similar messages [Mon Feb 4 23:30:28 2019][1573525.373456] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 23:30:28 2019][1573525.383200] Lustre: Skipped 2 previous similar messages [Mon Feb 4 23:44:29 2019][1574367.022598] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 23:44:29 2019][1574367.032342] Lustre: Skipped 2 previous similar messages [Mon Feb 4 23:58:43 2019][1575220.303171] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Mon Feb 4 23:58:43 2019][1575220.312915] Lustre: Skipped 2 previous similar messages [Tue Feb 5 00:27:00 2019][1576917.684781] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 00:27:00 2019][1576917.694520] Lustre: Skipped 2 previous similar messages [Tue Feb 5 00:56:06 2019][1578663.210789] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 00:56:06 2019][1578663.220536] Lustre: Skipped 2 previous similar messages [Tue Feb 5 01:24:36 2019][1580373.882915] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 01:24:36 2019][1580373.892655] Lustre: Skipped 2 previous similar messages [Tue Feb 5 01:52:49 2019][1582066.633096] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 01:52:49 2019][1582066.642856] Lustre: Skipped 2 previous similar messages [Tue Feb 5 02:21:04 2019][1583761.607489] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 02:21:04 2019][1583761.617239] Lustre: Skipped 2 previous similar messages [Tue Feb 5 02:48:54 2019][1585432.115415] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 02:48:54 2019][1585432.125163] Lustre: Skipped 2 previous similar messages [Tue Feb 5 03:18:04 2019][1587182.008054] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 03:18:04 2019][1587182.017799] Lustre: Skipped 2 previous similar messages [Tue Feb 5 03:46:47 2019][1588905.381159] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 03:46:47 2019][1588905.390905] Lustre: Skipped 2 previous similar messages [Tue Feb 5 04:15:15 2019][1590613.216776] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 04:15:15 2019][1590613.226528] Lustre: Skipped 2 previous similar messages [Tue Feb 5 04:43:44 2019][1592321.476195] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 04:43:44 2019][1592321.485933] Lustre: Skipped 2 previous similar messages [Tue Feb 5 05:16:36 2019][1594293.785670] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 05:16:36 2019][1594293.795416] Lustre: Skipped 2 previous similar messages [Tue Feb 5 05:45:17 2019][1596014.940855] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 05:45:17 2019][1596014.950600] Lustre: Skipped 2 previous similar messages [Tue Feb 5 06:13:37 2019][1597715.295022] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 06:13:37 2019][1597715.304767] Lustre: Skipped 2 previous similar messages [Tue Feb 5 06:27:41 2019][1598558.737363] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 06:27:41 2019][1598558.747106] Lustre: Skipped 2 previous similar messages [Tue Feb 5 06:54:52 2019][1600190.514406] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 06:54:52 2019][1600190.524149] Lustre: Skipped 2 previous similar messages [Tue Feb 5 07:07:57 2019][1600975.416526] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 07:07:57 2019][1600975.426273] Lustre: Skipped 2 previous similar messages [Tue Feb 5 07:34:05 2019][1602542.955123] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 07:34:05 2019][1602542.964869] Lustre: Skipped 2 previous similar messages [Tue Feb 5 08:00:01 2019][1604099.312097] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 08:00:01 2019][1604099.321851] Lustre: Skipped 2 previous similar messages [Tue Feb 5 08:26:11 2019][1605669.586333] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 08:26:11 2019][1605669.596076] Lustre: Skipped 2 previous similar messages [Tue Feb 5 08:52:28 2019][1607246.621142] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 08:52:28 2019][1607246.630886] Lustre: Skipped 2 previous similar messages [Tue Feb 5 09:18:25 2019][1608802.872301] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 09:18:25 2019][1608802.882046] Lustre: Skipped 2 previous similar messages [Tue Feb 5 09:39:00 2019][1610037.957321] Lustre: MGS: Connection restored to 74841472-5fb7-9c47-503a-c07169a69783 (at 10.9.101.60@o2ib4) [Tue Feb 5 09:39:00 2019][1610037.967239] Lustre: Skipped 2 previous similar messages [Tue Feb 5 09:44:19 2019][1610357.003967] Lustre: MGS: Connection restored to 9a3fec86-6c87-9861-a86b-61fc0ee356f3 (at 10.8.11.9@o2ib6) [Tue Feb 5 09:44:19 2019][1610357.013718] Lustre: Skipped 2 previous similar messages [Tue Feb 5 09:44:43 2019][1610381.386882] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 09:44:43 2019][1610381.396633] Lustre: Skipped 2 previous similar messages [Tue Feb 5 09:45:39 2019][1610436.955497] Lustre: MGS: Connection restored to 74841472-5fb7-9c47-503a-c07169a69783 (at 10.9.101.60@o2ib4) [Tue Feb 5 09:45:39 2019][1610436.965410] Lustre: Skipped 2 previous similar messages [Tue Feb 5 09:49:47 2019][1610685.411721] Lustre: MGS: Connection restored to 74841472-5fb7-9c47-503a-c07169a69783 (at 10.9.101.60@o2ib4) [Tue Feb 5 09:49:47 2019][1610685.421646] Lustre: Skipped 2 previous similar messages [Tue Feb 5 10:00:32 2019][1611330.092249] Lustre: fir-MDT0002: haven't heard from client 28764730-adc8-1b1a-0aa7-9bb3363d4889 (at 10.9.101.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9129cfca5800, cur 1549389670 expire 1549389520 last 1549389443 [Tue Feb 5 10:00:32 2019][1611330.114224] Lustre: Skipped 3 previous similar messages [Tue Feb 5 10:00:32 2019][1611330.856709] Lustre: MGS: haven't heard from client b45f97c4-4cf7-fe2a-965d-6f2f61bac156 (at 10.9.101.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9139fa0cd800, cur 1549389670 expire 1549389520 last 1549389443 [Tue Feb 5 10:10:15 2019][1611913.246397] Lustre: MGS: Connection restored to 9a3fec86-6c87-9861-a86b-61fc0ee356f3 (at 10.8.11.9@o2ib6) [Tue Feb 5 10:10:15 2019][1611913.256137] Lustre: Skipped 2 previous similar messages [Tue Feb 5 10:10:59 2019][1611956.949303] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 10:10:59 2019][1611956.959043] Lustre: Skipped 2 previous similar messages [Tue Feb 5 10:11:11 2019][1611969.725511] Lustre: MGS: Connection restored to c3ea57d5-46ff-5c6e-1800-776bdd796fdc (at 10.8.20.15@o2ib6) [Tue Feb 5 10:11:11 2019][1611969.735342] Lustre: Skipped 2 previous similar messages [Tue Feb 5 10:34:47 2019][1613385.470473] Lustre: MGS: Connection restored to b45f97c4-4cf7-fe2a-965d-6f2f61bac156 (at 10.9.101.59@o2ib4) [Tue Feb 5 10:34:47 2019][1613385.480393] Lustre: Skipped 2 previous similar messages [Tue Feb 5 10:36:34 2019][1613492.662780] Lustre: MGS: Connection restored to 9a3fec86-6c87-9861-a86b-61fc0ee356f3 (at 10.8.11.9@o2ib6) [Tue Feb 5 10:36:34 2019][1613492.672525] Lustre: Skipped 2 previous similar messages [Tue Feb 5 10:36:51 2019][1613509.233338] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 10:36:51 2019][1613509.243078] Lustre: Skipped 2 previous similar messages [Tue Feb 5 10:37:04 2019][1613522.271141] Lustre: MGS: Connection restored to c3ea57d5-46ff-5c6e-1800-776bdd796fdc (at 10.8.20.15@o2ib6) [Tue Feb 5 10:37:04 2019][1613522.280979] Lustre: Skipped 2 previous similar messages [Tue Feb 5 10:44:55 2019][1613993.921881] Lustre: MGS: Connection restored to b45f97c4-4cf7-fe2a-965d-6f2f61bac156 (at 10.9.101.59@o2ib4) [Tue Feb 5 10:44:55 2019][1613993.931799] Lustre: Skipped 2 previous similar messages [Tue Feb 5 11:20:44 2019][1616142.362274] Lustre: MGS: Connection restored to 9a3fec86-6c87-9861-a86b-61fc0ee356f3 (at 10.8.11.9@o2ib6) [Tue Feb 5 11:20:44 2019][1616142.372015] Lustre: Skipped 2 previous similar messages [Tue Feb 5 11:21:04 2019][1616162.239934] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 11:21:04 2019][1616162.249673] Lustre: Skipped 2 previous similar messages [Tue Feb 5 11:21:09 2019][1616167.972289] Lustre: MGS: Connection restored to c3ea57d5-46ff-5c6e-1800-776bdd796fdc (at 10.8.20.15@o2ib6) [Tue Feb 5 11:21:09 2019][1616167.982118] Lustre: Skipped 2 previous similar messages [Tue Feb 5 11:22:23 2019][1616241.467709] Lustre: MGS: Connection restored to eebf3425-6403-e9a2-7364-9e59b8e7da62 (at 10.8.1.29@o2ib6) [Tue Feb 5 11:22:23 2019][1616241.477456] Lustre: Skipped 2 previous similar messages [Tue Feb 5 11:25:19 2019][1616417.534009] Lustre: MGS: Connection restored to 894bcf35-2ea8-fdc8-5c4d-393875333293 (at 10.9.107.69@o2ib4) [Tue Feb 5 11:25:19 2019][1616417.543922] Lustre: Skipped 2 previous similar messages [Tue Feb 5 11:33:31 2019][1616909.057148] Lustre: MGS: Connection restored to 1580e2c4-8df5-2686-8895-9de4ab24ba8a (at 10.8.1.1@o2ib6) [Tue Feb 5 11:33:31 2019][1616909.066808] Lustre: Skipped 5 previous similar messages [Tue Feb 5 11:38:18 2019][1617196.388907] Lustre: MGS: Connection restored to 0ea9fee6-e077-d061-e432-f6b956a8e0e1 (at 10.8.1.33@o2ib6) [Tue Feb 5 11:38:18 2019][1617196.398657] Lustre: Skipped 2 previous similar messages [Tue Feb 5 11:38:24 2019][1617202.321026] Lustre: fir-MDT0000: haven't heard from client e19ebed6-7d88-5c59-21b2-1eb456a47be8 (at 10.9.101.60@o2ib4) in 228 seconds. I think it's dead, and I am evicting it. exp ffff9137524cb800, cur 1549395542 expire 1549395392 last 1549395314 [Tue Feb 5 11:38:24 2019][1617202.342989] Lustre: Skipped 1 previous similar message [Tue Feb 5 11:38:27 2019][1617205.991174] Lustre: fir-MDT0002: haven't heard from client e19ebed6-7d88-5c59-21b2-1eb456a47be8 (at 10.9.101.60@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91239cd43800, cur 1549395545 expire 1549395395 last 1549395318 [Tue Feb 5 11:38:27 2019][1617206.013147] Lustre: Skipped 1 previous similar message [Tue Feb 5 11:38:30 2019][1617208.625925] Lustre: MGS: haven't heard from client f31e5def-7677-d08b-216d-9c40369c9872 (at 10.9.101.60@o2ib4) in 230 seconds. I think it's dead, and I am evicting it. exp ffff9122d08c1400, cur 1549395548 expire 1549395398 last 1549395318 [Tue Feb 5 11:38:30 2019][1617208.647194] Lustre: Skipped 1 previous similar message [Tue Feb 5 11:39:31 2019][1617269.730232] Lustre: MGS: Connection restored to 9c0fbda2-4abb-be73-acca-d060a5e87982 (at 10.8.1.31@o2ib6) [Tue Feb 5 11:39:31 2019][1617269.739980] Lustre: Skipped 6 previous similar messages [Tue Feb 5 11:41:25 2019][1617383.179491] Lustre: MGS: haven't heard from client 02db8660-0dc4-b9e2-c9c0-e812077d4d43 (at 10.8.1.4@o2ib6) in 229 seconds. I think it's dead, and I am evicting it. exp ffff9122d08c1000, cur 1549395723 expire 1549395573 last 1549395494 [Tue Feb 5 11:41:25 2019][1617383.200502] Lustre: Skipped 1 previous similar message [Tue Feb 5 11:46:08 2019][1617666.459673] Lustre: MGS: Connection restored to 9a3fec86-6c87-9861-a86b-61fc0ee356f3 (at 10.8.11.9@o2ib6) [Tue Feb 5 11:46:08 2019][1617666.469427] Lustre: Skipped 5 previous similar messages [Tue Feb 5 12:01:33 2019][1618591.537469] Lustre: MGS: Connection restored to 0cbf0a49-12e3-d62d-9d6d-05e24199e930 (at 10.8.1.2@o2ib6) [Tue Feb 5 12:01:33 2019][1618591.547126] Lustre: Skipped 8 previous similar messages [Tue Feb 5 12:05:14 2019][1618812.466232] Lustre: MGS: Connection restored to 02db8660-0dc4-b9e2-c9c0-e812077d4d43 (at 10.8.1.4@o2ib6) [Tue Feb 5 12:05:14 2019][1618812.475895] Lustre: Skipped 2 previous similar messages [Tue Feb 5 12:11:59 2019][1619217.431031] Lustre: MGS: Connection restored to 9a3fec86-6c87-9861-a86b-61fc0ee356f3 (at 10.8.11.9@o2ib6) [Tue Feb 5 12:11:59 2019][1619217.440768] Lustre: Skipped 2 previous similar messages [Tue Feb 5 12:16:02 2019][1619460.891906] Lustre: MGS: Connection restored to 02db8660-0dc4-b9e2-c9c0-e812077d4d43 (at 10.8.1.4@o2ib6) [Tue Feb 5 12:16:02 2019][1619460.901575] Lustre: Skipped 11 previous similar messages [Tue Feb 5 12:27:07 2019][1620125.154150] Lustre: fir-MDT0002: haven't heard from client 0efd883e-065d-0727-44f7-75a6230140a1 (at 10.9.0.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9129cb53ec00, cur 1549398465 expire 1549398315 last 1549398238 [Tue Feb 5 12:28:14 2019][1620192.855498] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Feb 5 12:37:00 2019][1620719.014621] Lustre: 15218:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 12:37:00 2019][1620719.026474] Lustre: 15218:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Feb 5 12:37:33 2019][1620751.597534] Lustre: MGS: Connection restored to 9a3fec86-6c87-9861-a86b-61fc0ee356f3 (at 10.8.11.9@o2ib6) [Tue Feb 5 12:37:33 2019][1620751.607273] Lustre: Skipped 2 previous similar messages [Tue Feb 5 12:40:42 2019][1620940.695909] Lustre: MGS: Connection restored to eea8150a-0569-63a3-187d-7721c9515ca4 (at 10.9.103.39@o2ib4) [Tue Feb 5 12:40:42 2019][1620940.705828] Lustre: Skipped 8 previous similar messages [Tue Feb 5 12:47:39 2019][1621357.345819] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 12:47:39 2019][1621357.357619] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Tue Feb 5 12:53:16 2019][1621695.128422] Lustre: 15409:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Feb 5 13:00:46 2019][1622144.670224] Lustre: 15169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Feb 5 13:01:27 2019][1622185.182026] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Feb 5 13:03:37 2019][1622315.475032] Lustre: MGS: Connection restored to 9a3fec86-6c87-9861-a86b-61fc0ee356f3 (at 10.8.11.9@o2ib6) [Tue Feb 5 13:03:37 2019][1622315.484772] Lustre: Skipped 8 previous similar messages [Tue Feb 5 13:03:52 2019][1622330.421121] Lustre: MGS: Connection restored to ff1032ee-a0fe-1065-6b12-2d8b61fa9745 (at 10.8.15.7@o2ib6) [Tue Feb 5 13:03:52 2019][1622330.430862] Lustre: Skipped 2 previous similar messages [Tue Feb 5 13:09:52 2019][1622690.463927] Lustre: 9181:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Feb 5 13:17:50 2019][1623168.411201] Lustre: MGS: Connection restored to 88130480-2e5c-e317-8933-a7c5aa30a9a9 (at 10.9.0.2@o2ib4) [Tue Feb 5 13:17:50 2019][1623168.420858] Lustre: Skipped 5 previous similar messages [Tue Feb 5 13:21:38 2019][1623396.419004] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Tue Feb 5 13:21:38 2019][1623396.429263] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.201@o2ib7 (37): c: 7, oc: 0, rc: 8 [Tue Feb 5 13:21:44 2019][1623403.073184] Lustre: 9182:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549401698/real 1549401698] req@ffff912929a28300 x1623679301214896/t0(0) o106->fir-MDT0000@10.8.1.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401742 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Tue Feb 5 13:21:44 2019][1623403.074202] LNetError: 15278:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.25@o2ib6 from [Tue Feb 5 13:21:44 2019][1623403.111121] Lustre: 9182:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5116 previous similar messages [Tue Feb 5 13:21:45 2019][1623403.574178] LNetError: 9182:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.29@o2ib6 from [Tue Feb 5 13:21:45 2019][1623403.574181] Lustre: 15278:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549401743/real 1549401743] req@ffff911d391ec200 x1623679301214928/t0(0) o106->fir-MDT0000@10.8.1.25@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401787 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:21:45 2019][1623403.574184] Lustre: 15278:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 95770 previous similar messages [Tue Feb 5 13:21:45 2019][1623403.622743] LNetError: 9182:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 102294 previous similar messages [Tue Feb 5 13:21:46 2019][1623404.574202] LNetError: 15278:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.25@o2ib6 from [Tue Feb 5 13:21:46 2019][1623404.574206] Lustre: 9182:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549401744/real 1549401744] req@ffff912929a28300 x1623679301214896/t0(0) o106->fir-MDT0000@10.8.1.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401788 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:21:46 2019][1623404.574209] Lustre: 9182:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 198592 previous similar messages [Tue Feb 5 13:21:46 2019][1623404.622740] LNetError: 15278:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 198592 previous similar messages [Tue Feb 5 13:21:48 2019][1623406.574250] LNetError: 15278:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.25@o2ib6 from [Tue Feb 5 13:21:48 2019][1623406.574253] Lustre: 9182:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549401746/real 1549401746] req@ffff912929a28300 x1623679301214896/t0(0) o106->fir-MDT0000@10.8.1.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401790 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:21:48 2019][1623406.574257] Lustre: 9182:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 409704 previous similar messages [Tue Feb 5 13:21:48 2019][1623406.622804] LNetError: 15278:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 409704 previous similar messages [Tue Feb 5 13:21:52 2019][1623410.574353] LNetError: 15278:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.25@o2ib6 from [Tue Feb 5 13:21:52 2019][1623410.574358] Lustre: 9182:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549401750/real 1549401750] req@ffff912929a28300 x1623679301214896/t0(0) o106->fir-MDT0000@10.8.1.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401794 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:21:52 2019][1623410.574362] Lustre: 9182:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 825830 previous similar messages [Tue Feb 5 13:21:52 2019][1623410.622905] LNetError: 15278:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 825830 previous similar messages [Tue Feb 5 13:22:00 2019][1623418.574556] Lustre: 9182:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549401758/real 1549401758] req@ffff912929a28300 x1623679301214896/t0(0) o106->fir-MDT0000@10.8.1.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401802 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:22:00 2019][1623418.574558] LNetError: 15278:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.25@o2ib6 from [Tue Feb 5 13:22:00 2019][1623418.574562] LNetError: 15278:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 1667393 previous similar messages [Tue Feb 5 13:22:00 2019][1623418.623975] Lustre: 9182:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1674291 previous similar messages [Tue Feb 5 13:22:16 2019][1623434.574957] Lustre: 15278:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549401774/real 1549401774] req@ffff911d391ec200 x1623679301214928/t0(0) o106->fir-MDT0000@10.8.1.25@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401818 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:22:16 2019][1623434.574959] LNetError: 9182:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.29@o2ib6 from [Tue Feb 5 13:22:16 2019][1623434.574964] LNetError: 9182:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 3403702 previous similar messages [Tue Feb 5 13:22:16 2019][1623434.624289] Lustre: 15278:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3402247 previous similar messages [Tue Feb 5 13:22:48 2019][1623466.575757] LNetError: 15278:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.25@o2ib6 from [Tue Feb 5 13:22:48 2019][1623466.575763] Lustre: 9182:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549401806/real 1549401806] req@ffff912929a28300 x1623679301214896/t0(0) o106->fir-MDT0000@10.8.1.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401850 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:22:48 2019][1623466.575767] Lustre: 9182:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6803608 previous similar messages [Tue Feb 5 13:22:48 2019][1623466.624393] LNetError: 15278:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 6811956 previous similar messages [Tue Feb 5 13:23:52 2019][1623530.577363] Lustre: 9182:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549401870/real 1549401870] req@ffff912929a28300 x1623679301214896/t0(0) o106->fir-MDT0000@10.8.1.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549401914 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Tue Feb 5 13:23:52 2019][1623530.577372] LNetError: 15278:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.8.1.25@o2ib6 from [Tue Feb 5 13:23:52 2019][1623530.577375] LNetError: 15278:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 13514212 previous similar messages [Tue Feb 5 13:23:52 2019][1623530.626848] Lustre: 9182:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 13521115 previous similar messages [Tue Feb 5 13:24:21 2019][1623559.535149] Lustre: fir-MDT0000: haven't heard from client 9daa8e10-c154-ca96-d85d-3f6355b9ec3f (at 10.8.2.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff913909b10800, cur 1549401899 expire 1549401749 last 1549401672 [Tue Feb 5 13:24:21 2019][1623559.556853] Lustre: Skipped 2 previous similar messages [Tue Feb 5 13:24:24 2019][1623562.603714] Lustre: MGS: haven't heard from client 16f622df-9b71-a16e-7fd5-e6f7aaabcc11 (at 10.8.2.5@o2ib6) in 230 seconds. I think it's dead, and I am evicting it. exp ffff9128dba93800, cur 1549401902 expire 1549401752 last 1549401672 [Tue Feb 5 13:24:24 2019][1623562.624725] Lustre: Skipped 43 previous similar messages [Tue Feb 5 13:25:36 2019][1623634.472649] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 13:25:36 2019][1623634.484383] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Feb 5 13:26:36 2019][1623694.426505] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.211@o2ib7: 0 seconds [Tue Feb 5 13:26:36 2019][1623694.436774] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 297 previous similar messages [Tue Feb 5 13:31:37 2019][1623995.434102] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.203@o2ib7: 0 seconds [Tue Feb 5 13:31:37 2019][1623995.444363] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 7 previous similar messages [Tue Feb 5 13:36:37 2019][1624295.441623] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.211@o2ib7: 0 seconds [Tue Feb 5 13:36:37 2019][1624295.451879] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 7 previous similar messages [Tue Feb 5 13:43:02 2019][1624680.909106] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 13:43:02 2019][1624680.920839] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Feb 5 13:43:09 2019][1624687.787158] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 13:44:43 2019][1624781.498903] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Feb 5 13:44:43 2019][1624781.510639] Lustre: 6168:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Feb 5 13:56:45 2019][1625503.471935] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.201@o2ib7: 1 seconds [Tue Feb 5 13:56:45 2019][1625503.482192] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 7 previous similar messages [Tue Feb 5 14:00:36 2019][1625734.634241] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Feb 5 14:00:36 2019][1625734.645978] Lustre: 8985:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Feb 5 14:00:44 2019][1625742.342333] Lustre: MGS: Connection restored to 88130480-2e5c-e317-8933-a7c5aa30a9a9 (at 10.9.0.2@o2ib4) [Tue Feb 5 14:00:44 2019][1625742.351990] Lustre: Skipped 4 previous similar messages [Tue Feb 5 14:01:09 2019][1625767.683801] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.0.1@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Tue Feb 5 14:01:09 2019][1625767.701083] LustreError: Skipped 1 previous similar message [Tue Feb 5 14:01:34 2019][1625792.754883] Lustre: fir-MDT0000: Connection restored to 6caeea3e-5b74-9a8d-d8ac-3bf9f06c410e (at 10.9.0.1@o2ib4) [Tue Feb 5 14:01:34 2019][1625792.765241] Lustre: Skipped 1 previous similar message [Tue Feb 5 14:02:24 2019][1625842.872477] Lustre: MGS: Connection restored to 6caeea3e-5b74-9a8d-d8ac-3bf9f06c410e (at 10.9.0.1@o2ib4) [Tue Feb 5 14:24:36 2019][1627174.728055] Lustre: MGS: Connection restored to 1d321062-c34c-dd01-05fd-e0641a0c44bf (at 10.9.101.72@o2ib4) [Tue Feb 5 14:24:37 2019][1627175.396812] Lustre: MGS: Connection restored to a9d6cfb1-3bc0-d83b-b1ac-1e5d9756c3cd (at 10.9.101.40@o2ib4) [Tue Feb 5 14:24:37 2019][1627175.406726] Lustre: Skipped 3 previous similar messages [Tue Feb 5 14:24:38 2019][1627176.416061] Lustre: fir-MDT0000: Connection restored to 6766a8ef-62c5-8b34-b10f-17cc39a12b2e (at 10.9.101.38@o2ib4) [Tue Feb 5 14:24:38 2019][1627176.426674] Lustre: Skipped 16 previous similar messages [Tue Feb 5 14:24:40 2019][1627178.589465] Lustre: MGS: Connection restored to fee33f29-4e44-b25b-c350-4e102ce2b9ce (at 10.9.101.2@o2ib4) [Tue Feb 5 14:24:40 2019][1627178.599297] Lustre: Skipped 45 previous similar messages [Tue Feb 5 14:24:44 2019][1627182.720790] Lustre: MGS: Connection restored to 2788af89-b37a-a98b-a1f8-e3beabbac70d (at 10.9.101.22@o2ib4) [Tue Feb 5 14:24:44 2019][1627182.730707] Lustre: Skipped 16 previous similar messages [Tue Feb 5 14:53:03 2019][1628881.866799] Lustre: MGS: Connection restored to 075b4687-a8f3-141b-e94e-ec1a373e1d5d (at 10.8.1.12@o2ib6) [Tue Feb 5 14:53:03 2019][1628881.876537] Lustre: Skipped 2 previous similar messages [Tue Feb 5 14:53:06 2019][1628884.727135] Lustre: MGS: Connection restored to 8d8952a9-6a5e-9492-eedc-0f2b6318be8c (at 10.8.1.34@o2ib6) [Tue Feb 5 14:53:06 2019][1628884.736871] Lustre: Skipped 2 previous similar messages [Tue Feb 5 14:53:34 2019][1628913.119288] Lustre: MGS: Connection restored to ef7b1236-588f-a3d3-857e-6157f71b8619 (at 10.8.1.30@o2ib6) [Tue Feb 5 14:53:34 2019][1628913.129027] Lustre: Skipped 2 previous similar messages [Tue Feb 5 14:54:22 2019][1628960.751958] Lustre: MGS: Connection restored to 8d2ed7d3-f294-026b-dfd6-2dd338af0435 (at 10.8.1.27@o2ib6) [Tue Feb 5 14:54:22 2019][1628960.761695] Lustre: Skipped 2 previous similar messages [Tue Feb 5 14:55:05 2019][1629003.794476] Lustre: MGS: Connection restored to 9d46bed9-b861-e3c5-2e5a-00857072bb88 (at 10.8.1.3@o2ib6) [Tue Feb 5 14:55:05 2019][1629003.804127] Lustre: Skipped 5 previous similar messages [Tue Feb 5 14:55:24 2019][1629022.542449] Lustre: MGS: Connection restored to 845403b3-652f-cdc1-e252-3c0dda41beee (at 10.8.1.28@o2ib6) [Tue Feb 5 14:55:24 2019][1629022.552192] Lustre: Skipped 2 previous similar messages [Tue Feb 5 14:56:16 2019][1629075.008841] Lustre: MGS: Connection restored to 39b927b3-e170-edeb-fe6b-14133c18cc93 (at 10.8.1.11@o2ib6) [Tue Feb 5 14:56:16 2019][1629075.018585] Lustre: Skipped 5 previous similar messages [Tue Feb 5 14:57:24 2019][1629142.946259] Lustre: MGS: Connection restored to 89ef5239-20b5-bc7b-c518-4ca992c279c2 (at 10.8.1.9@o2ib6) [Tue Feb 5 14:57:24 2019][1629142.955913] Lustre: Skipped 23 previous similar messages [Tue Feb 5 14:59:35 2019][1629274.031840] Lustre: MGS: Connection restored to f70b3128-22d8-68cd-6dcf-bbf43cdf1204 (at 10.8.1.21@o2ib6) [Tue Feb 5 14:59:35 2019][1629274.041585] Lustre: Skipped 56 previous similar messages [Tue Feb 5 15:14:17 2019][1630155.433032] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Feb 5 15:14:17 2019][1630155.444776] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Feb 5 15:15:12 2019][1630211.091803] Lustre: MGS: Connection restored to bf5430c3-da5b-8e92-fc51-db19caf54c5f (at 10.8.2.8@o2ib6) [Tue Feb 5 15:15:12 2019][1630211.101463] Lustre: Skipped 20 previous similar messages [Tue Feb 5 15:15:45 2019][1630243.950071] Lustre: MGS: Connection restored to 8c3de5c6-e855-3331-f3cd-f8cb5efde160 (at 10.9.102.13@o2ib4) [Tue Feb 5 15:15:45 2019][1630243.960001] Lustre: Skipped 26 previous similar messages [Tue Feb 5 15:16:00 2019][1630258.933632] Lustre: DEBUG MARKER: Tue Feb 5 15:16:38 2019 [Tue Feb 5 15:16:00 2019][1630258.933632] [Tue Feb 5 15:16:50 2019][1630309.182948] Lustre: MGS: Connection restored to c0acc1c3-0acf-a446-06df-b0e137379408 (at 10.9.102.33@o2ib4) [Tue Feb 5 15:16:50 2019][1630309.192869] Lustre: Skipped 11 previous similar messages [Tue Feb 5 15:18:21 2019][1630400.170423] Lustre: 15386:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Feb 5 15:19:18 2019][1630456.796359] Lustre: MGS: Connection restored to ff28cdc3-9cd5-780d-2524-0d7f7b2ab40a (at 10.9.102.71@o2ib4) [Tue Feb 5 15:19:18 2019][1630456.806278] Lustre: Skipped 29 previous similar messages [Tue Feb 5 15:22:49 2019][1630667.755518] Lustre: 8929:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Feb 5 15:23:28 2019][1630707.036952] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Feb 5 15:23:28 2019][1630707.048694] Lustre: 6616:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Feb 5 15:23:34 2019][1630713.117425] Lustre: MGS: Connection restored to d834456b-e128-9430-6eb9-e0a4fdc5d63c (at 10.9.102.6@o2ib4) [Tue Feb 5 15:23:34 2019][1630713.127255] Lustre: Skipped 188 previous similar messages [Tue Feb 5 15:24:05 2019][1630744.049690] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Feb 5 15:24:05 2019][1630744.061439] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Feb 5 15:25:03 2019][1630801.558530] Lustre: 15261:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Feb 5 15:25:03 2019][1630801.570392] Lustre: 15261:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Feb 5 15:27:35 2019][1630953.762071] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 15:27:35 2019][1630953.773804] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Tue Feb 5 15:29:41 2019][1631079.653237] Lustre: 15169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 15:29:41 2019][1631079.665065] Lustre: 15169:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Tue Feb 5 15:31:09 2019][1631167.344817] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 15:31:09 2019][1631167.356633] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages [Tue Feb 5 15:32:17 2019][1631236.118296] Lustre: MGS: Connection restored to ef699483-aaaf-8929-c882-5a01d472632d (at 10.9.101.12@o2ib4) [Tue Feb 5 15:32:17 2019][1631236.128218] Lustre: Skipped 56 previous similar messages [Tue Feb 5 15:34:45 2019][1631383.837508] Lustre: 15278:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Feb 5 15:34:45 2019][1631383.849344] Lustre: 15278:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages [Tue Feb 5 15:40:14 2019][1631713.245340] Lustre: 15169:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 15:40:14 2019][1631713.257178] Lustre: 15169:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Tue Feb 5 15:44:00 2019][1631938.841968] Lustre: MGS: Connection restored to b05292a0-3b20-282a-37f5-f4d5b16cd3f5 (at 10.9.102.56@o2ib4) [Tue Feb 5 15:44:00 2019][1631938.851894] Lustre: Skipped 98 previous similar messages [Tue Feb 5 15:50:45 2019][1632343.757786] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 15:50:45 2019][1632343.769599] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Feb 5 15:54:16 2019][1632555.111283] Lustre: MGS: Connection restored to 4c0a6847-7a14-1c98-ab37-7777b07c3f81 (at 10.9.0.64@o2ib4) [Tue Feb 5 15:54:16 2019][1632555.121031] Lustre: Skipped 5 previous similar messages [Tue Feb 5 16:00:50 2019][1632948.750675] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Tue Feb 5 16:00:50 2019][1632948.762417] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 48 previous similar messages [Tue Feb 5 16:09:57 2019][1633496.081184] Lustre: MGS: Connection restored to 0f228072-b31a-f3c8-f62d-47ff97ac7259 (at 10.9.0.63@o2ib4) [Tue Feb 5 16:09:57 2019][1633496.090936] Lustre: Skipped 7 previous similar messages [Tue Feb 5 16:11:17 2019][1633575.514653] Lustre: DEBUG MARKER: Tue Feb 5 16:11:55 2019 [Tue Feb 5 16:11:17 2019][1633575.514653] [Tue Feb 5 16:12:13 2019][1633632.246617] Lustre: 15407:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 16:12:13 2019][1633632.258593] Lustre: 15407:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Tue Feb 5 16:22:19 2019][1634238.167774] Lustre: 15391:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 16:22:19 2019][1634238.179603] Lustre: 15391:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages [Tue Feb 5 16:22:21 2019][1634239.888749] Lustre: MGS: Connection restored to 030cc203-3f8d-56fb-8dea-10d7c076053d (at 10.8.11.22@o2ib6) [Tue Feb 5 16:22:21 2019][1634239.898586] Lustre: Skipped 8 previous similar messages [Tue Feb 5 16:32:40 2019][1634859.111459] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 16:32:40 2019][1634859.123202] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 22 previous similar messages [Tue Feb 5 16:33:53 2019][1634931.595716] Lustre: MGS: Connection restored to cab31c76-ca03-11b2-b093-0105ff7d87b5 (at 10.9.104.3@o2ib4) [Tue Feb 5 16:33:53 2019][1634931.605550] Lustre: Skipped 2 previous similar messages [Tue Feb 5 16:35:08 2019][1635007.174985] Lustre: DEBUG MARKER: Tue Feb 5 16:35:46 2019 [Tue Feb 5 16:35:08 2019][1635007.174985] [Tue Feb 5 16:35:11 2019][1635009.528549] Lustre: fir-MDT0002: haven't heard from client 61d0fcbf-b5c3-a78f-9cdf-55e61d025648 (at 10.9.0.81@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91359b2ab800, cur 1549413349 expire 1549413199 last 1549413122 [Tue Feb 5 16:35:11 2019][1635009.550342] Lustre: Skipped 21 previous similar messages [Tue Feb 5 16:42:53 2019][1635472.388274] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 16:42:53 2019][1635472.400022] Lustre: 6167:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 61 previous similar messages [Tue Feb 5 16:46:05 2019][1635663.526167] Lustre: MGS: Connection restored to 366ef3ed-06bb-647a-d8db-99cc53f0dcc3 (at 10.9.104.8@o2ib4) [Tue Feb 5 16:46:05 2019][1635663.536024] Lustre: Skipped 5 previous similar messages [Tue Feb 5 16:53:16 2019][1636095.478393] Lustre: 15221:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 16:53:17 2019][1636095.490222] Lustre: 15221:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 53 previous similar messages [Tue Feb 5 17:00:23 2019][1636521.895528] Lustre: MGS: Connection restored to 9a3fec86-6c87-9861-a86b-61fc0ee356f3 (at 10.8.11.9@o2ib6) [Tue Feb 5 17:00:23 2019][1636521.905284] Lustre: Skipped 8 previous similar messages [Tue Feb 5 17:04:07 2019][1636746.016836] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 17:04:07 2019][1636746.028572] Lustre: 8995:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 40 previous similar messages [Tue Feb 5 17:07:58 2019][1636976.576803] Lustre: fir-MDT0002: haven't heard from client 5475bc15-b344-a614-5762-811f76d6d107 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91205e3a5000, cur 1549415316 expire 1549415166 last 1549415089 [Tue Feb 5 17:07:58 2019][1636976.598599] Lustre: Skipped 2 previous similar messages [Tue Feb 5 17:14:08 2019][1637346.967833] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 17:14:08 2019][1637346.979573] Lustre: 6883:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 49 previous similar messages [Tue Feb 5 17:19:55 2019][1637693.938801] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.9.0.1@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff9140756c0b40/0x21ed9350c5f8b7c4 lrc: 3/0,0 mode: PR/PR res: [0x2c00014f4:0x19a50:0x0].0x0 bits 0x58/0x0 rrc: 4 type: IBT flags: 0x60200400010020 nid: 10.9.0.1@o2ib4 remote: 0xc4e69a80f22a73e1 expref: 779386 pid: 17220 timeout: 1637652 lvb_type: 0 [Tue Feb 5 17:19:55 2019][1637693.979078] LustreError: 20043:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.9.0.1@o2ib4 arrived at 1549416033 with bad export cookie 2444772143427161870 [Tue Feb 5 17:19:55 2019][1637693.994541] LustreError: 20043:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 5 previous similar messages [Tue Feb 5 17:19:55 2019][1637693.997211] Lustre: fir-MDT0002: Connection restored to 6caeea3e-5b74-9a8d-d8ac-3bf9f06c410e (at 10.9.0.1@o2ib4) [Tue Feb 5 17:19:55 2019][1637693.997214] Lustre: Skipped 2 previous similar messages [Tue Feb 5 17:20:00 2019][1637698.938925] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.9.0.1@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffff9159a724f740/0x21ed9350c625c7e0 lrc: 3/0,0 mode: PR/PR res: [0x200001750:0x12454:0x0].0x0 bits 0x5b/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.9.0.1@o2ib4 remote: 0xc4e69a80f22a7404 expref: 1242060 pid: 6951 timeout: 1637657 lvb_type: 0 [Tue Feb 5 17:20:00 2019][1637698.977951] LustreError: 6769:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.9.0.1@o2ib4 arrived at 1549416038 with bad export cookie 2444772143427161863 [Tue Feb 5 17:20:00 2019][1637698.993329] LustreError: 6769:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 8 previous similar messages [Tue Feb 5 17:21:24 2019][1637783.476038] LNet: Service thread pid 17220 was inactive for 200.72s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Tue Feb 5 17:21:24 2019][1637783.493150] Pid: 17220, comm: mdt03_025 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Tue Feb 5 17:21:24 2019][1637783.503062] Call Trace: [Tue Feb 5 17:21:24 2019][1637783.505704] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Tue Feb 5 17:21:24 2019][1637783.512818] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Tue Feb 5 17:21:24 2019][1637783.520185] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Tue Feb 5 17:21:24 2019][1637783.527111] [] mdt_reint_unlink+0x331/0x14a0 [mdt] [Tue Feb 5 17:21:25 2019][1637783.533767] [] mdt_reint_rec+0x83/0x210 [mdt] [Tue Feb 5 17:21:25 2019][1637783.539988] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Tue Feb 5 17:21:25 2019][1637783.546732] [] mdt_reint+0x67/0x140 [mdt] [Tue Feb 5 17:21:25 2019][1637783.552609] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Tue Feb 5 17:21:25 2019][1637783.559733] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Feb 5 17:21:25 2019][1637783.567629] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Tue Feb 5 17:21:25 2019][1637783.574137] [] kthread+0xd1/0xe0 [Tue Feb 5 17:21:25 2019][1637783.579224] [] ret_from_fork_nospec_begin+0xe/0x21 [Tue Feb 5 17:21:25 2019][1637783.585870] [] 0xffffffffffffffff [Tue Feb 5 17:21:25 2019][1637783.591067] LustreError: dumping log to /tmp/lustre-log.1549416122.17220 [Tue Feb 5 17:21:28 2019][1637787.293866] Lustre: 17220:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (118:86s); client may timeout. req@ffff914b3ebbaa00 x1624565563654256/t13578466217(0) o36->673a177f-1230-a616-2a0b-8995b4882735@10.9.0.1@o2ib4:510/0 lens 536/424 e 3 to 0 dl 1549416040 ref 1 fl Complete:/0/0 rc 0/0 [Tue Feb 5 17:21:28 2019][1637787.323115] Lustre: 17220:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 3 previous similar messages [Tue Feb 5 17:21:28 2019][1637787.333565] LNet: Service thread pid 17220 completed after 204.58s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Tue Feb 5 17:21:41 2019][1637799.860451] LNet: Service thread pid 6168 was inactive for 200.29s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Tue Feb 5 17:21:41 2019][1637799.877471] Pid: 6168, comm: mdt00_001 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Tue Feb 5 17:21:41 2019][1637799.887297] Call Trace: [Tue Feb 5 17:21:41 2019][1637799.889939] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Tue Feb 5 17:21:41 2019][1637799.897070] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Tue Feb 5 17:21:41 2019][1637799.904442] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Tue Feb 5 17:21:41 2019][1637799.911448] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Tue Feb 5 17:21:41 2019][1637799.918625] [] mdt_reint_object_lock+0x2c/0x60 [mdt] [Tue Feb 5 17:21:41 2019][1637799.925454] [] mdt_reint_striped_lock+0x8c/0x510 [mdt] [Tue Feb 5 17:21:41 2019][1637799.932457] [] mdt_reint_unlink+0x6b2/0x14a0 [mdt] [Tue Feb 5 17:21:41 2019][1637799.939144] [] mdt_reint_rec+0x83/0x210 [mdt] [Tue Feb 5 17:21:41 2019][1637799.945369] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Tue Feb 5 17:21:41 2019][1637799.952112] [] mdt_reint+0x67/0x140 [mdt] [Tue Feb 5 17:21:41 2019][1637799.957997] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Tue Feb 5 17:21:41 2019][1637799.965135] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Feb 5 17:21:41 2019][1637799.973027] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Tue Feb 5 17:21:41 2019][1637799.979526] [] kthread+0xd1/0xe0 [Tue Feb 5 17:21:41 2019][1637799.984613] [] ret_from_fork_nospec_begin+0xe/0x21 [Tue Feb 5 17:21:41 2019][1637799.991261] [] 0xffffffffffffffff [Tue Feb 5 17:21:41 2019][1637799.996456] LustreError: dumping log to /tmp/lustre-log.1549416139.6168 [Tue Feb 5 17:21:47 2019][1637805.783661] Lustre: MGS: haven't heard from client f90ae9a7-5e11-d6a2-66fe-1781433dc2ea (at 10.9.0.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91575eaff000, cur 1549416145 expire 1549415995 last 1549415918 [Tue Feb 5 17:21:47 2019][1637805.804663] Lustre: Skipped 2 previous similar messages [Tue Feb 5 17:22:43 2019][1637861.804675] Lustre: 6168:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (106:157s); client may timeout. req@ffff91285a10f800 x1624565611102400/t14332766862(0) o36->673a177f-1230-a616-2a0b-8995b4882735@10.9.0.1@o2ib4:514/0 lens 488/424 e 2 to 0 dl 1549416044 ref 1 fl Complete:/0/0 rc 0/0 [Tue Feb 5 17:22:43 2019][1637861.834112] LNet: Service thread pid 6168 completed after 262.26s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Tue Feb 5 17:23:42 2019][1637920.650404] Lustre: fir-MDT0002: haven't heard from client 673a177f-1230-a616-2a0b-8995b4882735 (at 10.9.0.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff913860e53c00, cur 1549416260 expire 1549416110 last 1549416033 [Tue Feb 5 17:23:47 2019][1637925.600659] Lustre: fir-MDT0000: haven't heard from client 673a177f-1230-a616-2a0b-8995b4882735 (at 10.9.0.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91447eeae800, cur 1549416265 expire 1549416115 last 1549416038 [Tue Feb 5 17:23:59 2019][1637937.614697] LustreError: 15260:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9139a2df8600 x1623680419429696/t0(0) o104->fir-MDT0000@10.9.0.1@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Tue Feb 5 17:24:06 2019][1637945.162784] LustreError: 7415:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff913cce3a1e00 x1623680420078384/t0(0) o104->fir-MDT0000@10.9.0.1@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Tue Feb 5 17:24:08 2019][1637946.978282] LustreError: 6167:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff912869cd4500 x1623680420249760/t0(0) o104->fir-MDT0000@10.9.0.1@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Tue Feb 5 17:24:13 2019][1637951.658181] LustreError: 9014:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff912bc4415a00 x1623680420651712/t0(0) o104->fir-MDT0000@10.9.0.1@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Tue Feb 5 17:24:20 2019][1637959.251225] LustreError: 15329:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9143f8c79500 x1623680421282912/t0(0) o104->fir-MDT0000@10.9.0.1@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Tue Feb 5 17:24:20 2019][1637959.272236] LustreError: 15329:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [Tue Feb 5 17:24:30 2019][1637968.727525] LustreError: 6173:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9144030d6300 x1623680422151168/t0(0) o104->fir-MDT0000@10.9.0.1@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Tue Feb 5 17:24:30 2019][1637968.748465] LustreError: 6173:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1 previous similar message [Tue Feb 5 17:25:38 2019][1638036.948408] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.9.0.1@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffff913b6db82400/0x21ed9350b95e18f2 lrc: 3/0,0 mode: PW/PW res: [0x2000016fe:0x12cab:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.9.0.1@o2ib4 remote: 0xc4e69a80ef16d7d3 expref: 341588 pid: 15509 timeout: 1637995 lvb_type: 0 [Tue Feb 5 17:25:46 2019][1638044.948604] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.9.0.1@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffff915357af6e40/0x21ed9350b95e4e2f lrc: 3/0,0 mode: PW/PW res: [0x2000016fe:0x12cac:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.9.0.1@o2ib4 remote: 0xc4e69a80ef16de63 expref: 330950 pid: 6682 timeout: 1638003 lvb_type: 0 [Tue Feb 5 17:25:54 2019][1638052.948809] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.9.0.1@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffff915218a69b00/0x21ed9350b99a7f5f lrc: 3/0,0 mode: PW/PW res: [0x2000016fe:0x12cb3:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.9.0.1@o2ib4 remote: 0xc4e69a80ef24461c expref: 320436 pid: 6951 timeout: 1638011 lvb_type: 0 [Tue Feb 5 17:25:54 2019][1638052.986723] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [Tue Feb 5 17:25:55 2019][1638054.438937] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 17:25:55 2019][1638054.450668] Lustre: 9182:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 24 previous similar messages [Tue Feb 5 17:27:19 2019][1638137.788931] LNet: Service thread pid 15260 was inactive for 200.16s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Tue Feb 5 17:27:19 2019][1638137.806043] Pid: 15260, comm: mdt01_061 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Tue Feb 5 17:27:19 2019][1638137.815977] Call Trace: [Tue Feb 5 17:27:19 2019][1638137.818616] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Tue Feb 5 17:27:19 2019][1638137.825737] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Tue Feb 5 17:27:19 2019][1638137.833104] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Tue Feb 5 17:27:19 2019][1638137.840107] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Tue Feb 5 17:27:19 2019][1638137.847298] [] mdt_object_lock+0x20/0x30 [mdt] [Tue Feb 5 17:27:19 2019][1638137.853612] [] mdt_brw_enqueue+0x44b/0x760 [mdt] [Tue Feb 5 17:27:19 2019][1638137.860122] [] mdt_intent_brw+0x1f/0x30 [mdt] [Tue Feb 5 17:27:19 2019][1638137.866354] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Tue Feb 5 17:27:19 2019][1638137.873057] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Tue Feb 5 17:27:19 2019][1638137.879992] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Tue Feb 5 17:27:19 2019][1638137.887274] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Tue Feb 5 17:27:19 2019][1638137.893617] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Tue Feb 5 17:27:19 2019][1638137.900733] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Feb 5 17:27:19 2019][1638137.908628] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Tue Feb 5 17:27:19 2019][1638137.915132] [] kthread+0xd1/0xe0 [Tue Feb 5 17:27:19 2019][1638137.920225] [] ret_from_fork_nospec_begin+0xe/0x21 [Tue Feb 5 17:27:19 2019][1638137.926890] [] 0xffffffffffffffff [Tue Feb 5 17:27:19 2019][1638137.932096] LustreError: dumping log to /tmp/lustre-log.1549416477.15260 [Tue Feb 5 17:27:40 2019][1638159.293469] LNet: Service thread pid 15329 was inactive for 200.03s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Tue Feb 5 17:27:40 2019][1638159.310582] Pid: 15329, comm: mdt02_041 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Tue Feb 5 17:27:40 2019][1638159.320495] Call Trace: [Tue Feb 5 17:27:40 2019][1638159.323143] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Tue Feb 5 17:27:40 2019][1638159.330275] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Tue Feb 5 17:27:40 2019][1638159.337674] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Tue Feb 5 17:27:40 2019][1638159.344698] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Tue Feb 5 17:27:40 2019][1638159.351892] [] mdt_object_lock+0x20/0x30 [mdt] [Tue Feb 5 17:27:40 2019][1638159.358209] [] mdt_brw_enqueue+0x44b/0x760 [mdt] [Tue Feb 5 17:27:40 2019][1638159.364716] [] mdt_intent_brw+0x1f/0x30 [mdt] [Tue Feb 5 17:27:40 2019][1638159.370941] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Tue Feb 5 17:27:40 2019][1638159.377612] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Tue Feb 5 17:27:40 2019][1638159.384556] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Tue Feb 5 17:27:40 2019][1638159.391853] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Tue Feb 5 17:27:40 2019][1638159.398207] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Tue Feb 5 17:27:40 2019][1638159.405337] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Feb 5 17:27:40 2019][1638159.413236] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Tue Feb 5 17:27:40 2019][1638159.419751] [] kthread+0xd1/0xe0 [Tue Feb 5 17:27:40 2019][1638159.424841] [] ret_from_fork_nospec_begin+0xe/0x21 [Tue Feb 5 17:27:40 2019][1638159.431504] [] 0xffffffffffffffff [Tue Feb 5 17:27:40 2019][1638159.436699] LustreError: dumping log to /tmp/lustre-log.1549416498.15329 [Tue Feb 5 17:27:59 2019][1638177.648946] Lustre: MGS: haven't heard from client f90ae9a7-5e11-d6a2-66fe-1781433dc2ea (at 10.9.0.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff914426d92400, cur 1549416517 expire 1549416367 last 1549416290 [Tue Feb 5 17:28:07 2019][1638186.020649] LNet: Service thread pid 15329 completed after 226.76s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Tue Feb 5 17:28:59 2019][1638237.642440] LustreError: 15260:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549416277, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9135255a1b00/0x21ed9350c967b626 lrc: 3/1,0 mode: --/PR res: [0x2000016fe:0x12cab:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 15260 timeout: 0 lvb_type: 0 [Tue Feb 5 17:28:59 2019][1638237.682259] LustreError: dumping log to /tmp/lustre-log.1549416577.15260 [Tue Feb 5 17:29:08 2019][1638247.006678] LustreError: 6167:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549416286, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9124c7299200/0x21ed9350c97cb523 lrc: 3/1,0 mode: --/PR res: [0x2000016fe:0x12cad:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 6167 timeout: 0 lvb_type: 0 [Tue Feb 5 17:30:18 2019][1638316.855330] LNet: Service thread pid 15260 completed after 379.23s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Tue Feb 5 17:31:06 2019][1638365.005037] Lustre: MGS: Connection restored to 58c319e2-128b-6d7f-bc22-e6642c995c3b (at 10.8.0.66@o2ib6) [Tue Feb 5 17:31:06 2019][1638365.014782] Lustre: Skipped 2 previous similar messages [Tue Feb 5 17:31:39 2019][1638398.210599] Lustre: DEBUG MARKER: Tue Feb 5 17:32:17 2019 [Tue Feb 5 17:31:39 2019][1638398.210599] [Tue Feb 5 17:38:11 2019][1638789.823556] Lustre: 15407:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 17:38:11 2019][1638789.835381] Lustre: 15407:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 20 previous similar messages [Tue Feb 5 17:40:36 2019][1638934.568934] Lustre: 48636:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549417266/real 1549417266] req@ffff914fed271500 x1623680644395568/t0(0) o104->fir-MDT0000@10.9.0.2@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1549417273 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Tue Feb 5 17:40:36 2019][1638934.596200] Lustre: 48636:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6107702 previous similar messages [Tue Feb 5 17:40:57 2019][1638955.607455] Lustre: 48636:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549417287/real 1549417287] req@ffff914fed271500 x1623680644395568/t0(0) o104->fir-MDT0000@10.9.0.2@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1549417294 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Tue Feb 5 17:40:57 2019][1638955.634719] Lustre: 48636:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [Tue Feb 5 17:41:32 2019][1638990.647344] Lustre: 48636:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549417323/real 1549417323] req@ffff914fed271500 x1623680644395568/t0(0) o104->fir-MDT0000@10.9.0.2@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1549417330 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Tue Feb 5 17:41:32 2019][1638990.674632] Lustre: 48636:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [Tue Feb 5 17:42:08 2019][1639026.972249] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.9.0.2@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffff912225f84ec0/0x21ed9350e3813420 lrc: 3/0,0 mode: PR/PR res: [0x2000014a5:0xbada:0x0].0x0 bits 0x58/0x0 rrc: 5 type: IBT flags: 0x60200400010020 nid: 10.9.0.2@o2ib4 remote: 0x5d2fc5668c2abd00 expref: 1975718 pid: 15228 timeout: 1638985 lvb_type: 0 [Tue Feb 5 17:42:08 2019][1639027.010293] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages [Tue Feb 5 17:42:26 2019][1639045.109654] LustreError: 17203:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff91507f3f8600 x1623680665106944/t0(0) o104->fir-MDT0000@10.9.0.2@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Tue Feb 5 17:43:49 2019][1639128.021776] LNet: Service thread pid 6845 was inactive for 200.44s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Tue Feb 5 17:43:49 2019][1639128.038799] Pid: 6845, comm: mdt03_008 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Tue Feb 5 17:43:49 2019][1639128.048628] Call Trace: [Tue Feb 5 17:43:49 2019][1639128.051271] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Tue Feb 5 17:43:49 2019][1639128.058393] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Tue Feb 5 17:43:49 2019][1639128.065760] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Tue Feb 5 17:43:49 2019][1639128.072762] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Tue Feb 5 17:43:49 2019][1639128.079937] [] mdt_reint_object_lock+0x2c/0x60 [mdt] [Tue Feb 5 17:43:49 2019][1639128.086772] [] mdt_reint_unlink+0x1e7/0x14a0 [mdt] [Tue Feb 5 17:43:49 2019][1639128.093435] [] mdt_reint_rec+0x83/0x210 [mdt] [Tue Feb 5 17:43:49 2019][1639128.099656] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Tue Feb 5 17:43:49 2019][1639128.106397] [] mdt_reint+0x67/0x140 [mdt] [Tue Feb 5 17:43:49 2019][1639128.112275] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Tue Feb 5 17:43:49 2019][1639128.119398] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Feb 5 17:43:49 2019][1639128.127302] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Tue Feb 5 17:43:49 2019][1639128.133812] [] kthread+0xd1/0xe0 [Tue Feb 5 17:43:49 2019][1639128.138917] [] ret_from_fork_nospec_begin+0xe/0x21 [Tue Feb 5 17:43:49 2019][1639128.145567] [] 0xffffffffffffffff [Tue Feb 5 17:43:49 2019][1639128.150796] LustreError: dumping log to /tmp/lustre-log.1549417467.6845 [Tue Feb 5 17:43:49 2019][1639128.158325] Pid: 15389, comm: mdt01_073 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Tue Feb 5 17:43:49 2019][1639128.168242] Call Trace: [Tue Feb 5 17:43:49 2019][1639128.170897] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Tue Feb 5 17:43:49 2019][1639128.178020] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Tue Feb 5 17:43:49 2019][1639128.185389] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Tue Feb 5 17:43:49 2019][1639128.192314] [] mdt_reint_unlink+0x331/0x14a0 [mdt] [Tue Feb 5 17:43:49 2019][1639128.198969] [] mdt_reint_rec+0x83/0x210 [mdt] [Tue Feb 5 17:43:49 2019][1639128.205208] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Tue Feb 5 17:43:49 2019][1639128.211953] [] mdt_reint+0x67/0x140 [mdt] [Tue Feb 5 17:43:49 2019][1639128.217830] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Tue Feb 5 17:43:49 2019][1639128.224952] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Feb 5 17:43:49 2019][1639128.232863] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Tue Feb 5 17:43:49 2019][1639128.239394] [] kthread+0xd1/0xe0 [Tue Feb 5 17:43:49 2019][1639128.244480] [] ret_from_fork_nospec_begin+0xe/0x21 [Tue Feb 5 17:43:49 2019][1639128.251128] [] 0xffffffffffffffff [Tue Feb 5 17:43:51 2019][1639129.630915] Lustre: fir-MDT0002: haven't heard from client db4d27b6-f63b-4d46-2c87-13eadaadf021 (at 10.9.0.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9132194c4000, cur 1549417469 expire 1549417319 last 1549417242 [Tue Feb 5 17:44:06 2019][1639144.975222] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.9.0.2@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffff914bc3ee3f00/0x21ed9350e36a77c3 lrc: 3/0,0 mode: PR/PR res: [0x200001481:0x145ac:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.9.0.2@o2ib4 remote: 0x5d2fc5668c29d5d6 expref: 1316133 pid: 17224 timeout: 1639103 lvb_type: 0 [Tue Feb 5 17:44:32 2019][1639171.352762] Lustre: MGS: Connection restored to 88130480-2e5c-e317-8933-a7c5aa30a9a9 (at 10.9.0.2@o2ib4) [Tue Feb 5 17:44:32 2019][1639171.362417] Lustre: Skipped 8 previous similar messages [Tue Feb 5 17:45:29 2019][1639227.575280] LustreError: 6845:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549417266, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff914b2d7c8d80/0x21ed9350e381745b lrc: 3/0,1 mode: --/PW res: [0x2000014a4:0xc667:0x0].0x9a9626fb bits 0x2/0x0 rrc: 3 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 6845 timeout: 0 lvb_type: 0 [Tue Feb 5 17:45:29 2019][1639227.615274] LustreError: 6845:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Tue Feb 5 17:46:48 2019][1639307.226275] LNet: Service thread pid 17203 was inactive for 262.11s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Tue Feb 5 17:46:48 2019][1639307.243381] LNet: Skipped 1 previous similar message [Tue Feb 5 17:46:48 2019][1639307.248529] Pid: 17203, comm: mdt03_018 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Tue Feb 5 17:46:48 2019][1639307.258442] Call Trace: [Tue Feb 5 17:46:48 2019][1639307.261076] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Tue Feb 5 17:46:48 2019][1639307.268191] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Tue Feb 5 17:46:48 2019][1639307.275557] [] mdt_object_local_lock+0x438/0xb20 [mdt] [Tue Feb 5 17:46:48 2019][1639307.282559] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Tue Feb 5 17:46:48 2019][1639307.289735] [] mdt_reint_object_lock+0x2c/0x60 [mdt] [Tue Feb 5 17:46:48 2019][1639307.296566] [] mdt_reint_unlink+0x1e7/0x14a0 [mdt] [Tue Feb 5 17:46:48 2019][1639307.303221] [] mdt_reint_rec+0x83/0x210 [mdt] [Tue Feb 5 17:46:48 2019][1639307.309448] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Tue Feb 5 17:46:48 2019][1639307.316196] [] mdt_reint+0x67/0x140 [mdt] [Tue Feb 5 17:46:48 2019][1639307.322073] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Tue Feb 5 17:46:48 2019][1639307.329204] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Feb 5 17:46:48 2019][1639307.337102] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Tue Feb 5 17:46:48 2019][1639307.343602] [] kthread+0xd1/0xe0 [Tue Feb 5 17:46:48 2019][1639307.348687] [] ret_from_fork_nospec_begin+0xe/0x21 [Tue Feb 5 17:46:48 2019][1639307.355328] [] 0xffffffffffffffff [Tue Feb 5 17:46:48 2019][1639307.360520] LustreError: dumping log to /tmp/lustre-log.1549417646.17203 [Tue Feb 5 17:47:26 2019][1639345.139231] LustreError: 17203:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549417384, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9152d9761680/0x21ed9350e5c68cfe lrc: 3/0,1 mode: --/CW res: [0x200001481:0x145ac:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 17203 timeout: 0 lvb_type: 0 [Tue Feb 5 17:48:52 2019][1639430.862954] Lustre: 6845:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (106:398s); client may timeout. req@ffff914ee3a7e000 x1624666931150960/t0(0) o36->db4d27b6-f63b-4d46-2c87-13eadaadf021@10.9.0.2@o2ib4:332/0 lens 528/424 e 4 to 0 dl 1549417372 ref 1 fl Complete:/0/0 rc -2/-2 [Tue Feb 5 17:48:52 2019][1639430.863073] LNet: Service thread pid 15389 completed after 503.28s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Tue Feb 5 17:48:52 2019][1639430.873906] LustreError: 6690:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff911d66fbce00 x1623680736994512/t0(0) o104->fir-MDT0000@10.9.0.2@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Tue Feb 5 17:48:56 2019][1639435.168411] LustreError: 6806:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9124fb280900 x1623680737700016/t0(0) o104->fir-MDT0000@10.9.0.2@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Tue Feb 5 17:49:01 2019][1639440.248935] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 17:49:01 2019][1639440.260669] Lustre: 6806:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 29 previous similar messages [Tue Feb 5 17:49:23 2019][1639461.652424] LustreError: 15232:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff91317a535a00 x1623680741586560/t0(0) o104->fir-MDT0000@10.9.0.2@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Tue Feb 5 17:49:23 2019][1639461.673435] LustreError: 15232:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1 previous similar message [Tue Feb 5 17:49:41 2019][1639479.881918] LustreError: 6806:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff911fbf65ef00 x1623680744040880/t0(0) o104->fir-MDT0000@10.9.0.2@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Tue Feb 5 17:49:41 2019][1639479.902874] LustreError: 6806:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [Tue Feb 5 17:50:27 2019][1639526.149943] LustreError: 17052:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9150ab654500 x1623680750516240/t0(0) o104->fir-MDT0000@10.9.0.2@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Tue Feb 5 17:50:27 2019][1639526.170951] LustreError: 17052:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [Tue Feb 5 17:51:14 2019][1639572.811167] LNet: Service thread pid 17203 completed after 527.68s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Tue Feb 5 17:51:14 2019][1639572.827523] LNet: Skipped 1 previous similar message [Tue Feb 5 17:51:46 2019][1639605.537496] LustreError: 8971:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff913760533f00 x1623680763086656/t0(0) o104->fir-MDT0000@10.9.0.2@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Tue Feb 5 17:51:46 2019][1639605.558447] LustreError: 8971:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [Tue Feb 5 17:53:56 2019][1639735.351033] LustreError: 6758:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff915013354800 x1623680784794000/t0(0) o104->fir-MDT0000@10.9.0.2@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Tue Feb 5 17:53:56 2019][1639735.371956] LustreError: 6758:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 13 previous similar messages [Tue Feb 5 17:59:11 2019][1640049.881039] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 17:59:11 2019][1640049.892806] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 78 previous similar messages [Tue Feb 5 18:09:51 2019][1640690.309232] Lustre: 15281:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 18:09:51 2019][1640690.321058] Lustre: 15281:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 25 previous similar messages [Tue Feb 5 18:16:12 2019][1641070.651616] Lustre: MGS: Connection restored to 030cc203-3f8d-56fb-8dea-10d7c076053d (at 10.8.11.22@o2ib6) [Tue Feb 5 18:16:12 2019][1641070.661451] Lustre: Skipped 5 previous similar messages [Tue Feb 5 18:18:02 2019][1641180.682633] Lustre: fir-MDT0002: haven't heard from client a60735d0-fd55-3a91-e138-5b2efd5cb4f9 (at 10.9.104.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff911dd9d1f000, cur 1549419520 expire 1549419370 last 1549419293 [Tue Feb 5 18:18:02 2019][1641180.704539] Lustre: Skipped 1 previous similar message [Tue Feb 5 18:22:01 2019][1641419.709959] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 18:22:01 2019][1641419.721739] Lustre: 9174:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Tue Feb 5 18:32:11 2019][1642029.758508] Lustre: 15232:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 18:32:11 2019][1642029.770336] Lustre: 15232:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 143 previous similar messages [Tue Feb 5 18:41:38 2019][1642596.881703] Lustre: MGS: Connection restored to 030cc203-3f8d-56fb-8dea-10d7c076053d (at 10.8.11.22@o2ib6) [Tue Feb 5 18:41:38 2019][1642596.891533] Lustre: Skipped 2 previous similar messages [Tue Feb 5 18:47:57 2019][1642976.288910] Lustre: 15221:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 18:47:57 2019][1642976.300729] Lustre: 15221:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 71 previous similar messages [Tue Feb 5 18:58:06 2019][1643585.434547] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 18:58:06 2019][1643585.446347] Lustre: 8967:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 117 previous similar messages [Tue Feb 5 19:00:54 2019][1643753.399686] Lustre: MGS: Connection restored to 7a684df6-241f-9190-0d39-cba11021de57 (at 10.9.103.31@o2ib4) [Tue Feb 5 19:00:54 2019][1643753.409611] Lustre: Skipped 2 previous similar messages [Tue Feb 5 19:02:11 2019][1643829.823063] Lustre: MGS: Connection restored to 268d1c3c-774f-f536-a37d-fb49887ef928 (at 10.9.103.42@o2ib4) [Tue Feb 5 19:02:11 2019][1643829.832991] Lustre: Skipped 2 previous similar messages [Tue Feb 5 19:02:14 2019][1643833.265421] Lustre: MGS: Connection restored to eea8150a-0569-63a3-187d-7721c9515ca4 (at 10.9.103.39@o2ib4) [Tue Feb 5 19:02:14 2019][1643833.275339] Lustre: Skipped 2 previous similar messages [Tue Feb 5 19:08:25 2019][1644204.506449] Lustre: 15281:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 19:08:25 2019][1644204.518303] Lustre: 15281:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 344 previous similar messages [Tue Feb 5 19:20:08 2019][1644907.336118] Lustre: 15218:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 19:20:08 2019][1644907.347944] Lustre: 15218:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 109 previous similar messages [Tue Feb 5 19:21:02 2019][1644961.207827] Lustre: MGS: Connection restored to 030cc203-3f8d-56fb-8dea-10d7c076053d (at 10.8.11.22@o2ib6) [Tue Feb 5 19:21:02 2019][1644961.217683] Lustre: Skipped 2 previous similar messages [Tue Feb 5 19:37:25 2019][1645944.616447] Lustre: 15407:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 19:37:25 2019][1645944.628276] Lustre: 15407:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Tue Feb 5 19:46:18 2019][1646477.740895] Lustre: MGS: Connection restored to 030cc203-3f8d-56fb-8dea-10d7c076053d (at 10.8.11.22@o2ib6) [Tue Feb 5 19:46:19 2019][1646477.750729] Lustre: Skipped 2 previous similar messages [Tue Feb 5 19:52:28 2019][1646847.696215] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 19:52:28 2019][1646847.707967] Lustre: 8968:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Feb 5 20:11:57 2019][1648015.945379] Lustre: MGS: Connection restored to 030cc203-3f8d-56fb-8dea-10d7c076053d (at 10.8.11.22@o2ib6) [Tue Feb 5 20:11:57 2019][1648015.955305] Lustre: Skipped 2 previous similar messages [Tue Feb 5 20:12:12 2019][1648031.218055] Lustre: MGS: Connection restored to 280f0da5-15c3-c668-f031-073d4081bde7 (at 10.9.103.44@o2ib4) [Tue Feb 5 20:12:12 2019][1648031.227982] Lustre: Skipped 2 previous similar messages [Tue Feb 5 20:13:22 2019][1648101.153751] Lustre: MGS: Connection restored to 9d6f1a6c-86f9-aa7e-88b7-c0bd61e3f7da (at 10.8.3.13@o2ib6) [Tue Feb 5 20:13:22 2019][1648101.163496] Lustre: Skipped 2 previous similar messages [Tue Feb 5 20:13:57 2019][1648136.285942] Lustre: MGS: Connection restored to 64ea649c-2c23-8bd1-64f4-8cf178315ba1 (at 10.8.3.35@o2ib6) [Tue Feb 5 20:13:57 2019][1648136.295689] Lustre: Skipped 2 previous similar messages [Tue Feb 5 20:15:18 2019][1648216.890514] Lustre: MGS: Connection restored to a06e32f5-be05-2ce3-f3a9-dc278ccf571e (at 10.8.3.19@o2ib6) [Tue Feb 5 20:15:18 2019][1648216.900257] Lustre: Skipped 2 previous similar messages [Tue Feb 5 20:17:59 2019][1648377.812390] Lustre: MGS: Connection restored to c07ed071-ac97-169e-c9f5-822455127b7b (at 10.8.3.7@o2ib6) [Tue Feb 5 20:17:59 2019][1648377.822044] Lustre: Skipped 2 previous similar messages [Tue Feb 5 20:24:33 2019][1648772.668191] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 20:24:33 2019][1648772.679951] Lustre: 7500:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages [Tue Feb 5 20:37:23 2019][1649542.042851] Lustre: MGS: Connection restored to 030cc203-3f8d-56fb-8dea-10d7c076053d (at 10.8.11.22@o2ib6) [Tue Feb 5 20:37:23 2019][1649542.052687] Lustre: Skipped 2 previous similar messages [Tue Feb 5 20:39:23 2019][1649662.795326] Lustre: MGS: Connection restored to 13a355c6-e0cb-5c1c-c1dd-335f5e5a5755 (at 10.8.0.65@o2ib6) [Tue Feb 5 20:39:24 2019][1649662.805068] Lustre: Skipped 1 previous similar message [Tue Feb 5 20:39:30 2019][1649668.856024] Lustre: 15218:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 20:41:08 2019][1649766.911354] Lustre: fir-MDT0002: haven't heard from client f9688477-20ef-48d9-107d-c35a4ee3a4ab (at 10.9.0.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9143d9c8e000, cur 1549428106 expire 1549427956 last 1549427879 [Tue Feb 5 20:41:08 2019][1649766.933057] Lustre: Skipped 5 previous similar messages [Tue Feb 5 20:43:40 2019][1649919.501171] Lustre: 6690:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 20:44:29 2019][1649968.539204] Lustre: MGS: Connection restored to 6caeea3e-5b74-9a8d-d8ac-3bf9f06c410e (at 10.9.0.1@o2ib4) [Tue Feb 5 20:44:29 2019][1649968.548858] Lustre: Skipped 2 previous similar messages [Tue Feb 5 20:57:50 2019][1650769.723862] Lustre: 15281:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 21:02:32 2019][1651050.857069] Lustre: MGS: Connection restored to 030cc203-3f8d-56fb-8dea-10d7c076053d (at 10.8.11.22@o2ib6) [Tue Feb 5 21:02:32 2019][1651050.866900] Lustre: Skipped 2 previous similar messages [Tue Feb 5 21:05:13 2019][1651212.770800] Lustre: 8969:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 21:06:06 2019][1651265.243453] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 21:09:29 2019][1651468.547480] Lustre: MGS: Connection restored to ac310a0e-e3ec-09c0-b73a-0dae38c131ec (at 10.8.3.33@o2ib6) [Tue Feb 5 21:09:29 2019][1651468.557219] Lustre: Skipped 2 previous similar messages [Tue Feb 5 21:11:30 2019][1651589.850409] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 21:16:38 2019][1651897.031665] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 21:16:38 2019][1651897.043401] Lustre: 7191:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Tue Feb 5 21:22:06 2019][1652225.284590] Lustre: 15407:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Tue Feb 5 21:27:56 2019][1652575.157835] Lustre: MGS: Connection restored to 030cc203-3f8d-56fb-8dea-10d7c076053d (at 10.8.11.22@o2ib6) [Tue Feb 5 21:27:56 2019][1652575.167663] Lustre: Skipped 2 previous similar messages [Tue Feb 5 21:40:21 2019][1653320.686250] Lustre: MGS: Connection restored to 9a3fec86-6c87-9861-a86b-61fc0ee356f3 (at 10.8.11.9@o2ib6) [Tue Feb 5 21:40:21 2019][1653320.695994] Lustre: Skipped 2 previous similar messages [Tue Feb 5 21:45:02 2019][1653601.384882] Lustre: MGS: Connection restored to e421d59a-4482-eafc-ea85-c173e2c58e62 (at 10.8.3.11@o2ib6) [Tue Feb 5 21:45:02 2019][1653601.394625] Lustre: Skipped 2 previous similar messages [Tue Feb 5 21:49:14 2019][1653853.067783] Lustre: fir-MDT0002: haven't heard from client 81ce647f-387b-b748-de78-f885bffe2ce2 (at 10.8.3.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff913886194400, cur 1549432192 expire 1549432042 last 1549431965 [Tue Feb 5 21:49:14 2019][1653853.089575] Lustre: Skipped 1 previous similar message [Tue Feb 5 21:51:05 2019][1653964.157382] Lustre: MGS: Connection restored to 0686d8e8-a5ee-caf3-27c7-73ca8bda20b8 (at 10.9.103.40@o2ib4) [Tue Feb 5 21:51:05 2019][1653964.167302] Lustre: Skipped 2 previous similar messages [Tue Feb 5 21:51:11 2019][1653970.732556] Lustre: MGS: Connection restored to f6065da2-d0e9-d83f-4307-4b8a941f7c7b (at 10.9.103.36@o2ib4) [Tue Feb 5 21:51:11 2019][1653970.742498] Lustre: Skipped 2 previous similar messages [Tue Feb 5 21:51:26 2019][1653985.375851] Lustre: MGS: Connection restored to 01ef18f3-802a-530c-7b43-96deb847ea2e (at 10.9.103.37@o2ib4) [Tue Feb 5 21:51:26 2019][1653985.385777] Lustre: Skipped 5 previous similar messages [Tue Feb 5 21:51:42 2019][1654001.306763] Lustre: MGS: Connection restored to e8504e50-3d13-6642-d095-2ac0e7affbdc (at 10.9.103.38@o2ib4) [Tue Feb 5 21:51:42 2019][1654001.316676] Lustre: Skipped 2 previous similar messages [Tue Feb 5 21:52:34 2019][1654053.830965] Lustre: MGS: Connection restored to 280f0da5-15c3-c668-f031-073d4081bde7 (at 10.9.103.44@o2ib4) [Tue Feb 5 21:52:34 2019][1654053.840880] Lustre: Skipped 5 previous similar messages [Tue Feb 5 21:53:25 2019][1654104.092109] Lustre: MGS: Connection restored to eea8150a-0569-63a3-187d-7721c9515ca4 (at 10.9.103.39@o2ib4) [Tue Feb 5 21:53:25 2019][1654104.102023] Lustre: Skipped 2 previous similar messages [Tue Feb 5 21:55:13 2019][1654212.096902] Lustre: MGS: Connection restored to 030cc203-3f8d-56fb-8dea-10d7c076053d (at 10.8.11.22@o2ib6) [Tue Feb 5 21:55:13 2019][1654212.106730] Lustre: Skipped 5 previous similar messages [Tue Feb 5 21:57:21 2019][1654340.775913] Lustre: MGS: Connection restored to 7690a4eb-01d5-36eb-a4b1-affbc86df541 (at 10.9.103.25@o2ib4) [Tue Feb 5 21:57:21 2019][1654340.785833] Lustre: Skipped 23 previous similar messages [Tue Feb 5 22:09:47 2019][1655086.517884] Lustre: MGS: Connection restored to 795c3f0b-ea8c-a263-d8c9-3a172af5c95b (at 10.8.3.26@o2ib6) [Tue Feb 5 22:09:47 2019][1655086.527627] Lustre: Skipped 62 previous similar messages [Tue Feb 5 22:18:19 2019][1655598.815970] Lustre: fir-MDT0000: Connection restored to 7d55a611-6993-ed8f-69a9-2367782a7615 (at 10.9.104.56@o2ib4) [Tue Feb 5 22:18:19 2019][1655598.826574] Lustre: Skipped 148 previous similar messages [Tue Feb 5 22:27:00 2019][1656119.225246] Lustre: MGS: haven't heard from client d54e60dc-425d-0ab2-6ac1-5aa5fae7e2e8 (at 10.8.0.67@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff913210289c00, cur 1549434458 expire 1549434308 last 1549434231 [Tue Feb 5 22:27:00 2019][1656119.246342] Lustre: Skipped 17 previous similar messages [Tue Feb 5 22:32:42 2019][1656461.581436] Lustre: MGS: Connection restored to efd45469-5432-2f56-890e-baca8e1c979d (at 10.9.102.39@o2ib4) [Tue Feb 5 22:32:42 2019][1656461.591363] Lustre: Skipped 355 previous similar messages [Tue Feb 5 22:45:34 2019][1657233.368301] Lustre: MGS: Connection restored to 5d3196b8-02a1-9b6d-839d-3d83a9b907be (at 10.9.107.56@o2ib4) [Tue Feb 5 22:45:34 2019][1657233.378219] Lustre: Skipped 95 previous similar messages [Tue Feb 5 22:59:08 2019][1658047.768848] Lustre: MGS: Connection restored to 8ee55f5c-0afc-d6ce-15f0-19a29e7ea7aa (at 10.9.114.7@o2ib4) [Tue Feb 5 22:59:08 2019][1658047.778690] Lustre: Skipped 74 previous similar messages [Tue Feb 5 23:08:34 2019][1658613.201872] Lustre: MGS: haven't heard from client 771fb934-0748-a1bd-8397-8a9108b66475 (at 10.9.115.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9141f5b15000, cur 1549436952 expire 1549436802 last 1549436725 [Tue Feb 5 23:09:25 2019][1658664.155690] Lustre: fir-MDT0002: haven't heard from client 59e5976b-e92e-fe58-4cc6-1a51bbca02ac (at 10.8.3.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff912b1c292000, cur 1549437003 expire 1549436853 last 1549436776 [Tue Feb 5 23:09:25 2019][1658664.177482] Lustre: Skipped 32 previous similar messages [Tue Feb 5 23:30:11 2019][1659910.759985] Lustre: MGS: Connection restored to 4ac771f0-3783-33a4-47c5-b768435597bd (at 10.8.19.1@o2ib6) [Tue Feb 5 23:30:11 2019][1659910.769740] Lustre: Skipped 84 previous similar messages [Tue Feb 5 23:42:26 2019][1660645.390353] Lustre: MGS: Connection restored to c3ea57d5-46ff-5c6e-1800-776bdd796fdc (at 10.8.20.15@o2ib6) [Tue Feb 5 23:42:26 2019][1660645.400187] Lustre: Skipped 2 previous similar messages [Tue Feb 5 23:47:00 2019][1660919.075316] Lustre: MGS: Connection restored to 7d4a446a-adb6-b506-22c4-3f678182447c (at 10.8.27.16@o2ib6) [Tue Feb 5 23:47:00 2019][1660919.085142] Lustre: Skipped 2 previous similar messages [Tue Feb 5 23:49:35 2019][1661074.509107] Lustre: MGS: Connection restored to f678a0b0-7267-5205-8583-0ea2251f1a66 (at 10.8.19.7@o2ib6) [Tue Feb 5 23:49:35 2019][1661074.518851] Lustre: Skipped 2 previous similar messages [Tue Feb 5 23:51:13 2019][1661172.580452] Lustre: MGS: Connection restored to 903ac7c0-5d7f-8184-801e-07f23b76e9e6 (at 10.8.22.13@o2ib6) [Tue Feb 5 23:51:13 2019][1661172.590278] Lustre: Skipped 2 previous similar messages [Tue Feb 5 23:52:28 2019][1661247.893121] Lustre: MGS: Connection restored to 677b2254-0924-783d-1b6e-fdbcd5f8bf80 (at 10.8.18.15@o2ib6) [Tue Feb 5 23:52:28 2019][1661247.902956] Lustre: Skipped 158 previous similar messages [Tue Feb 5 23:54:59 2019][1661398.119808] Lustre: fir-MDT0000: Connection restored to 559f02f1-82c7-2052-8402-d0fd546dc70a (at 10.8.18.2@o2ib6) [Tue Feb 5 23:54:59 2019][1661398.130249] Lustre: Skipped 295 previous similar messages [Tue Feb 5 23:58:48 2019][1661627.244395] Lustre: fir-MDT0002: haven't heard from client 225eea3d-b7b7-fb2f-9cfe-041d20b24f14 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff912586594800, cur 1549439966 expire 1549439816 last 1549439739 [Tue Feb 5 23:58:48 2019][1661627.266272] Lustre: Skipped 63 previous similar messages [Wed Feb 6 00:00:51 2019][1661750.348203] Lustre: MGS: Connection restored to e72bf792-821b-26bb-ac07-22ee49a2a665 (at 10.8.20.10@o2ib6) [Wed Feb 6 00:00:51 2019][1661750.358037] Lustre: Skipped 54 previous similar messages [Wed Feb 6 00:21:25 2019][1662984.733837] LNetError: 6806:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.15.5@tcp from [Wed Feb 6 00:21:25 2019][1662984.744381] LNetError: 6806:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 6116062 previous similar messages [Wed Feb 6 00:21:25 2019][1662984.755502] Lustre: 6806:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549441323/real 1549441323] req@ffff91268533f800 x1623682050406720/t0(0) o104->fir-MDT0000@10.10.15.5@tcp:15/16 lens 296/224 e 0 to 1 dl 1549441334 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 [Wed Feb 6 00:21:25 2019][1662984.783028] Lustre: 6806:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3497 previous similar messages [Wed Feb 6 00:21:33 2019][1662992.755634] Lustre: 17221:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549441331/real 1549441331] req@ffff914f37bc7200 x1623682050429328/t0(0) o104->fir-MDT0002@10.10.112.12@tcp:15/16 lens 296/224 e 0 to 1 dl 1549441342 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:21:33 2019][1662992.783396] Lustre: 17221:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2254031 previous similar messages [Wed Feb 6 00:21:41 2019][1663000.733840] LNetError: 6690:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 00:21:41 2019][1663000.744539] LNetError: 6690:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 5472152 previous similar messages [Wed Feb 6 00:21:49 2019][1663008.756039] Lustre: 6690:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549441347/real 1549441347] req@ffff911d9d270300 x1623682050426368/t0(0) o104->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/224 e 0 to 1 dl 1549441358 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:21:49 2019][1663008.783721] Lustre: 6690:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6584094 previous similar messages [Wed Feb 6 00:22:13 2019][1663032.734640] LNetError: 6806:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.29.8@tcp from [Wed Feb 6 00:22:13 2019][1663032.745157] LNetError: 6806:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 13001014 previous similar messages [Wed Feb 6 00:22:21 2019][1663040.756841] Lustre: 6690:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549441379/real 1549441379] req@ffff911d9d270300 x1623682050426368/t0(0) o104->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/224 e 0 to 1 dl 1549441390 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:22:21 2019][1663040.784524] Lustre: 6690:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 13190798 previous similar messages [Wed Feb 6 00:23:17 2019][1663096.736244] LNetError: 17221:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.112.12@tcp from [Wed Feb 6 00:23:17 2019][1663096.747025] LNetError: 17221:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 26028373 previous similar messages [Wed Feb 6 00:23:25 2019][1663104.758449] Lustre: 17221:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549441443/real 1549441443] req@ffff914f37bc7200 x1623682050429328/t0(0) o104->fir-MDT0002@10.10.112.12@tcp:15/16 lens 296/224 e 0 to 1 dl 1549441454 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:23:25 2019][1663104.786218] Lustre: 17221:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 26395128 previous similar messages [Wed Feb 6 00:23:28 2019][1663107.231179] Lustre: MGS: Connection restored to c3ea57d5-46ff-5c6e-1800-776bdd796fdc (at 10.8.20.15@o2ib6) [Wed Feb 6 00:23:28 2019][1663107.241012] Lustre: Skipped 8 previous similar messages [Wed Feb 6 00:23:56 2019][1663135.577232] LustreError: 6806:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.29.8@tcp) failed to reply to blocking AST (req@ffff912685338300 x1623682050406736 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff91351fccb3c0/0x21ed935180414c36 lrc: 4/0,0 mode: PR/PR res: [0x200001760:0x2:0x0].0x0 bits 0x13/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d5452 expref: 20 pid: 7500 timeout: 1663242 lvb_type: 0 [Wed Feb 6 00:23:56 2019][1663135.577242] LustreError: 138-a: fir-MDT0000: A client on nid 10.10.29.8@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:23:56 2019][1663135.577244] LustreError: Skipped 1 previous similar message [Wed Feb 6 00:23:56 2019][1663135.577273] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.29.8@tcp ns: mdt-fir-MDT0000_UUID lock: ffff912ae042d100/0x21ed935180414b56 lrc: 3/0,0 mode: PR/PR res: [0x20000173d:0x4e4a:0x0].0x0 bits 0x13/0x0 rrc: 12 type: IBT flags: 0x60200400000020 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d5428 expref: 21 pid: 9030 timeout: 0 lvb_type: 0 [Wed Feb 6 00:23:56 2019][1663135.674769] LustreError: 6806:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 5 previous similar messages [Wed Feb 6 00:23:56 2019][1663135.685120] LustreError: 8978:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff913b0832e000 ns: mdt-fir-MDT0000_UUID lock: ffff913cbc839d40/0x21ed935180415a52 lrc: 3/0,0 mode: PR/PR res: [0x200001760:0x2:0x0].0x0 bits 0x13/0x0 rrc: 11 type: IBT flags: 0x50200400000020 nid: 10.10.113.3@tcp remote: 0x4d977b31312bd49c expref: 3 pid: 8978 timeout: 0 lvb_type: 0 [Wed Feb 6 00:24:01 2019][1663140.577356] LustreError: 17052:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff9156256f8900 x1623682050429584 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff9150b5aae300/0x21ed93518044199a lrc: 4/0,0 mode: PR/PR res: [0x2c0001679:0x3:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e13e0d expref: 14 pid: 6682 timeout: 1663247 lvb_type: 0 [Wed Feb 6 00:24:01 2019][1663140.577362] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.112.12@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:24:01 2019][1663140.577364] LustreError: Skipped 4 previous similar messages [Wed Feb 6 00:24:01 2019][1663140.577381] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.12@tcp ns: mdt-fir-MDT0002_UUID lock: ffff913a67b39200/0x21ed935180441016 lrc: 3/0,0 mode: PR/PR res: [0x2c0001679:0x7:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.112.12@tcp remote: 0xefba423e1327e383 expref: 16 pid: 17052 timeout: 0 lvb_type: 0 [Wed Feb 6 00:24:01 2019][1663140.577384] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages [Wed Feb 6 00:24:01 2019][1663140.685431] LustreError: 17052:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages [Wed Feb 6 00:24:46 2019][1663185.457488] LNet: Service thread pid 6806 was inactive for 200.71s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:24:46 2019][1663185.474510] Pid: 6806, comm: mdt00_008 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:24:46 2019][1663185.484341] Call Trace: [Wed Feb 6 00:24:46 2019][1663185.486975] [] 0xffffffffffffffff [Wed Feb 6 00:24:46 2019][1663185.492182] LustreError: dumping log to /tmp/lustre-log.1549441524.6806 [Wed Feb 6 00:24:51 2019][1663190.577619] LNet: Service thread pid 6683 was inactive for 200.03s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:24:51 2019][1663190.594638] Pid: 6683, comm: mdt01_005 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:24:51 2019][1663190.604467] Call Trace: [Wed Feb 6 00:24:51 2019][1663190.607110] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663190.614240] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663190.621607] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 00:24:51 2019][1663190.629821] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:24:51 2019][1663190.636210] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:24:51 2019][1663190.642424] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:24:51 2019][1663190.649158] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:24:51 2019][1663190.655025] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663190.662149] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663190.670035] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663190.676535] [] kthread+0xd1/0xe0 [Wed Feb 6 00:24:51 2019][1663190.681625] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:24:51 2019][1663190.688270] [] 0xffffffffffffffff [Wed Feb 6 00:24:51 2019][1663190.693476] LustreError: dumping log to /tmp/lustre-log.1549441529.6683 [Wed Feb 6 00:24:51 2019][1663190.836851] Pid: 15371, comm: mdt02_045 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:24:51 2019][1663190.846764] Call Trace: [Wed Feb 6 00:24:51 2019][1663190.849405] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663190.856522] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663190.863793] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663190.870640] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:24:51 2019][1663190.877373] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:24:51 2019][1663190.883770] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:24:51 2019][1663190.890072] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:24:51 2019][1663190.897507] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:24:51 2019][1663190.904414] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:24:51 2019][1663190.912631] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:24:51 2019][1663190.919027] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:24:51 2019][1663190.925241] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:24:51 2019][1663190.931975] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:24:51 2019][1663190.937895] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663190.945011] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663190.952907] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663190.959406] [] kthread+0xd1/0xe0 [Wed Feb 6 00:24:51 2019][1663190.964494] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:24:51 2019][1663190.971141] [] 0xffffffffffffffff [Wed Feb 6 00:24:51 2019][1663190.976340] Pid: 9009, comm: mdt01_027 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:24:51 2019][1663190.986162] Call Trace: [Wed Feb 6 00:24:51 2019][1663190.988792] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663190.995895] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663191.003174] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663191.010012] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:24:51 2019][1663191.016747] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:24:51 2019][1663191.023134] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:24:51 2019][1663191.029435] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:24:51 2019][1663191.036862] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:24:51 2019][1663191.043770] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:24:51 2019][1663191.051986] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:24:51 2019][1663191.058374] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:24:51 2019][1663191.064614] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:24:51 2019][1663191.071349] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:24:51 2019][1663191.077216] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663191.084330] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663191.092218] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:24:51 2019][1663191.098719] [] kthread+0xd1/0xe0 [Wed Feb 6 00:24:51 2019][1663191.103796] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:24:51 2019][1663191.110435] [] 0xffffffffffffffff [Wed Feb 6 00:24:51 2019][1663191.115617] Pid: 53422, comm: mdt02_027 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:24:51 2019][1663191.125527] Call Trace: [Wed Feb 6 00:24:52 2019][1663191.128157] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 00:24:52 2019][1663191.135266] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:24:52 2019][1663191.142539] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:24:52 2019][1663191.149376] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:24:52 2019][1663191.156109] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:24:52 2019][1663191.162498] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:24:52 2019][1663191.168799] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:24:52 2019][1663191.176226] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:24:52 2019][1663191.183135] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:24:52 2019][1663191.191375] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:24:52 2019][1663191.197773] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:24:52 2019][1663191.203987] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:24:52 2019][1663191.210719] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:24:52 2019][1663191.216588] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:24:52 2019][1663191.223696] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:24:52 2019][1663191.231581] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:24:52 2019][1663191.238082] [] kthread+0xd1/0xe0 [Wed Feb 6 00:24:52 2019][1663191.243162] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:24:52 2019][1663191.249800] [] 0xffffffffffffffff [Wed Feb 6 00:24:52 2019][1663191.254977] LNet: Service thread pid 9014 was inactive for 201.08s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:24:52 2019][1663191.601641] LustreError: dumping log to /tmp/lustre-log.1549441530.6173 [Wed Feb 6 00:24:53 2019][1663192.113650] LNet: Service thread pid 15164 was inactive for 200.50s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:24:53 2019][1663192.126683] LNet: Skipped 11 previous similar messages [Wed Feb 6 00:24:53 2019][1663192.625666] LustreError: dumping log to /tmp/lustre-log.1549441531.15293 [Wed Feb 6 00:24:54 2019][1663193.137674] LNet: Service thread pid 8829 was inactive for 200.36s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:24:54 2019][1663193.150631] LNet: Skipped 6 previous similar messages [Wed Feb 6 00:24:54 2019][1663193.649685] LustreError: dumping log to /tmp/lustre-log.1549441532.8984 [Wed Feb 6 00:24:55 2019][1663194.673711] LustreError: dumping log to /tmp/lustre-log.1549441533.15292 [Wed Feb 6 00:24:56 2019][1663195.185721] LNet: Service thread pid 15392 was inactive for 200.39s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:24:56 2019][1663195.198758] LNet: Skipped 6 previous similar messages [Wed Feb 6 00:24:59 2019][1663198.769812] LustreError: dumping log to /tmp/lustre-log.1549441537.17219 [Wed Feb 6 00:25:00 2019][1663199.793844] LNet: Service thread pid 15510 was inactive for 200.03s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:25:00 2019][1663199.806881] LNet: Skipped 3 previous similar messages [Wed Feb 6 00:25:00 2019][1663199.812109] LustreError: dumping log to /tmp/lustre-log.1549441538.15510 [Wed Feb 6 00:25:02 2019][1663201.841893] LustreError: dumping log to /tmp/lustre-log.1549441540.9019 [Wed Feb 6 00:25:05 2019][1663204.913970] LustreError: dumping log to /tmp/lustre-log.1549441543.48636 [Wed Feb 6 00:25:08 2019][1663207.986047] LNet: Service thread pid 6883 was inactive for 200.20s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:25:08 2019][1663207.998989] LNet: Skipped 5 previous similar messages [Wed Feb 6 00:25:08 2019][1663208.004226] LustreError: dumping log to /tmp/lustre-log.1549441546.6883 [Wed Feb 6 00:25:09 2019][1663209.010074] LustreError: dumping log to /tmp/lustre-log.1549441547.15222 [Wed Feb 6 00:25:16 2019][1663215.154249] LustreError: dumping log to /tmp/lustre-log.1549441553.6177 [Wed Feb 6 00:25:25 2019][1663224.739464] LNetError: 6806:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.15.5@tcp from [Wed Feb 6 00:25:25 2019][1663224.749986] LNetError: 6806:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 44910137 previous similar messages [Wed Feb 6 00:25:31 2019][1663230.514609] LNet: Service thread pid 6804 was inactive for 200.59s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:25:31 2019][1663230.527560] LNet: Skipped 6 previous similar messages [Wed Feb 6 00:25:31 2019][1663230.532795] LustreError: dumping log to /tmp/lustre-log.1549441569.6804 [Wed Feb 6 00:25:32 2019][1663231.538641] LustreError: dumping log to /tmp/lustre-log.1549441570.15170 [Wed Feb 6 00:25:33 2019][1663232.761659] Lustre: 6806:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549441571/real 1549441571] req@ffff913216dabf00 x1623682050531216/t0(0) o105->fir-MDT0000@10.10.15.5@tcp:15/16 lens 304/224 e 0 to 1 dl 1549441582 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:25:33 2019][1663232.789168] Lustre: 6806:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 44626287 previous similar messages [Wed Feb 6 00:25:36 2019][1663235.634745] LustreError: dumping log to /tmp/lustre-log.1549441574.6617 [Wed Feb 6 00:25:41 2019][1663240.242855] LustreError: dumping log to /tmp/lustre-log.1549441579.8970 [Wed Feb 6 00:26:27 2019][1663286.581021] LustreError: 15331:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.29.8@tcp) failed to reply to blocking AST (req@ffff91512db9b300 x1623682050531152 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff913754a20240/0x21ed93518048fc78 lrc: 4/0,0 mode: PR/PR res: [0x200001761:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d55a9 expref: 7 pid: 8996 timeout: 1663393 lvb_type: 0 [Wed Feb 6 00:26:27 2019][1663286.581023] LustreError: 138-a: fir-MDT0000: A client on nid 10.10.29.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:26:27 2019][1663286.581027] LustreError: Skipped 2 previous similar messages [Wed Feb 6 00:26:27 2019][1663286.581050] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.29.6@tcp ns: mdt-fir-MDT0000_UUID lock: ffff912931b1a1c0/0x21ed93518041c59e lrc: 3/0,0 mode: PR/PR res: [0x20000173d:0x4e4a:0x0].0x0 bits 0x13/0x0 rrc: 9 type: IBT flags: 0x60200400000020 nid: 10.10.29.6@tcp remote: 0xc2974fb1157c3001 expref: 16 pid: 7191 timeout: 0 lvb_type: 0 [Wed Feb 6 00:26:27 2019][1663286.581053] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [Wed Feb 6 00:26:27 2019][1663286.581100] Lustre: 6806:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (181:121s); client may timeout. req@ffff91250d966300 x1624705202173536/t0(0) o36->ef153e15-215b-fb2f-42df-dec383d871e7@10.10.29.7@tcp:304/0 lens 512/416 e 7 to 0 dl 1549441504 ref 1 fl Complete:/0/0 rc -17/-17 [Wed Feb 6 00:26:27 2019][1663286.581289] LNet: Service thread pid 6806 completed after 301.83s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 00:26:27 2019][1663286.733187] LustreError: 15331:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages [Wed Feb 6 00:26:27 2019][1663286.773110] Lustre: fir-MDT0000: Connection restored to 81252705-51ba-7971-7ef9-ced40cce7ddd (at 10.10.29.8@tcp) [Wed Feb 6 00:26:27 2019][1663286.783457] Lustre: Skipped 8 previous similar messages [Wed Feb 6 00:26:30 2019][1663290.015115] LustreError: 8999:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441328, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff912750fe69c0/0x21ed93518044377d lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 162 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 8999 timeout: 0 lvb_type: 0 [Wed Feb 6 00:26:30 2019][1663290.028104] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:26:30 2019][1663290.028107] Lustre: Skipped 31 previous similar messages [Wed Feb 6 00:26:30 2019][1663290.028122] LustreError: 17050:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441328, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff9155c0a045c0/0x21ed9351804439de lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 45 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351804439fa expref: -99 pid: 17050 timeout: 0 lvb_type: 0 [Wed Feb 6 00:26:31 2019][1663290.028191] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Feb 6 00:26:31 2019][1663290.538119] LustreError: 15260:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441329, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9137066ebcc0/0x21ed93518044d590 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 162 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 15260 timeout: 0 lvb_type: 0 [Wed Feb 6 00:26:31 2019][1663290.577623] LustreError: 15260:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Feb 6 00:26:31 2019][1663290.750127] LustreError: 53422:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441329, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff913cb1db3f00/0x21ed93518044fc87 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 45 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518044fc8e expref: -99 pid: 53422 timeout: 0 lvb_type: 0 [Wed Feb 6 00:26:31 2019][1663290.790380] LustreError: 53422:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 6 previous similar messages [Wed Feb 6 00:26:32 2019][1663291.581154] LustreError: 6690:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.112.8@tcp) failed to reply to blocking AST (req@ffff9125f867ad00 x1623682050533232 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff91597a6660c0/0x21ed93518048ffc0 lrc: 4/0,0 mode: PR/PR res: [0x2c0001689:0x3:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.112.8@tcp remote: 0x8ea3e4d27142f19a expref: 9 pid: 6682 timeout: 1663398 lvb_type: 0 [Wed Feb 6 00:26:32 2019][1663291.623399] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.112.8@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:26:32 2019][1663291.635999] LustreError: Skipped 2 previous similar messages [Wed Feb 6 00:26:32 2019][1663291.641857] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.8@tcp ns: mdt-fir-MDT0002_UUID lock: ffff91597a6660c0/0x21ed93518048ffc0 lrc: 3/0,0 mode: PR/PR res: [0x2c0001689:0x3:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.112.8@tcp remote: 0x8ea3e4d27142f19a expref: 10 pid: 6682 timeout: 0 lvb_type: 0 [Wed Feb 6 00:26:32 2019][1663291.670040] LNet: Service thread pid 8999 completed after 301.65s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 00:26:32 2019][1663291.695028] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages [Wed Feb 6 00:28:56 2019][1663435.726761] LustreError: 17053:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441474, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff915723a92d00/0x21ed93518048fcda lrc: 3/1,0 mode: --/PR res: [0x200001760:0x2:0x0].0x0 bits 0x13/0x0 rrc: 16 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 17053 timeout: 0 lvb_type: 0 [Wed Feb 6 00:28:56 2019][1663435.766228] LustreError: 17053:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 5 previous similar messages [Wed Feb 6 00:28:58 2019][1663437.584834] LustreError: 8967:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.3@tcp) failed to reply to blocking AST (req@ffff912e6531f500 x1623682050590080 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff91265ab2e9c0/0x21ed93518048fdc8 lrc: 4/0,0 mode: PR/PR res: [0x20000173d:0x4e4a:0x0].0x0 bits 0x13/0x0 rrc: 10 type: IBT flags: 0x60200400000020 nid: 10.10.113.3@tcp remote: 0x4d977b31312bd694 expref: 11 pid: 8967 timeout: 1663544 lvb_type: 0 [Wed Feb 6 00:28:58 2019][1663437.585035] LustreError: 15163:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff913586160c00 ns: mdt-fir-MDT0000_UUID lock: ffff9139761a2d00/0x21ed93518048fd27 lrc: 3/0,0 mode: PR/PR res: [0x200001760:0x2:0x0].0x0 bits 0x13/0x0 rrc: 15 type: IBT flags: 0x50200400000020 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7bdd8 expref: 2 pid: 15163 timeout: 0 lvb_type: 0 [Wed Feb 6 00:28:58 2019][1663437.585061] Lustre: 15163:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (215:87s); client may timeout. req@ffff913719a6f500 x1624705235731968/t0(0) o101->ffafe5be-09a9-59b7-d26e-f32fde550fcb@10.10.15.6@tcp:489/0 lens 576/616 e 0 to 0 dl 1549441689 ref 1 fl Complete:/0/0 rc -107/-107 [Wed Feb 6 00:28:58 2019][1663437.690841] LustreError: 138-a: fir-MDT0000: A client on nid 10.10.113.3@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:28:58 2019][1663437.703465] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.3@tcp ns: mdt-fir-MDT0000_UUID lock: ffff91265ab2e9c0/0x21ed93518048fdc8 lrc: 3/0,0 mode: PR/PR res: [0x20000173d:0x4e4a:0x0].0x0 bits 0x13/0x0 rrc: 12 type: IBT flags: 0x60200400000020 nid: 10.10.113.3@tcp remote: 0x4d977b31312bd694 expref: 12 pid: 8967 timeout: 0 lvb_type: 0 [Wed Feb 6 00:29:01 2019][1663440.587887] LustreError: 47983:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441479, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff914f7da77080/0x21ed935180490221 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 26 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935180490228 expref: -99 pid: 47983 timeout: 0 lvb_type: 0 [Wed Feb 6 00:29:01 2019][1663440.628135] LustreError: 47983:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 8 previous similar messages [Wed Feb 6 00:29:03 2019][1663442.920947] LustreError: 7415:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441481, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff913700fed580/0x21ed93518049118d lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 149 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 7415 timeout: 0 lvb_type: 0 [Wed Feb 6 00:29:10 2019][1663449.857116] LustreError: 15280:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441488, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff91264bd36c00/0x21ed9351804914ab lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 153 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 15280 timeout: 0 lvb_type: 0 [Wed Feb 6 00:29:10 2019][1663449.896589] LustreError: 15280:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Feb 6 00:29:15 2019][1663454.458513] Lustre: fir-MDT0000: Client ef153e15-215b-fb2f-42df-dec383d871e7 (at 10.10.29.7@tcp) reconnecting [Wed Feb 6 00:29:15 2019][1663454.468624] Lustre: fir-MDT0000: Connection restored to 4be18f20-2685-70bf-2364-3d91cfde7616 (at 10.10.29.7@tcp) [Wed Feb 6 00:29:15 2019][1663454.478971] Lustre: Skipped 8 previous similar messages [Wed Feb 6 00:29:33 2019][1663472.184674] LNet: Service thread pid 8996 was inactive for 336.51s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:29:33 2019][1663472.197625] LNet: Skipped 6 previous similar messages [Wed Feb 6 00:29:33 2019][1663472.202863] LustreError: dumping log to /tmp/lustre-log.1549441810.8996 [Wed Feb 6 00:29:38 2019][1663477.304805] LustreError: dumping log to /tmp/lustre-log.1549441816.6172 [Wed Feb 6 00:29:40 2019][1663479.352859] LustreError: dumping log to /tmp/lustre-log.1549441818.7415 [Wed Feb 6 00:29:41 2019][1663480.745881] LNetError: 15511:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.112.8@tcp from [Wed Feb 6 00:29:41 2019][1663480.756570] LNetError: 15511:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 71763762 previous similar messages [Wed Feb 6 00:29:49 2019][1663488.768081] Lustre: 17204:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549441827/real 1549441827] req@ffff9156ee220c00 x1623682050724368/t0(0) o104->fir-MDT0000@10.10.15.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549441838 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:29:49 2019][1663488.795680] Lustre: 17204:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 71944548 previous similar messages [Wed Feb 6 00:30:03 2019][1663502.905450] LNet: Service thread pid 9030 was inactive for 362.20s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:30:03 2019][1663502.922467] LNet: Skipped 3 previous similar messages [Wed Feb 6 00:30:03 2019][1663502.927701] Pid: 9030, comm: mdt02_020 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:30:03 2019][1663502.937529] Call Trace: [Wed Feb 6 00:30:03 2019][1663502.940172] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:30:03 2019][1663502.947284] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:30:03 2019][1663502.954564] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:30:03 2019][1663502.961413] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:30:03 2019][1663502.968148] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:30:03 2019][1663502.974544] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:30:03 2019][1663502.980852] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:30:03 2019][1663502.988297] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:30:03 2019][1663502.995204] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:30:03 2019][1663503.003420] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:30:03 2019][1663503.009817] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:30:03 2019][1663503.016041] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:30:03 2019][1663503.022782] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:30:03 2019][1663503.028650] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:30:03 2019][1663503.035775] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:30:03 2019][1663503.043661] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:30:03 2019][1663503.050161] [] kthread+0xd1/0xe0 [Wed Feb 6 00:30:03 2019][1663503.055248] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:30:03 2019][1663503.061897] [] 0xffffffffffffffff [Wed Feb 6 00:30:03 2019][1663503.067088] LustreError: dumping log to /tmp/lustre-log.1549441841.9030 [Wed Feb 6 00:30:04 2019][1663503.183271] Pid: 8995, comm: mdt00_018 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:30:04 2019][1663503.193097] Call Trace: [Wed Feb 6 00:30:04 2019][1663503.195735] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:30:04 2019][1663503.202853] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:30:04 2019][1663503.210133] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:30:04 2019][1663503.216982] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:30:04 2019][1663503.223724] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:30:04 2019][1663503.230124] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:30:04 2019][1663503.236443] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:30:04 2019][1663503.243884] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:30:04 2019][1663503.250802] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:30:04 2019][1663503.259016] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:30:04 2019][1663503.265415] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:30:04 2019][1663503.271636] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:30:04 2019][1663503.278370] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:30:04 2019][1663503.284238] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:30:04 2019][1663503.291361] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:30:04 2019][1663503.299249] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:30:04 2019][1663503.305751] [] kthread+0xd1/0xe0 [Wed Feb 6 00:30:04 2019][1663503.310845] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:30:04 2019][1663503.317492] [] 0xffffffffffffffff [Wed Feb 6 00:30:04 2019][1663503.322686] Pid: 15511, comm: mdt02_049 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:30:04 2019][1663503.332599] Call Trace: [Wed Feb 6 00:30:04 2019][1663503.335225] [] 0xffffffffffffffff [Wed Feb 6 00:30:14 2019][1663513.145703] LNet: Service thread pid 15280 was inactive for 363.28s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:30:14 2019][1663513.162816] LNet: Skipped 2 previous similar messages [Wed Feb 6 00:30:14 2019][1663513.168050] Pid: 15280, comm: mdt00_043 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:30:14 2019][1663513.177964] Call Trace: [Wed Feb 6 00:30:14 2019][1663513.180598] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:30:14 2019][1663513.187721] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 00:30:14 2019][1663513.195085] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 00:30:14 2019][1663513.203312] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:30:14 2019][1663513.209709] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:30:14 2019][1663513.215932] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:30:14 2019][1663513.222672] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:30:14 2019][1663513.228550] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:30:14 2019][1663513.235707] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:30:14 2019][1663513.243595] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:30:14 2019][1663513.250097] [] kthread+0xd1/0xe0 [Wed Feb 6 00:30:14 2019][1663513.255194] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:30:14 2019][1663513.261838] [] 0xffffffffffffffff [Wed Feb 6 00:30:14 2019][1663513.267033] LustreError: dumping log to /tmp/lustre-log.1549441852.15280 [Wed Feb 6 00:30:14 2019][1663513.382943] Pid: 9199, comm: mdt02_023 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:30:14 2019][1663513.392771] Call Trace: [Wed Feb 6 00:30:14 2019][1663513.395409] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:30:14 2019][1663513.402528] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 00:30:14 2019][1663513.409894] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 00:30:14 2019][1663513.418111] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:30:14 2019][1663513.424507] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:30:14 2019][1663513.430720] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:30:14 2019][1663513.437465] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:30:14 2019][1663513.443341] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:30:14 2019][1663513.450465] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:30:14 2019][1663513.458361] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:30:14 2019][1663513.464869] [] kthread+0xd1/0xe0 [Wed Feb 6 00:30:14 2019][1663513.469969] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:30:14 2019][1663513.476622] [] 0xffffffffffffffff [Wed Feb 6 00:30:42 2019][1663541.818424] LNet: Service thread pid 15384 was inactive for 386.11s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:30:42 2019][1663541.831455] LNet: Skipped 4 previous similar messages [Wed Feb 6 00:30:42 2019][1663541.836692] LustreError: dumping log to /tmp/lustre-log.1549441880.15384 [Wed Feb 6 00:31:27 2019][1663586.588553] LustreError: 8947:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441625, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9139bb5d0000/0x21ed93518049cee1 lrc: 3/0,1 mode: --/CW res: [0x200001760:0x2:0x0].0x0 bits 0x2/0x0 rrc: 15 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 8947 timeout: 0 lvb_type: 0 [Wed Feb 6 00:31:27 2019][1663586.627762] LustreError: 8947:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 3 previous similar messages [Wed Feb 6 00:31:29 2019][1663588.588608] LustreError: 8996:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.29.7@tcp) failed to reply to completion AST (req@ffff91398640f200 x1623682050724208 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff91228dc1e9c0/0x21ed9351804918c5 lrc: 4/0,0 mode: PR/PR res: [0x200001760:0x2:0x0].0x0 bits 0x2/0x0 rrc: 15 type: IBT flags: 0x60000000000020 nid: 10.10.29.7@tcp remote: 0xf3c0006a65b8fa89 expref: 6 pid: 15228 timeout: 1663695 lvb_type: 0 [Wed Feb 6 00:31:29 2019][1663588.588635] LustreError: 138-a: fir-MDT0000: A client on nid 10.10.29.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:31:29 2019][1663588.588637] LustreError: Skipped 1 previous similar message [Wed Feb 6 00:31:29 2019][1663588.588664] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.29.6@tcp ns: mdt-fir-MDT0000_UUID lock: ffff913c7020d580/0x21ed93518049d165 lrc: 3/0,0 mode: PR/PR res: [0x20000173d:0x4e4a:0x0].0x0 bits 0x13/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.10.29.6@tcp remote: 0xc2974fb1157c3047 expref: 11 pid: 9029 timeout: 0 lvb_type: 0 [Wed Feb 6 00:31:29 2019][1663588.588666] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [Wed Feb 6 00:31:29 2019][1663588.695868] LustreError: 8996:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 5 previous similar messages [Wed Feb 6 00:31:29 2019][1663588.706095] Lustre: 8996:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (231:222s); client may timeout. req@ffff91382d866900 x1624705184348896/t14368173921(0) o36->361df7fd-56ca-4920-f3c9-3d9bba670835@10.10.15.5@tcp:505/0 lens 504/424 e 0 to 0 dl 1549441705 ref 1 fl Complete:/0/0 rc 0/0 [Wed Feb 6 00:31:29 2019][1663588.706201] LustreError: 8967:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9138879ec400 ns: mdt-fir-MDT0000_UUID lock: ffff91258d7260c0/0x21ed9351804ea39c lrc: 3/0,0 mode: PR/PR res: [0x200001760:0x2:0x0].0x0 bits 0x13/0x0 rrc: 13 type: IBT flags: 0x50200000000000 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d55fd expref: 2 pid: 8967 timeout: 0 lvb_type: 0 [Wed Feb 6 00:31:29 2019][1663588.706204] LustreError: 8967:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message [Wed Feb 6 00:31:29 2019][1663588.779609] Lustre: 8996:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1 previous similar message [Wed Feb 6 00:31:29 2019][1663588.789706] LNet: Service thread pid 8996 completed after 453.11s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 00:31:29 2019][1663588.805949] LNet: Skipped 53 previous similar messages [Wed Feb 6 00:31:32 2019][1663591.547528] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 00:31:32 2019][1663591.558569] Lustre: Skipped 2 previous similar messages [Wed Feb 6 00:31:32 2019][1663591.590678] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:31:32 2019][1663591.605804] LustreError: 6845:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441630, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff91535d28f740/0x21ed93518049d245 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 25 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518049d24c expref: -99 pid: 6845 timeout: 0 lvb_type: 0 [Wed Feb 6 00:31:32 2019][1663591.645879] LustreError: 6845:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 6 previous similar messages [Wed Feb 6 00:31:34 2019][1663593.588994] Lustre: 15511:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (319:134s); client may timeout. req@ffff913c36babc00 x1624702076434000/t13599834062(0) o36->0161d142-b9e6-5768-fef0-f19a8b36b0dc@10.10.114.10@tcp:598/0 lens 520/424 e 1 to 0 dl 1549441798 ref 1 fl Complete:/0/0 rc 0/0 [Wed Feb 6 00:31:34 2019][1663593.588998] LustreError: 15386:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9129f984ec00 ns: mdt-fir-MDT0002_UUID lock: ffff9128b8899440/0x21ed9351804ea5a9 lrc: 3/0,0 mode: PR/PR res: [0x2c0001689:0x3:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT flags: 0x50200400000020 nid: 10.10.115.6@tcp remote: 0x9795773f7ddd3091 expref: 3 pid: 15386 timeout: 0 lvb_type: 0 [Wed Feb 6 00:31:34 2019][1663593.589002] LustreError: 15386:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 4 previous similar messages [Wed Feb 6 00:31:34 2019][1663593.663111] LNet: Service thread pid 15511 completed after 453.06s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 00:31:39 2019][1663598.137839] LustreError: 8994:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441636, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff911d467ce540/0x21ed9351804bf200 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 25 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351804bf207 expref: -99 pid: 8994 timeout: 0 lvb_type: 0 [Wed Feb 6 00:31:39 2019][1663598.177918] LustreError: 8994:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 13 previous similar messages [Wed Feb 6 00:31:47 2019][1663606.536050] LustreError: 8999:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441645, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9128197c7980/0x21ed9351804c28a2 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 180 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 8999 timeout: 0 lvb_type: 0 [Wed Feb 6 00:31:47 2019][1663606.575348] LustreError: 8999:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 16 previous similar messages [Wed Feb 6 00:31:49 2019][1663608.753112] LustreError: 15182:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441647, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff9153e3fd2880/0x21ed9351804c3a1b lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 25 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351804c3a22 expref: -99 pid: 15182 timeout: 0 lvb_type: 0 [Wed Feb 6 00:33:05 2019][1663684.349005] LustreError: 15242:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441723, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff91326b72f740/0x21ed9351804d7d56 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 198 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 15242 timeout: 0 lvb_type: 0 [Wed Feb 6 00:33:05 2019][1663684.388478] LustreError: 15242:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages [Wed Feb 6 00:33:19 2019][1663698.333071] LNet: Service thread pid 9030 completed after 557.62s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 00:33:19 2019][1663698.349327] LNet: Skipped 4 previous similar messages [Wed Feb 6 00:33:29 2019][1663708.269425] Lustre: fir-MDT0000: haven't heard from client 5b464c0c-042f-5769-56f1-e3068a3e016c (at 10.9.105.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff913d770ddc00, cur 1549442047 expire 1549441897 last 1549441820 [Wed Feb 6 00:33:29 2019][1663708.291384] Lustre: Skipped 2 previous similar messages [Wed Feb 6 00:33:29 2019][1663708.297314] LNet: Service thread pid 7415 completed after 565.36s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 00:33:29 2019][1663708.313556] LNet: Skipped 3 previous similar messages [Wed Feb 6 00:34:00 2019][1663739.592391] LustreError: 8950:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.3@tcp) failed to reply to blocking AST (req@ffff913f77af2400 x1623682050797840 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff91231764f080/0x21ed9351804efba1 lrc: 4/0,0 mode: PR/PR res: [0x20000173d:0x4e4a:0x0].0x0 bits 0x13/0x0 rrc: 14 type: IBT flags: 0x60200400000020 nid: 10.10.113.3@tcp remote: 0x4d977b31312bd6e1 expref: 10 pid: 15178 timeout: 1663846 lvb_type: 0 [Wed Feb 6 00:34:00 2019][1663739.635153] LustreError: 8950:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages [Wed Feb 6 00:34:00 2019][1663739.645332] LustreError: 138-a: fir-MDT0000: A client on nid 10.10.113.3@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:34:00 2019][1663739.657925] LustreError: Skipped 7 previous similar messages [Wed Feb 6 00:34:00 2019][1663739.663782] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.3@tcp ns: mdt-fir-MDT0000_UUID lock: ffff91231764f080/0x21ed9351804efba1 lrc: 3/0,0 mode: PR/PR res: [0x20000173d:0x4e4a:0x0].0x0 bits 0x13/0x0 rrc: 14 type: IBT flags: 0x60200400000020 nid: 10.10.113.3@tcp remote: 0x4d977b31312bd6e1 expref: 11 pid: 15178 timeout: 0 lvb_type: 0 [Wed Feb 6 00:34:00 2019][1663739.701082] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 7 previous similar messages [Wed Feb 6 00:34:03 2019][1663742.603466] LustreError: 38457:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441781, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff9128f2220fc0/0x21ed9351804ea731 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 25 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351804ea738 expref: -99 pid: 38457 timeout: 0 lvb_type: 0 [Wed Feb 6 00:34:17 2019][1663756.416429] Lustre: fir-MDT0000: Connection restored to 93c760cc-c474-1df6-ff3d-2c7363337311 (at 10.10.113.3@tcp) [Wed Feb 6 00:34:17 2019][1663756.426862] Lustre: Skipped 14 previous similar messages [Wed Feb 6 00:34:55 2019][1663794.733774] LustreError: 15329:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441833, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff91457107bf00/0x21ed9351804fd1d7 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 197 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 15329 timeout: 0 lvb_type: 0 [Wed Feb 6 00:34:55 2019][1663794.773247] LustreError: 15329:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Feb 6 00:35:49 2019][1663848.595137] LustreError: 8968:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.112.12@tcp) failed to reply to blocking AST (req@ffff912d14368300 x1623682050851888 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff9155c0a03600/0x21ed93518051150b lrc: 4/0,0 mode: PR/PR res: [0x2c0001679:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.10.112.12@tcp remote: 0xefba423e1327e551 expref: 15 pid: 17052 timeout: 1663955 lvb_type: 0 [Wed Feb 6 00:35:49 2019][1663848.637732] LustreError: 8968:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 4 previous similar messages [Wed Feb 6 00:35:49 2019][1663848.647910] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.112.12@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:35:49 2019][1663848.660592] LustreError: Skipped 4 previous similar messages [Wed Feb 6 00:35:49 2019][1663848.666456] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 150s: evicting client at 10.10.112.12@tcp ns: mdt-fir-MDT0002_UUID lock: ffff9155c0a03600/0x21ed93518051150b lrc: 3/0,0 mode: PR/PR res: [0x2c0001679:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.10.112.12@tcp remote: 0xefba423e1327e551 expref: 16 pid: 17052 timeout: 0 lvb_type: 0 [Wed Feb 6 00:35:49 2019][1663848.703589] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 4 previous similar messages [Wed Feb 6 00:36:22 2019][1663881.906967] Lustre: 9171:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:36:22 2019][1663881.906967] req@ffff91429d787b00 x1624701847814208/t0(0) o36->480f6595-8ac1-2fa1-010d-61ed12a06db6@10.10.112.13@tcp:270/0 lens 520/2888 e 3 to 0 dl 1549442225 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:36:27 2019][1663886.659086] Lustre: 16071:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:36:27 2019][1663886.659086] req@ffff9155f238b600 x1624701905496368/t0(0) o36->10f5ca51-c1e4-ea2c-70bb-5e5b8eb0ed33@10.9.115.7@o2ib4:275/0 lens 520/2888 e 8 to 0 dl 1549442230 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:36:28 2019][1663887.642290] Lustre: fir-MDT0000: Client 480f6595-8ac1-2fa1-010d-61ed12a06db6 (at 10.10.112.13@tcp) reconnecting [Wed Feb 6 00:36:28 2019][1663887.661112] Lustre: 16071:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:36:28 2019][1663887.661112] req@ffff915083ea6c00 x1624673063876096/t0(0) o36->d6335190-14e9-cc30-2081-663fdf52e20a@10.9.102.12@o2ib4:276/0 lens 520/2888 e 8 to 0 dl 1549442231 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:36:28 2019][1663887.690262] Lustre: 16071:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 15 previous similar messages [Wed Feb 6 00:36:32 2019][1663891.899223] Lustre: 8947:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:36:32 2019][1663891.899223] req@ffff912ce4e96600 x1624705085760192/t0(0) o36->2ae35386-7dbf-e9e4-77f5-0f6e19b82987@10.8.30.25@o2ib6:280/0 lens 520/2888 e 3 to 0 dl 1549442235 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:36:32 2019][1663891.928407] Lustre: 8947:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 9 previous similar messages [Wed Feb 6 00:36:34 2019][1663893.601249] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:36:34 2019][1663893.616376] LustreError: 9008:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441932, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff913105db2f40/0x21ed93518050815c lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 20 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935180508163 expref: -99 pid: 9008 timeout: 0 lvb_type: 0 [Wed Feb 6 00:36:34 2019][1663893.656518] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Feb 6 00:36:34 2019][1663893.666520] Lustre: Skipped 2 previous similar messages [Wed Feb 6 00:36:39 2019][1663899.037415] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 00:36:42 2019][1663901.939473] Lustre: 15510:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:36:42 2019][1663901.939473] req@ffff9145fc7a2a00 x1624705128875040/t0(0) o36->e2149003-749e-c138-5dd7-5ff91f76a584@10.8.10.4@o2ib6:290/0 lens 520/2888 e 3 to 0 dl 1549442245 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:36:42 2019][1663901.968459] Lustre: 15510:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages [Wed Feb 6 00:36:48 2019][1663907.617986] Lustre: fir-MDT0000: Client e2149003-749e-c138-5dd7-5ff91f76a584 (at 10.8.10.4@o2ib6) reconnecting [Wed Feb 6 00:36:48 2019][1663907.628161] Lustre: Skipped 5 previous similar messages [Wed Feb 6 00:36:51 2019][1663910.227682] Lustre: 6616:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:36:51 2019][1663910.227682] req@ffff911fbf3fe600 x1624669493357376/t0(0) o36->bf617e6a-e2c4-2972-fed8-58b2cb638da2@10.9.101.25@o2ib4:298/0 lens 536/2888 e 3 to 0 dl 1549442253 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:36:51 2019][1663910.256753] Lustre: 6616:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages [Wed Feb 6 00:36:57 2019][1663916.596835] LustreError: 17224:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.114.10@tcp) failed to reply to blocking AST (req@ffff91567575ce00 x1623682050876016 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff913d1e58a1c0/0x21ed93518051444b lrc: 4/0,0 mode: PR/PR res: [0x2c0001689:0xb:0x0].0x0 bits 0x5b/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.10.114.10@tcp remote: 0xd3460787bfacf32f expref: 19 pid: 15228 timeout: 1664023 lvb_type: 0 [Wed Feb 6 00:36:57 2019][1663916.639553] LustreError: 17224:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 9 previous similar messages [Wed Feb 6 00:36:57 2019][1663916.649819] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.114.10@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:36:57 2019][1663916.662497] LustreError: Skipped 9 previous similar messages [Wed Feb 6 00:36:57 2019][1663916.668353] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.114.10@tcp ns: mdt-fir-MDT0002_UUID lock: ffff913d1e58a1c0/0x21ed93518051444b lrc: 3/0,0 mode: PR/PR res: [0x2c0001689:0xb:0x0].0x0 bits 0x5b/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.10.114.10@tcp remote: 0xd3460787bfacf32f expref: 20 pid: 15228 timeout: 0 lvb_type: 0 [Wed Feb 6 00:36:57 2019][1663916.705481] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 6 previous similar messages [Wed Feb 6 00:36:57 2019][1663916.715674] LustreError: 17224:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff914f19716400 ns: mdt-fir-MDT0002_UUID lock: ffff9154d1afcc80/0x21ed9351805144d0 lrc: 1/0,0 mode: EX/EX res: [0x2c0001689:0xb:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT flags: 0x54801000000000 nid: 10.10.114.10@tcp remote: 0xd3460787bfacf360 expref: 4 pid: 17224 timeout: 0 lvb_type: 3 [Wed Feb 6 00:36:57 2019][1663916.750034] LustreError: 17224:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message [Wed Feb 6 00:37:04 2019][1663924.112079] LustreError: 17203:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549441962, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9124841c72c0/0x21ed93518050d309 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 191 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 17203 timeout: 0 lvb_type: 0 [Wed Feb 6 00:37:05 2019][1663924.151547] LustreError: 17203:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 19 previous similar messages [Wed Feb 6 00:37:13 2019][1663932.996237] LNet: Service thread pid 15164 was inactive for 636.17s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:37:13 2019][1663933.013341] LNet: Skipped 1 previous similar message [Wed Feb 6 00:37:13 2019][1663933.018483] Pid: 15164, comm: mdt01_042 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:37:13 2019][1663933.028396] Call Trace: [Wed Feb 6 00:37:13 2019][1663933.031031] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:37:13 2019][1663933.038135] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 00:37:13 2019][1663933.045502] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 00:37:13 2019][1663933.053726] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:37:13 2019][1663933.060123] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:37:13 2019][1663933.066337] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:37:13 2019][1663933.073071] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:37:13 2019][1663933.078940] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:37:13 2019][1663933.086053] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:37:13 2019][1663933.093942] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:37:13 2019][1663933.100442] [] kthread+0xd1/0xe0 [Wed Feb 6 00:37:13 2019][1663933.105528] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:37:13 2019][1663933.112177] [] 0xffffffffffffffff [Wed Feb 6 00:37:13 2019][1663933.117370] LustreError: dumping log to /tmp/lustre-log.1549442271.15164 [Wed Feb 6 00:37:15 2019][1663935.044292] Pid: 8994, comm: mdt00_017 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:37:15 2019][1663935.054115] Call Trace: [Wed Feb 6 00:37:15 2019][1663935.056753] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:37:15 2019][1663935.063872] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:37:15 2019][1663935.071143] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:37:15 2019][1663935.077993] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:37:15 2019][1663935.084732] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:37:15 2019][1663935.091127] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:37:15 2019][1663935.097430] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:37:15 2019][1663935.104865] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:37:15 2019][1663935.111781] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:37:15 2019][1663935.119997] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:37:15 2019][1663935.126394] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:37:15 2019][1663935.132618] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:37:16 2019][1663935.139360] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:37:16 2019][1663935.145237] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:37:16 2019][1663935.152352] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:37:16 2019][1663935.160238] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:37:16 2019][1663935.166738] [] kthread+0xd1/0xe0 [Wed Feb 6 00:37:16 2019][1663935.171825] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:37:16 2019][1663935.178474] [] 0xffffffffffffffff [Wed Feb 6 00:37:16 2019][1663935.183667] LustreError: dumping log to /tmp/lustre-log.1549442273.8994 [Wed Feb 6 00:37:16 2019][1663935.358699] Pid: 15224, comm: mdt01_048 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:37:16 2019][1663935.368609] Call Trace: [Wed Feb 6 00:37:16 2019][1663935.371245] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:37:16 2019][1663935.378349] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 00:37:16 2019][1663935.385708] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 00:37:16 2019][1663935.393924] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:37:16 2019][1663935.400320] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:37:16 2019][1663935.406542] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:37:16 2019][1663935.413277] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:37:16 2019][1663935.419144] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:37:16 2019][1663935.426259] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:37:16 2019][1663935.434148] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:37:16 2019][1663935.440646] [] kthread+0xd1/0xe0 [Wed Feb 6 00:37:16 2019][1663935.445733] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:37:16 2019][1663935.452382] [] 0xffffffffffffffff [Wed Feb 6 00:37:17 2019][1663937.092345] Pid: 53422, comm: mdt02_027 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:37:17 2019][1663937.102255] Call Trace: [Wed Feb 6 00:37:17 2019][1663937.104894] [] 0xffffffffffffffff [Wed Feb 6 00:37:17 2019][1663937.110088] LustreError: dumping log to /tmp/lustre-log.1549442275.53422 [Wed Feb 6 00:37:18 2019][1663937.286086] Pid: 15183, comm: mdt02_031 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:37:18 2019][1663937.295993] Call Trace: [Wed Feb 6 00:37:18 2019][1663937.298629] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:37:18 2019][1663937.305733] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 00:37:18 2019][1663937.313100] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 00:37:18 2019][1663937.321316] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:37:18 2019][1663937.327704] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:37:18 2019][1663937.333917] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:37:18 2019][1663937.340653] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:37:18 2019][1663937.346519] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:37:18 2019][1663937.353636] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:37:18 2019][1663937.361521] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:37:18 2019][1663937.368021] [] kthread+0xd1/0xe0 [Wed Feb 6 00:37:18 2019][1663937.373100] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:37:18 2019][1663937.379739] [] 0xffffffffffffffff [Wed Feb 6 00:37:24 2019][1663943.236498] LNet: Service thread pid 8999 was inactive for 636.69s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:37:24 2019][1663943.249452] LustreError: dumping log to /tmp/lustre-log.1549442281.8999 [Wed Feb 6 00:37:26 2019][1663945.284552] LustreError: dumping log to /tmp/lustre-log.1549442284.6804 [Wed Feb 6 00:37:28 2019][1663947.332602] LustreError: dumping log to /tmp/lustre-log.1549442286.15182 [Wed Feb 6 00:37:32 2019][1663951.428703] LustreError: dumping log to /tmp/lustre-log.1549442290.15391 [Wed Feb 6 00:37:42 2019][1663961.668960] LustreError: dumping log to /tmp/lustre-log.1549442300.6684 [Wed Feb 6 00:38:00 2019][1663979.941437] Lustre: 8996:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:38:00 2019][1663979.941437] req@ffff9136fc213300 x1624705148805680/t0(0) o36->811fd087-e95e-3790-3b1f-499e971702dc@10.8.12.21@o2ibnterpret:/0/0 rc 0/0 [Wed Feb 6 00:38:00 2019][1663979.970426] Lustre: 8996:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Feb 6 00:38:06 2019][1663985.408056] Lustre: fir-MDT0000: Client 811fd087-e95e-3790-3b1f-499e971702dc (at 10.8.12.21@o2ib6) reconnecting [Wed Feb 6 00:38:06 2019][1663985.418319] Lustre: Skipped 5 previous similar messages [Wed Feb 6 00:38:13 2019][1663992.758734] LNetError: 6883:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.15.5@tcp from [Wed Feb 6 00:38:13 2019][1663992.769260] LNetError: 6883:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 156057130 previous similar messages [Wed Feb 6 00:38:19 2019][1663998.358888] LustreError: 7029:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549442037, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff9140bdbdee40/0x21ed935180511a75 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 35 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935180511a83 expref: -99 pid: 7029 timeout: 0 lvb_type: 0 [Wed Feb 6 00:38:19 2019][1663998.398968] LustreError: 7029:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 4 previous similar messages [Wed Feb 6 00:38:21 2019][1664000.780934] Lustre: 53422:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549442339/real 1549442339] req@ffff91512db98600 x1623682050920112/t0(0) o104->fir-MDT0000@10.10.14.7@tcp:15/16 lens 296/224 e 0 to 1 dl 1549442350 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:38:21 2019][1664000.808525] Lustre: 53422:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 158821996 previous similar messages [Wed Feb 6 00:38:41 2019][1664021.062450] LustreError: dumping log to /tmp/lustre-log.1549442359.15242 [Wed Feb 6 00:38:58 2019][1664037.446875] Lustre: 9182:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 00:38:58 2019][1664037.446875] req@ffff9124fa99bc00 x1624701768122704/t0(0) o36->24d9d171-71c4-5c61-0044-18351be88bb7@10.9.114.7@o2ib4:426/0 lens 520/2888 e 2 to 0 dl 1549442381 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:39:02 2019][1664041.600488] LNet: Service thread pid 53422 completed after 744.77s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 00:39:26 2019][1664065.645955] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 00:39:28 2019][1664067.600634] LustreError: 8985:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.114.10@tcp) failed to reply to blocking AST (req@ffff9124c60c9200 x1623682050944944 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff914481201440/0x21ed9351805240e5 lrc: 4/0,0 mode: CR/CR res: [0x2c0001689:0x6:0x0].0x0 bits 0x8/0x0 rrc: 6 type: IBT flags: 0x60000400000020 nid: 10.10.114.10@tcp remote: 0xd3460787bfacf383 expref: 7 pid: 17224 timeout: 1664174 lvb_type: 3 [Wed Feb 6 00:39:28 2019][1664067.643052] LustreError: 8985:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 5 previous similar messages [Wed Feb 6 00:39:28 2019][1664067.653231] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.114.10@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:39:28 2019][1664067.665922] LustreError: Skipped 5 previous similar messages [Wed Feb 6 00:39:28 2019][1664067.671786] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.114.10@tcp ns: mdt-fir-MDT0002_UUID lock: ffff914481201440/0x21ed9351805240e5 lrc: 3/0,0 mode: CR/CR res: [0x2c0001689:0x6:0x0].0x0 bits 0x8/0x0 rrc: 6 type: IBT flags: 0x60000400000020 nid: 10.10.114.10@tcp remote: 0xd3460787bfacf383 expref: 8 pid: 17224 timeout: 0 lvb_type: 3 [Wed Feb 6 00:39:28 2019][1664067.708742] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages [Wed Feb 6 00:39:28 2019][1664067.718925] LustreError: 8985:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff911a00b1b000 ns: mdt-fir-MDT0002_UUID lock: ffff9121a6d09200/0x21ed9351805240ec lrc: 1/0,0 mode: EX/EX res: [0x2c0001689:0x6:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT flags: 0x54801000000000 nid: 10.10.114.10@tcp remote: 0xd3460787bfacf38a expref: 4 pid: 8985 timeout: 0 lvb_type: 3 [Wed Feb 6 00:40:34 2019][1664133.587277] LustreError: 15239:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549442172, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff913599979200/0x21ed935180514b3d lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 42 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935180514b44 expref: -99 pid: 15239 timeout: 0 lvb_type: 0 [Wed Feb 6 00:40:34 2019][1664133.627529] LustreError: 15239:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Feb 6 00:41:25 2019][1664184.893562] LustreError: 9171:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549442223, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff91231764e0c0/0x21ed93518051577d lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 228 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 9171 timeout: 0 lvb_type: 0 [Wed Feb 6 00:41:25 2019][1664184.932856] LustreError: 9171:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 13 previous similar messages [Wed Feb 6 00:41:27 2019][1664186.266606] Lustre: 6171:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/4), not sending early reply [Wed Feb 6 00:41:27 2019][1664186.266606] req@ffff9135bc305700 x1624697873014592/t0(0) o36->f8c56e48-2583-434a-b50d-30254178caf9@10.9.103.20@o2ib4:574/0 lens 520/2888 e 1 to 0 dl 1549442529 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:41:31 2019][1664190.741779] Lustre: fir-MDT0000: Client ffafe5be-09a9-59b7-d26e-f32fde550fcb (at 10.10.15.6@tcp) reconnecting [Wed Feb 6 00:41:33 2019][1664192.603789] LustreError: 6883:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9128f1bbef00 x1623682051023840/t0(0) o105->fir-MDT0000@10.10.112.17@tcp:15/16 lens 304/224 e 0 to 1 dl 1549442542 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:41:33 2019][1664192.625835] LustreError: 6883:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [Wed Feb 6 00:41:33 2019][1664192.636303] LustreError: 6688:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff913586a13000 ns: mdt-fir-MDT0000_UUID lock: ffff9150b2b27740/0x21ed93518051748e lrc: 3/0,0 mode: PR/PR res: [0x200001760:0x2:0x0].0x0 bits 0x13/0x0 rrc: 10 type: IBT flags: 0x50200400000020 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349becd expref: 2 pid: 6688 timeout: 0 lvb_type: 0 [Wed Feb 6 00:41:36 2019][1664195.770834] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:41:36 2019][1664195.786019] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Feb 6 00:41:56 2019][1664216.084671] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 00:42:16 2019][1664236.107847] LNet: Service thread pid 6178 was inactive for 636.66s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:42:16 2019][1664236.124868] LNet: Skipped 4 previous similar messages [Wed Feb 6 00:42:16 2019][1664236.130106] Pid: 6178, comm: mdt03_002 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:42:16 2019][1664236.139928] Call Trace: [Wed Feb 6 00:42:16 2019][1664236.142564] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:42:16 2019][1664236.149677] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 00:42:17 2019][1664236.157036] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 00:42:17 2019][1664236.165253] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:42:17 2019][1664236.171638] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:42:17 2019][1664236.177861] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:42:17 2019][1664236.184595] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:42:17 2019][1664236.190473] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:42:17 2019][1664236.197596] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:42:17 2019][1664236.205484] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:42:17 2019][1664236.211984] [] kthread+0xd1/0xe0 [Wed Feb 6 00:42:17 2019][1664236.217071] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:42:17 2019][1664236.223718] [] 0xffffffffffffffff [Wed Feb 6 00:42:17 2019][1664236.228913] LustreError: dumping log to /tmp/lustre-log.1549442574.6178 [Wed Feb 6 00:42:17 2019][1664236.572461] Pid: 17223, comm: mdt03_028 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:42:17 2019][1664236.582372] Call Trace: [Wed Feb 6 00:42:17 2019][1664236.585010] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:42:17 2019][1664236.592130] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 00:42:17 2019][1664236.599488] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 00:42:17 2019][1664236.607705] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:42:17 2019][1664236.614101] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:42:17 2019][1664236.620315] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:42:17 2019][1664236.627050] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:42:17 2019][1664236.632915] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:42:17 2019][1664236.640031] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:42:17 2019][1664236.647919] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:42:17 2019][1664236.654419] [] kthread+0xd1/0xe0 [Wed Feb 6 00:42:17 2019][1664236.659506] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:42:17 2019][1664236.666153] [] 0xffffffffffffffff [Wed Feb 6 00:42:19 2019][1664238.155904] Pid: 17205, comm: mdt03_020 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:42:19 2019][1664238.165810] Call Trace: [Wed Feb 6 00:42:19 2019][1664238.168442] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:42:19 2019][1664238.175552] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 00:42:19 2019][1664238.182911] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 00:42:19 2019][1664238.191125] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:42:19 2019][1664238.197513] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:42:19 2019][1664238.203727] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:42:19 2019][1664238.210460] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:42:19 2019][1664238.216327] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:42:19 2019][1664238.223448] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:42:19 2019][1664238.231331] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:42:19 2019][1664238.237830] [] kthread+0xd1/0xe0 [Wed Feb 6 00:42:19 2019][1664238.242910] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:42:19 2019][1664238.249548] [] 0xffffffffffffffff [Wed Feb 6 00:42:19 2019][1664238.254725] LustreError: dumping log to /tmp/lustre-log.1549442576.17205 [Wed Feb 6 00:42:25 2019][1664244.300052] Pid: 17222, comm: mdt03_027 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:42:25 2019][1664244.309961] Call Trace: [Wed Feb 6 00:42:25 2019][1664244.312598] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:42:25 2019][1664244.319719] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 00:42:25 2019][1664244.327076] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 00:42:25 2019][1664244.335293] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:42:25 2019][1664244.341681] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:42:25 2019][1664244.347893] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:42:25 2019][1664244.354630] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:42:25 2019][1664244.360496] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:42:25 2019][1664244.367612] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:42:25 2019][1664244.375499] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:42:25 2019][1664244.381998] [] kthread+0xd1/0xe0 [Wed Feb 6 00:42:25 2019][1664244.387086] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:42:25 2019][1664244.393735] [] 0xffffffffffffffff [Wed Feb 6 00:42:25 2019][1664244.398927] LustreError: dumping log to /tmp/lustre-log.1549442583.17222 [Wed Feb 6 00:42:41 2019][1664260.684470] LNet: Service thread pid 17203 was inactive for 636.56s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:42:41 2019][1664260.701572] LNet: Skipped 3 previous similar messages [Wed Feb 6 00:42:41 2019][1664260.706808] Pid: 17203, comm: mdt03_018 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:42:41 2019][1664260.716719] Call Trace: [Wed Feb 6 00:42:41 2019][1664260.719355] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:42:41 2019][1664260.726467] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 00:42:41 2019][1664260.733826] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 00:42:41 2019][1664260.742043] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:42:41 2019][1664260.748430] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:42:41 2019][1664260.754643] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:42:41 2019][1664260.761376] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:42:41 2019][1664260.767245] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:42:41 2019][1664260.774362] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:42:41 2019][1664260.782248] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:42:41 2019][1664260.788747] [] kthread+0xd1/0xe0 [Wed Feb 6 00:42:41 2019][1664260.793825] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:42:41 2019][1664260.800466] [] 0xffffffffffffffff [Wed Feb 6 00:42:41 2019][1664260.805660] LustreError: dumping log to /tmp/lustre-log.1549442599.17203 [Wed Feb 6 00:43:18 2019][1664297.549398] LNet: Service thread pid 17207 was inactive for 636.27s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:43:18 2019][1664297.562433] LNet: Skipped 6 previous similar messages [Wed Feb 6 00:43:18 2019][1664297.567668] LustreError: dumping log to /tmp/lustre-log.1549442636.17207 [Wed Feb 6 00:44:00 2019][1664340.046464] Lustre: 17220:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-80), not sending early reply [Wed Feb 6 00:44:00 2019][1664340.046464] req@ffff914f2ae56000 x1624697819519648/t0(0) o36->62faecbe-6e06-a70f-dde9-8810f97cf707@10.9.103.17@o2ib4:728/0 lens 520/2888 e 0 to 0 dl 1549442683 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:44:00 2019][1664340.075790] Lustre: 17220:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 23 previous similar messages [Wed Feb 6 00:44:04 2019][1664343.607547] LustreError: 6617:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.3@tcp) failed to reply to completion AST (req@ffff9150a1ea8300 x1623682051110272 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff91250cbd9200/0x21ed93518053d032 lrc: 4/0,0 mode: PR/PR res: [0x200001760:0x2:0x0].0x0 bits 0x2/0x0 rrc: 14 type: IBT flags: 0x60000000000020 nid: 10.10.113.3@tcp remote: 0x4d977b31312bda61 expref: 6 pid: 6814 timeout: 1664450 lvb_type: 0 [Wed Feb 6 00:44:04 2019][1664343.607554] LustreError: 138-a: fir-MDT0000: A client on nid 10.10.15.5@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:44:04 2019][1664343.607557] LustreError: Skipped 4 previous similar messages [Wed Feb 6 00:44:04 2019][1664343.607575] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.15.5@tcp ns: mdt-fir-MDT0000_UUID lock: ffff91397bdc18c0/0x21ed93518053d970 lrc: 3/0,0 mode: PR/PR res: [0x20000173d:0x4e4a:0x0].0x0 bits 0x13/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.10.15.5@tcp remote: 0xfe5e771a1349bf3d expref: 10 pid: 15224 timeout: 0 lvb_type: 0 [Wed Feb 6 00:44:04 2019][1664343.607577] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 4 previous similar messages [Wed Feb 6 00:44:04 2019][1664343.715150] LustreError: 6617:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 9 previous similar messages [Wed Feb 6 00:44:04 2019][1664343.725403] Lustre: 6617:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (590:165s); client may timeout. req@ffff913da7905d00 x1624701947442288/t14368177930(0) o36->7646fd6e-8364-5414-6410-5e73509a82fd@10.10.113.3@tcp:562/0 lens 520/424 e 0 to 0 dl 1549442517 ref 1 fl Complete:/0/0 rc 0/0 [Wed Feb 6 00:44:04 2019][1664343.725601] LustreError: 8947:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff915613bd3c00 ns: mdt-fir-MDT0000_UUID lock: ffff9140b24cde80/0x21ed93518054b381 lrc: 3/0,0 mode: PR/PR res: [0x200001760:0x2:0x0].0x0 bits 0x13/0x0 rrc: 12 type: IBT flags: 0x50200000000000 nid: 10.10.15.6@tcp remote: 0x83214b66a5c7bfb4 expref: 2 pid: 8947 timeout: 0 lvb_type: 0 [Wed Feb 6 00:44:04 2019][1664343.725604] LustreError: 8947:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 3 previous similar messages [Wed Feb 6 00:44:06 2019][1664346.026469] Lustre: fir-MDT0000: Client 62faecbe-6e06-a70f-dde9-8810f97cf707 (at 10.9.103.17@o2ib4) reconnecting [Wed Feb 6 00:44:06 2019][1664346.036818] Lustre: Skipped 24 previous similar messages [Wed Feb 6 00:44:32 2019][1664371.648023] Lustre: fir-MDT0002: Connection restored to 7e45f28f-833d-2544-2adc-dcb031f529b8 (at 10.10.114.10@tcp) [Wed Feb 6 00:44:32 2019][1664371.658553] Lustre: Skipped 92 previous similar messages [Wed Feb 6 00:44:58 2019][1664397.517908] LustreError: 6613:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549442436, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff91299a7072c0/0x21ed93518053f729 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 45 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518053f730 expref: -99 pid: 6613 timeout: 0 lvb_type: 0 [Wed Feb 6 00:44:58 2019][1664397.557983] LustreError: 6613:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 29 previous similar messages [Wed Feb 6 00:45:00 2019][1664399.385476] Lustre: fir-MDT0002: haven't heard from client 0e69550c-a059-5e4e-973b-b3107042840e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9138dcd48000, cur 1549442738 expire 1549442588 last 1549442511 [Wed Feb 6 00:45:00 2019][1664399.407353] Lustre: Skipped 5 previous similar messages [Wed Feb 6 00:46:35 2019][1664494.612695] LNet: Service thread pid 17223 completed after 896.94s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 00:46:35 2019][1664494.629032] LNet: Skipped 14 previous similar messages [Wed Feb 6 00:46:35 2019][1664494.641460] Lustre: 17052:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (717:79s); client may timeout. req@ffff91203241d100 x1624702125714240/t0(0) o36->97d93834-3c87-3fed-069a-aff0074e2ab1@10.10.112.12@tcp:44/0 lens 520/424 e 0 to 0 dl 1549442754 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 00:47:56 2019][1664575.591855] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 00:48:13 2019][1664592.773790] LNetError: 15372:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 00:48:13 2019][1664592.784568] LNetError: 15372:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 188314604 previous similar messages [Wed Feb 6 00:48:15 2019][1664594.334928] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:48:21 2019][1664600.795990] Lustre: 8995:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549442939/real 1549442939] req@ffff913b2fcffb00 x1623682051211792/t0(0) o104->fir-MDT0002@10.10.113.15@tcp:15/16 lens 296/224 e 0 to 1 dl 1549442950 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:48:21 2019][1664600.823669] Lustre: 8995:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 189582146 previous similar messages [Wed Feb 6 00:48:23 2019][1664602.965066] Lustre: 8998:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-142), not sending early reply [Wed Feb 6 00:48:23 2019][1664602.965066] req@ffff9137678f6f00 x1624702125716512/t0(0) o36->97d93834-3c87-3fed-069a-aff0074e2ab1@10.10.112.12@tcp:236/0 lens 520/2888 e 0 to 0 dl 1549442946 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:48:23 2019][1664602.994310] Lustre: 8998:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 17 previous similar messages [Wed Feb 6 00:48:29 2019][1664608.717747] Lustre: fir-MDT0002: Client 97d93834-3c87-3fed-069a-aff0074e2ab1 (at 10.10.112.12@tcp) reconnecting [Wed Feb 6 00:48:29 2019][1664608.728033] Lustre: Skipped 14 previous similar messages [Wed Feb 6 00:49:06 2019][1664645.615444] LustreError: 15407:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff911f5aa5f800 ns: mdt-fir-MDT0000_UUID lock: ffff912750fe2f40/0x21ed93518054d998 lrc: 3/0,0 mode: PR/PR res: [0x2000017c7:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT flags: 0x50200000000000 nid: 10.10.113.3@tcp remote: 0x4d977b31312bdb95 expref: 3 pid: 15407 timeout: 0 lvb_type: 0 [Wed Feb 6 00:49:06 2019][1664645.624073] Lustre: 6615:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (746:160s); client may timeout. req@ffff91241cdd0300 x1624705211616272/t0(0) o36->2d2b361b-8ebd-4214-998a-1d2d5142855f@10.10.29.6@tcp:114/0 lens 520/424 e 0 to 0 dl 1549442824 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 00:49:06 2019][1664645.678343] LustreError: 15407:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 6 previous similar messages [Wed Feb 6 00:51:35 2019][1664794.645868] LustreError: 8971:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549442833, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff91357a38ca40/0x21ed93518054d688 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 227 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 8971 timeout: 0 lvb_type: 0 [Wed Feb 6 00:51:35 2019][1664794.685167] LustreError: 8971:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 36 previous similar messages [Wed Feb 6 00:51:37 2019][1664796.622294] Lustre: 15218:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (717:247s); client may timeout. req@ffff9129ac091b00 x1624705272438912/t0(0) o36->82eeb05c-9f6a-d608-7f3e-29db42f96cfa@10.10.14.7@tcp:178/0 lens 520/424 e 0 to 0 dl 1549442888 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 00:53:09 2019][1664888.858065] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 00:53:09 2019][1664888.869108] Lustre: Skipped 2 previous similar messages [Wed Feb 6 00:54:06 2019][1664945.631655] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:54:06 2019][1664945.646781] LustreError: 9030:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549442984, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff91472fddaf40/0x21ed93518055b0fb lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 49 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518055b102 expref: -99 pid: 9030 timeout: 0 lvb_type: 0 [Wed Feb 6 00:54:06 2019][1664945.686865] LustreError: 9030:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 10 previous similar messages [Wed Feb 6 00:54:08 2019][1664947.622707] LustreError: 48703:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.29.8@tcp) failed to reply to blocking AST (req@ffff91374c038300 x1623682051393536 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff912696d557c0/0x21ed9351806259bc lrc: 4/0,0 mode: PR/PR res: [0x2000017df:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d5a87 expref: 6 pid: 6804 timeout: 1665054 lvb_type: 0 [Wed Feb 6 00:54:08 2019][1664947.622724] LustreError: 138-a: fir-MDT0000: A client on nid 10.10.113.3@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 00:54:08 2019][1664947.622726] LustreError: Skipped 20 previous similar messages [Wed Feb 6 00:54:08 2019][1664947.622747] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.3@tcp ns: mdt-fir-MDT0000_UUID lock: ffff9140cd25c800/0x21ed93518055df5b lrc: 3/0,0 mode: PR/PR res: [0x20000173d:0x4e4a:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.113.3@tcp remote: 0x4d977b31312bdc28 expref: 12 pid: 15406 timeout: 0 lvb_type: 0 [Wed Feb 6 00:54:08 2019][1664947.622749] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 17 previous similar messages [Wed Feb 6 00:54:08 2019][1664947.623192] Lustre: 15260:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:302s); client may timeout. req@ffff912da1446300 x1624701936948272/t0(0) o36->95f41e85-91f0-6aee-d80e-dc57a1b83666@10.10.113.15@tcp:274/0 lens 520/424 e 0 to 0 dl 1549442984 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 00:54:08 2019][1664947.759291] LustreError: 48703:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 17 previous similar messages [Wed Feb 6 00:55:51 2019][1665050.395450] Lustre: fir-MDT0002: Connection restored to ac3968e7-fb77-be3e-6d47-52fa292f2074 (at 10.8.24.9@o2ib6) [Wed Feb 6 00:55:51 2019][1665050.405881] Lustre: Skipped 96 previous similar messages [Wed Feb 6 00:56:36 2019][1665095.751886] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 00:56:36 2019][1665095.762926] Lustre: Skipped 1 previous similar message [Wed Feb 6 00:57:01 2019][1665120.866049] LNet: Service thread pid 6616 was inactive for 1204.07s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:57:01 2019][1665120.883159] Pid: 6616, comm: mdt00_005 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:57:01 2019][1665120.892988] Call Trace: [Wed Feb 6 00:57:01 2019][1665120.895634] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:57:01 2019][1665120.902753] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:57:01 2019][1665120.910024] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:57:01 2019][1665120.916870] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:57:01 2019][1665120.923613] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:57:01 2019][1665120.930010] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:57:01 2019][1665120.936318] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:57:01 2019][1665120.943754] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:57:01 2019][1665120.950662] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:57:01 2019][1665120.958878] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:57:01 2019][1665120.965267] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:57:01 2019][1665120.971490] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:57:01 2019][1665120.978223] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:57:01 2019][1665120.984102] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:57:01 2019][1665120.991232] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:57:01 2019][1665120.999121] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:57:01 2019][1665121.005618] [] kthread+0xd1/0xe0 [Wed Feb 6 00:57:01 2019][1665121.010706] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:57:01 2019][1665121.017346] [] 0xffffffffffffffff [Wed Feb 6 00:57:01 2019][1665121.022538] LustreError: dumping log to /tmp/lustre-log.1549443459.6616 [Wed Feb 6 00:57:02 2019][1665121.648376] Pid: 6758, comm: mdt03_005 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:57:02 2019][1665121.658206] Call Trace: [Wed Feb 6 00:57:02 2019][1665121.660839] [] 0xffffffffffffffff [Wed Feb 6 00:57:30 2019][1665149.538769] LNet: Service thread pid 8970 was inactive for 1201.53s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 00:57:30 2019][1665149.555883] LNet: Skipped 1 previous similar message [Wed Feb 6 00:57:30 2019][1665149.561029] Pid: 8970, comm: mdt01_017 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:57:30 2019][1665149.570854] Call Trace: [Wed Feb 6 00:57:30 2019][1665149.573494] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665149.580611] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665149.587893] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665149.594729] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:57:30 2019][1665149.601474] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:57:30 2019][1665149.607870] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:57:30 2019][1665149.614179] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:57:30 2019][1665149.621616] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:57:30 2019][1665149.628523] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:57:30 2019][1665149.636738] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:57:30 2019][1665149.643125] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:57:30 2019][1665149.649341] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:57:30 2019][1665149.656074] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:57:30 2019][1665149.661942] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665149.669068] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665149.676958] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665149.683461] [] kthread+0xd1/0xe0 [Wed Feb 6 00:57:30 2019][1665149.688547] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:57:30 2019][1665149.695196] [] 0xffffffffffffffff [Wed Feb 6 00:57:30 2019][1665149.700391] LustreError: dumping log to /tmp/lustre-log.1549443488.8970 [Wed Feb 6 00:57:30 2019][1665149.780067] Pid: 6168, comm: mdt00_001 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:57:30 2019][1665149.789893] Call Trace: [Wed Feb 6 00:57:30 2019][1665149.792534] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665149.799650] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665149.806922] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665149.813764] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:57:30 2019][1665149.820510] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:57:30 2019][1665149.826907] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:57:30 2019][1665149.833215] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:57:30 2019][1665149.840653] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:57:30 2019][1665149.847568] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:57:30 2019][1665149.855786] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:57:30 2019][1665149.862181] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:57:30 2019][1665149.868403] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:57:30 2019][1665149.875147] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:57:30 2019][1665149.881013] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665149.888143] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665149.896035] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665149.902533] [] kthread+0xd1/0xe0 [Wed Feb 6 00:57:30 2019][1665149.907624] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:57:30 2019][1665149.914275] [] 0xffffffffffffffff [Wed Feb 6 00:57:30 2019][1665149.919473] Pid: 9199, comm: mdt02_023 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 00:57:30 2019][1665149.929298] Call Trace: [Wed Feb 6 00:57:30 2019][1665149.931931] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665149.939040] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665149.946311] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665149.953157] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 00:57:30 2019][1665149.959893] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 00:57:30 2019][1665149.966288] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 00:57:30 2019][1665149.972588] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 00:57:30 2019][1665149.980025] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 00:57:30 2019][1665149.986933] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 00:57:30 2019][1665149.995147] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 00:57:30 2019][1665150.001544] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 00:57:30 2019][1665150.007769] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 00:57:30 2019][1665150.014501] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 00:57:30 2019][1665150.020370] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665150.027486] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665150.035373] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 00:57:30 2019][1665150.041872] [] kthread+0xd1/0xe0 [Wed Feb 6 00:57:30 2019][1665150.046950] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 00:57:30 2019][1665150.053602] [] 0xffffffffffffffff [Wed Feb 6 00:58:03 2019][1665182.307595] LNet: Service thread pid 6169 was inactive for 1201.54s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 00:58:03 2019][1665182.320628] LNet: Skipped 1 previous similar message [Wed Feb 6 00:58:03 2019][1665182.325777] LustreError: dumping log to /tmp/lustre-log.1549443521.6169 [Wed Feb 6 00:58:07 2019][1665186.403696] LustreError: dumping log to /tmp/lustre-log.1549443525.15409 [Wed Feb 6 00:58:13 2019][1665192.788841] LNetError: 6804:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.115.6@tcp from [Wed Feb 6 00:58:13 2019][1665192.799447] LNetError: 6804:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 133720963 previous similar messages [Wed Feb 6 00:58:21 2019][1665200.811046] Lustre: 6804:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549443539/real 1549443539] req@ffff913b3418e600 x1623682051522752/t0(0) o104->fir-MDT0002@10.10.115.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549443550 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 00:58:21 2019][1665200.838651] Lustre: 6804:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 132601699 previous similar messages [Wed Feb 6 00:58:48 2019][1665227.364723] LustreError: dumping log to /tmp/lustre-log.1549443566.6827 [Wed Feb 6 00:58:56 2019][1665235.556935] LustreError: dumping log to /tmp/lustre-log.1549443574.15232 [Wed Feb 6 00:59:04 2019][1665243.749130] LustreError: dumping log to /tmp/lustre-log.1549443582.15329 [Wed Feb 6 00:59:05 2019][1665244.773172] Lustre: 8969:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 00:59:05 2019][1665244.773172] req@ffff9128161b6900 x1624672917233168/t0(0) o36->d405c92c-c32a-78c2-b321-55f4db7bb864@10.9.102.14@o2ib4:123/0 lens 528/2888 e 0 to 0 dl 1549443588 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 00:59:05 2019][1665244.802498] Lustre: 8969:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 65 previous similar messages [Wed Feb 6 00:59:08 2019][1665247.640265] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 00:59:08 2019][1665247.845241] LustreError: dumping log to /tmp/lustre-log.1549443586.38457 [Wed Feb 6 00:59:11 2019][1665250.699455] Lustre: fir-MDT0002: Client 797f0075-ec92-4d37-f23e-cc9ca768ea89 (at 10.9.113.5@o2ib4) reconnecting [Wed Feb 6 00:59:11 2019][1665250.709717] Lustre: Skipped 65 previous similar messages [Wed Feb 6 00:59:12 2019][1665251.941341] LustreError: dumping log to /tmp/lustre-log.1549443590.8994 [Wed Feb 6 00:59:20 2019][1665260.133544] LustreError: dumping log to /tmp/lustre-log.1549443598.9033 [Wed Feb 6 00:59:29 2019][1665268.325750] LustreError: dumping log to /tmp/lustre-log.1549443607.15389 [Wed Feb 6 00:59:33 2019][1665272.631062] Lustre: 6758:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:601s); client may timeout. req@ffff913d4c084800 x1624702092166752/t13599919677(0) o36->05683605-3c7a-66e7-65c2-872123284fcd@10.10.114.15@tcp:300/0 lens 520/424 e 0 to 0 dl 1549443010 ref 1 fl Complete:/0/0 rc 0/0 [Wed Feb 6 00:59:33 2019][1665272.631546] LNet: Service thread pid 6616 completed after 1355.83s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 00:59:33 2019][1665272.631548] LNet: Skipped 2 previous similar messages [Wed Feb 6 00:59:41 2019][1665280.614071] LustreError: dumping log to /tmp/lustre-log.1549443619.48673 [Wed Feb 6 01:00:01 2019][1665301.094567] LustreError: dumping log to /tmp/lustre-log.1549443639.8985 [Wed Feb 6 01:00:34 2019][1665333.863393] LustreError: dumping log to /tmp/lustre-log.1549443672.15277 [Wed Feb 6 01:00:38 2019][1665337.959501] LustreError: dumping log to /tmp/lustre-log.1549443676.15330 [Wed Feb 6 01:01:15 2019][1665374.824434] LustreError: dumping log to /tmp/lustre-log.1549443713.15185 [Wed Feb 6 01:01:36 2019][1665395.304943] LustreError: dumping log to /tmp/lustre-log.1549443734.15164 [Wed Feb 6 01:01:51 2019][1665410.631329] LustreError: 15281:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549443449, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9121317c57c0/0x21ed935180680dac lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 248 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 15281 timeout: 0 lvb_type: 0 [Wed Feb 6 01:01:51 2019][1665410.670800] LustreError: 15281:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 25 previous similar messages [Wed Feb 6 01:02:03 2019][1665422.724827] LNet: Service thread pid 6168 completed after 1474.53s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:02:03 2019][1665422.741157] LNet: Skipped 3 previous similar messages [Wed Feb 6 01:02:24 2019][1665443.635284] LustreError: 8969:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff915209ec5c00 ns: mdt-fir-MDT0002_UUID lock: ffff911ed7708480/0x21ed93518068e325 lrc: 1/0,0 mode: --/PW res: [0x2c0001689:0x7:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x54a01000000000 nid: 10.10.114.10@tcp remote: 0xd3460787bfacf7c0 expref: 5 pid: 8969 timeout: 0 lvb_type: 0 [Wed Feb 6 01:03:18 2019][1665497.707508] LNet: Service thread pid 15393 was inactive for 1203.34s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 01:03:18 2019][1665497.724709] LNet: Skipped 2 previous similar messages [Wed Feb 6 01:03:18 2019][1665497.729945] Pid: 15393, comm: mdt01_076 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:03:18 2019][1665497.739854] Call Trace: [Wed Feb 6 01:03:18 2019][1665497.742496] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:03:18 2019][1665497.749604] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:03:18 2019][1665497.756873] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:03:18 2019][1665497.763722] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:03:18 2019][1665497.770456] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:03:18 2019][1665497.776851] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:03:18 2019][1665497.783163] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:03:18 2019][1665497.790596] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:03:18 2019][1665497.797513] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:03:18 2019][1665497.805730] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:03:18 2019][1665497.812126] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:03:18 2019][1665497.818347] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:03:18 2019][1665497.825091] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:03:18 2019][1665497.830960] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:03:18 2019][1665497.838075] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:03:18 2019][1665497.845961] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:03:18 2019][1665497.852461] [] kthread+0xd1/0xe0 [Wed Feb 6 01:03:18 2019][1665497.857549] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:03:18 2019][1665497.864206] [] 0xffffffffffffffff [Wed Feb 6 01:03:18 2019][1665497.869397] LustreError: dumping log to /tmp/lustre-log.1549443836.15393 [Wed Feb 6 01:04:07 2019][1665546.860737] LNet: Service thread pid 17218 was inactive for 1203.21s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 01:04:07 2019][1665546.877963] Pid: 17218, comm: mdt03_023 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:04:07 2019][1665546.887882] Call Trace: [Wed Feb 6 01:04:07 2019][1665546.890527] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:04:07 2019][1665546.897644] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:04:07 2019][1665546.905010] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:04:07 2019][1665546.913238] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:04:07 2019][1665546.919641] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:04:07 2019][1665546.925873] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:04:07 2019][1665546.932624] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:04:07 2019][1665546.938501] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:04:07 2019][1665546.945633] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:04:07 2019][1665546.953528] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:04:07 2019][1665546.960028] [] kthread+0xd1/0xe0 [Wed Feb 6 01:04:07 2019][1665546.965116] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:04:07 2019][1665546.971763] [] 0xffffffffffffffff [Wed Feb 6 01:04:07 2019][1665546.976965] LustreError: dumping log to /tmp/lustre-log.1549443885.17218 [Wed Feb 6 01:04:07 2019][1665547.067410] Pid: 9015, comm: mdt01_031 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:04:07 2019][1665547.077239] Call Trace: [Wed Feb 6 01:04:07 2019][1665547.079878] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:04:07 2019][1665547.086996] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:04:07 2019][1665547.094389] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:04:07 2019][1665547.102606] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:04:07 2019][1665547.109002] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:04:07 2019][1665547.115225] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:04:07 2019][1665547.121966] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:04:07 2019][1665547.127850] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:04:07 2019][1665547.134966] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:04:07 2019][1665547.142869] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:04:07 2019][1665547.149371] [] kthread+0xd1/0xe0 [Wed Feb 6 01:04:07 2019][1665547.154470] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:04:07 2019][1665547.161113] [] 0xffffffffffffffff [Wed Feb 6 01:04:07 2019][1665547.166332] Pid: 6171, comm: mdt01_001 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:04:08 2019][1665547.176154] Call Trace: [Wed Feb 6 01:04:08 2019][1665547.178783] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:04:08 2019][1665547.185897] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:04:08 2019][1665547.193261] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:04:08 2019][1665547.201491] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:04:08 2019][1665547.207881] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:04:08 2019][1665547.214109] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:04:08 2019][1665547.220845] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:04:08 2019][1665547.226715] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:04:08 2019][1665547.233844] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:04:08 2019][1665547.241738] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:04:08 2019][1665547.248258] [] kthread+0xd1/0xe0 [Wed Feb 6 01:04:08 2019][1665547.253348] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:04:08 2019][1665547.260001] [] 0xffffffffffffffff [Wed Feb 6 01:04:33 2019][1665572.670967] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 01:04:33 2019][1665572.682009] Lustre: Skipped 2 previous similar messages [Wed Feb 6 01:04:34 2019][1665573.638415] LustreError: 8970:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.115.6@tcp) failed to reply to blocking AST (req@ffff912c7f658c00 x1623682051625536 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff912455ec5c40/0x21ed93518068e046 lrc: 4/0,0 mode: PR/PR res: [0x2c000168f:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.10.115.6@tcp remote: 0x9795773f7ddd3315 expref: 9 pid: 32929 timeout: 1665680 lvb_type: 0 [Wed Feb 6 01:04:34 2019][1665573.657374] LNet: Service thread pid 6169 completed after 1592.88s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:04:34 2019][1665573.697065] LustreError: 8970:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 5 previous similar messages [Wed Feb 6 01:04:34 2019][1665573.707238] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.115.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 01:04:34 2019][1665573.719832] LustreError: Skipped 7 previous similar messages [Wed Feb 6 01:04:34 2019][1665573.725683] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.115.6@tcp ns: mdt-fir-MDT0002_UUID lock: ffff912455ec5c40/0x21ed93518068e046 lrc: 3/0,0 mode: PR/PR res: [0x2c000168f:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.10.115.6@tcp remote: 0x9795773f7ddd3315 expref: 10 pid: 32929 timeout: 0 lvb_type: 0 [Wed Feb 6 01:04:34 2019][1665573.762673] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 8 previous similar messages [Wed Feb 6 01:04:35 2019][1665574.245427] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 01:04:35 2019][1665574.260558] LustreError: 8973:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549443612, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff913f4f344a40/0x21ed93518068dfb3 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 37 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518068dfba expref: -99 pid: 8973 timeout: 0 lvb_type: 0 [Wed Feb 6 01:04:35 2019][1665574.300633] LustreError: 8973:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 20 previous similar messages [Wed Feb 6 01:04:55 2019][1665594.639102] LustreError: 8969:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9122a9e86c00 ns: mdt-fir-MDT0002_UUID lock: ffff91212f2945c0/0x21ed935180690a46 lrc: 5/0,0 mode: PW/PW res: [0x2c0001689:0xb:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x50200000000000 nid: 10.10.114.10@tcp remote: 0xd3460787bfacf892 expref: 4 pid: 8969 timeout: 0 lvb_type: 0 [Wed Feb 6 01:06:39 2019][1665698.416542] LNet: Service thread pid 8971 was inactive for 1203.74s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 01:06:39 2019][1665698.433649] LNet: Skipped 2 previous similar messages [Wed Feb 6 01:06:39 2019][1665698.438878] Pid: 8971, comm: mdt01_018 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:06:39 2019][1665698.448704] Call Trace: [Wed Feb 6 01:06:39 2019][1665698.451339] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:06:39 2019][1665698.458460] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:06:39 2019][1665698.465826] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:06:39 2019][1665698.474043] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:06:39 2019][1665698.480439] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:06:39 2019][1665698.486655] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:06:39 2019][1665698.493387] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:06:39 2019][1665698.499254] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:06:39 2019][1665698.506378] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:06:39 2019][1665698.514267] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:06:39 2019][1665698.520767] [] kthread+0xd1/0xe0 [Wed Feb 6 01:06:39 2019][1665698.525853] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:06:39 2019][1665698.532501] [] 0xffffffffffffffff [Wed Feb 6 01:06:39 2019][1665698.537695] LustreError: dumping log to /tmp/lustre-log.1549444037.8971 [Wed Feb 6 01:06:44 2019][1665703.696304] Lustre: fir-MDT0000: Connection restored to 0ad7fb7a-17c5-3154-8262-b692b6d534dc (at 10.9.112.15@o2ib4) [Wed Feb 6 01:06:44 2019][1665703.706912] Lustre: Skipped 71 previous similar messages [Wed Feb 6 01:07:05 2019][1665724.642778] LNet: Service thread pid 15164 completed after 1531.92s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:07:05 2019][1665724.659198] LNet: Skipped 33 previous similar messages [Wed Feb 6 01:07:20 2019][1665739.377566] LustreError: dumping log to /tmp/lustre-log.1549444078.17205 [Wed Feb 6 01:07:26 2019][1665745.642877] LustreError: 15229:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff913d828c3000 ns: mdt-fir-MDT0002_UUID lock: ffff912c9a078fc0/0x21ed935180885499 lrc: 1/0,0 mode: --/PW res: [0x2c0001689:0xb:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT flags: 0x54a01000000000 nid: 10.10.114.10@tcp remote: 0xd3460787bfacf909 expref: 4 pid: 15229 timeout: 0 lvb_type: 0 [Wed Feb 6 01:08:13 2019][1665792.803905] LNetError: 15392:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.112.13@tcp from [Wed Feb 6 01:08:13 2019][1665792.814691] LNetError: 15392:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 110804017 previous similar messages [Wed Feb 6 01:08:21 2019][1665800.826101] Lustre: 8829:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549444139/real 1549444139] req@ffff91512db99200 x1623682051738240/t0(0) o104->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/224 e 0 to 1 dl 1549444150 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 01:08:21 2019][1665800.853788] Lustre: 8829:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 111569155 previous similar messages [Wed Feb 6 01:08:50 2019][1665829.491833] LNet: Service thread pid 6688 was inactive for 1202.22s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 01:08:50 2019][1665829.508938] Pid: 6688, comm: mdt02_005 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:08:50 2019][1665829.518767] Call Trace: [Wed Feb 6 01:08:50 2019][1665829.521410] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:08:50 2019][1665829.528539] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:08:50 2019][1665829.535905] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:08:50 2019][1665829.544131] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:08:50 2019][1665829.550527] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:08:50 2019][1665829.556749] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:08:50 2019][1665829.563492] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:08:50 2019][1665829.569368] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:08:50 2019][1665829.576493] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:08:50 2019][1665829.584380] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:08:50 2019][1665829.590878] [] kthread+0xd1/0xe0 [Wed Feb 6 01:08:50 2019][1665829.595968] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:08:50 2019][1665829.602615] [] 0xffffffffffffffff [Wed Feb 6 01:08:50 2019][1665829.607808] LustreError: dumping log to /tmp/lustre-log.1549444168.6688 [Wed Feb 6 01:08:54 2019][1665833.587935] Pid: 6682, comm: mdt03_003 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:08:54 2019][1665833.597757] Call Trace: [Wed Feb 6 01:08:54 2019][1665833.600395] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:08:54 2019][1665833.607516] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:08:54 2019][1665833.614874] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:08:54 2019][1665833.623091] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:08:54 2019][1665833.629485] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:08:54 2019][1665833.635709] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:08:54 2019][1665833.642452] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:08:54 2019][1665833.648327] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:08:54 2019][1665833.655452] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:08:54 2019][1665833.663342] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:08:54 2019][1665833.669840] [] kthread+0xd1/0xe0 [Wed Feb 6 01:08:54 2019][1665833.674927] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:08:54 2019][1665833.681574] [] 0xffffffffffffffff [Wed Feb 6 01:08:54 2019][1665833.686777] LustreError: dumping log to /tmp/lustre-log.1549444172.6682 [Wed Feb 6 01:09:02 2019][1665841.780141] Pid: 15230, comm: mdt01_050 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:09:02 2019][1665841.790053] Call Trace: [Wed Feb 6 01:09:02 2019][1665841.792700] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:09:02 2019][1665841.799809] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:09:02 2019][1665841.807174] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:09:02 2019][1665841.815399] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:09:02 2019][1665841.821806] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:09:02 2019][1665841.828028] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:09:02 2019][1665841.834770] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:09:02 2019][1665841.840648] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:09:02 2019][1665841.847761] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:09:02 2019][1665841.855650] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:09:02 2019][1665841.862149] [] kthread+0xd1/0xe0 [Wed Feb 6 01:09:02 2019][1665841.867236] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:09:02 2019][1665841.873884] [] 0xffffffffffffffff [Wed Feb 6 01:09:02 2019][1665841.879077] LustreError: dumping log to /tmp/lustre-log.1549444180.15230 [Wed Feb 6 01:09:02 2019][1665841.967918] Pid: 8996, comm: mdt01_021 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:09:02 2019][1665841.977741] Call Trace: [Wed Feb 6 01:09:02 2019][1665841.980379] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:09:02 2019][1665841.987499] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:09:02 2019][1665841.994864] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:09:02 2019][1665842.003089] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:09:02 2019][1665842.009486] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:09:02 2019][1665842.015710] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:09:02 2019][1665842.022444] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:09:02 2019][1665842.028311] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:09:02 2019][1665842.035426] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:09:02 2019][1665842.043314] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:09:02 2019][1665842.049813] [] kthread+0xd1/0xe0 [Wed Feb 6 01:09:02 2019][1665842.054899] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:09:02 2019][1665842.061549] [] 0xffffffffffffffff [Wed Feb 6 01:09:10 2019][1665849.460346] Lustre: 9006:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/-151), not sending early reply [Wed Feb 6 01:09:10 2019][1665849.460346] req@ffff9129ac091200 x1624702076481104/t0(0) o36->0161d142-b9e6-5768-fef0-f19a8b36b0dc@10.10.114.10@tcp:727/0 lens 520/2888 e 0 to 0 dl 1549444192 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 01:09:10 2019][1665849.489590] Lustre: 9006:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 49 previous similar messages [Wed Feb 6 01:09:10 2019][1665849.972350] Pid: 9008, comm: mdt01_026 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:09:10 2019][1665849.982181] Call Trace: [Wed Feb 6 01:09:10 2019][1665849.984827] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:09:10 2019][1665849.991936] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:09:10 2019][1665849.999304] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:09:10 2019][1665850.007519] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:09:10 2019][1665850.013914] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:09:10 2019][1665850.020138] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:09:10 2019][1665850.026901] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:09:10 2019][1665850.032767] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:09:10 2019][1665850.039881] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:09:10 2019][1665850.047771] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:09:10 2019][1665850.054268] [] kthread+0xd1/0xe0 [Wed Feb 6 01:09:10 2019][1665850.059355] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:09:10 2019][1665850.066004] [] 0xffffffffffffffff [Wed Feb 6 01:09:10 2019][1665850.071195] LustreError: dumping log to /tmp/lustre-log.1549444188.9008 [Wed Feb 6 01:09:10 2019][1665850.159690] LNet: Service thread pid 9030 was inactive for 1204.49s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 01:09:11 2019][1665850.172721] LNet: Skipped 38 previous similar messages [Wed Feb 6 01:09:16 2019][1665855.740574] Lustre: fir-MDT0000: Client db6e3be2-6dcf-bc59-61b2-a3b0152ca98d (at 10.10.112.17@tcp) reconnecting [Wed Feb 6 01:09:16 2019][1665855.750839] Lustre: Skipped 68 previous similar messages [Wed Feb 6 01:09:36 2019][1665875.646498] LNet: Service thread pid 15393 completed after 1581.27s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:09:57 2019][1665896.646653] LustreError: 8829:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9122d83a4800 ns: mdt-fir-MDT0002_UUID lock: ffff914238685e80/0x21ed935180c3f6b9 lrc: 1/0,0 mode: --/PW res: [0x2c0001689:0xb:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x54a01000000000 nid: 10.10.114.10@tcp remote: 0xd3460787bfacfdd9 expref: 14 pid: 8829 timeout: 0 lvb_type: 0 [Wed Feb 6 01:09:57 2019][1665896.646684] Lustre: 15178:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:345s); client may timeout. req@ffff913ba7130300 x1624702092174128/t0(0) o36->05683605-3c7a-66e7-65c2-872123284fcd@10.10.114.15@tcp:425/0 lens 520/424 e 0 to 0 dl 1549443890 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 01:10:24 2019][1665923.478201] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 01:11:38 2019][1665997.432047] LustreError: dumping log to /tmp/lustre-log.1549444336.15218 [Wed Feb 6 01:11:42 2019][1666001.528151] LustreError: dumping log to /tmp/lustre-log.1549444340.9171 [Wed Feb 6 01:11:46 2019][1666005.624256] LustreError: dumping log to /tmp/lustre-log.1549444344.17204 [Wed Feb 6 01:12:05 2019][1666024.752738] LustreError: 15186:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549444063, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff913f1ba87740/0x21ed935180bbda13 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 228 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 15186 timeout: 0 lvb_type: 0 [Wed Feb 6 01:12:05 2019][1666024.792207] LustreError: 15186:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 19 previous similar messages [Wed Feb 6 01:12:28 2019][1666047.650446] LustreError: 15409:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9152d5b30000 ns: mdt-fir-MDT0002_UUID lock: ffff912627b15a00/0x21ed93518104a292 lrc: 3/0,0 mode: PW/PW res: [0x2c0001689:0x6:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x50200000000000 nid: 10.10.114.10@tcp remote: 0xd3460787bfacfe3b expref: 6 pid: 15409 timeout: 0 lvb_type: 0 [Wed Feb 6 01:12:28 2019][1666047.650531] Lustre: 15261:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:496s); client may timeout. req@ffff9138e121bf00 x1624701947460720/t0(0) o36->7646fd6e-8364-5414-6410-5e73509a82fd@10.10.113.3@tcp:425/0 lens 520/424 e 0 to 0 dl 1549443890 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 01:12:28 2019][1666047.650534] Lustre: 15261:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1 previous similar message [Wed Feb 6 01:12:28 2019][1666047.650545] LNet: Service thread pid 15261 completed after 1250.99s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:12:28 2019][1666047.650546] LNet: Skipped 19 previous similar messages [Wed Feb 6 01:14:09 2019][1666148.987856] LNet: Service thread pid 8968 was inactive for 1201.28s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 01:14:09 2019][1666149.004967] LNet: Skipped 4 previous similar messages [Wed Feb 6 01:14:09 2019][1666149.010205] Pid: 8968, comm: mdt00_014 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:14:09 2019][1666149.020029] Call Trace: [Wed Feb 6 01:14:09 2019][1666149.022665] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:14:09 2019][1666149.029776] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:14:09 2019][1666149.037143] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:14:09 2019][1666149.045360] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:14:09 2019][1666149.051755] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:14:09 2019][1666149.057980] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:14:09 2019][1666149.064713] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:14:09 2019][1666149.070590] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:14:09 2019][1666149.077704] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:14:09 2019][1666149.085592] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:14:09 2019][1666149.092091] [] kthread+0xd1/0xe0 [Wed Feb 6 01:14:09 2019][1666149.097178] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:14:09 2019][1666149.103825] [] 0xffffffffffffffff [Wed Feb 6 01:14:09 2019][1666149.109020] LustreError: dumping log to /tmp/lustre-log.1549444487.8968 [Wed Feb 6 01:14:10 2019][1666149.256700] Pid: 7191, comm: mdt00_010 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:14:10 2019][1666149.266522] Call Trace: [Wed Feb 6 01:14:10 2019][1666149.269161] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.276281] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.283646] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:14:10 2019][1666149.291862] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:14:10 2019][1666149.298260] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:14:10 2019][1666149.304482] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:14:10 2019][1666149.311217] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:14:10 2019][1666149.317084] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.324199] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.332086] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.338585] [] kthread+0xd1/0xe0 [Wed Feb 6 01:14:10 2019][1666149.343674] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:14:10 2019][1666149.350322] [] 0xffffffffffffffff [Wed Feb 6 01:14:10 2019][1666149.355517] Pid: 48636, comm: mdt02_025 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:14:10 2019][1666149.365428] Call Trace: [Wed Feb 6 01:14:10 2019][1666149.368062] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.375161] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.382440] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.389278] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:14:10 2019][1666149.396012] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:14:10 2019][1666149.402409] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:14:10 2019][1666149.408709] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:14:10 2019][1666149.416148] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:14:10 2019][1666149.423062] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:14:10 2019][1666149.431279] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:14:10 2019][1666149.437674] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:14:10 2019][1666149.443897] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:14:10 2019][1666149.450642] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:14:10 2019][1666149.456509] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.463615] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.471501] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.478001] [] kthread+0xd1/0xe0 [Wed Feb 6 01:14:10 2019][1666149.483080] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:14:10 2019][1666149.489728] [] 0xffffffffffffffff [Wed Feb 6 01:14:10 2019][1666149.494906] Pid: 8573, comm: mdt03_011 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:14:10 2019][1666149.504732] Call Trace: [Wed Feb 6 01:14:10 2019][1666149.507364] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.514466] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.521830] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:14:10 2019][1666149.530037] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:14:10 2019][1666149.536435] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:14:10 2019][1666149.542656] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:14:10 2019][1666149.549391] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:14:10 2019][1666149.555258] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.562374] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.570262] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.576761] [] kthread+0xd1/0xe0 [Wed Feb 6 01:14:10 2019][1666149.581841] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:14:10 2019][1666149.588478] [] 0xffffffffffffffff [Wed Feb 6 01:14:10 2019][1666149.593646] Pid: 7029, comm: mdt02_010 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:14:10 2019][1666149.603466] Call Trace: [Wed Feb 6 01:14:10 2019][1666149.606095] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.613197] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.620555] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:14:10 2019][1666149.628763] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:14:10 2019][1666149.635158] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:14:10 2019][1666149.641374] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:14:10 2019][1666149.648116] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:14:10 2019][1666149.653982] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.661091] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.668976] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:14:10 2019][1666149.675478] [] kthread+0xd1/0xe0 [Wed Feb 6 01:14:10 2019][1666149.680556] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:14:10 2019][1666149.687194] [] 0xffffffffffffffff [Wed Feb 6 01:14:13 2019][1666153.083951] LustreError: dumping log to /tmp/lustre-log.1549444491.15224 [Wed Feb 6 01:14:18 2019][1666157.180061] LustreError: dumping log to /tmp/lustre-log.1549444495.8950 [Wed Feb 6 01:14:36 2019][1666175.664561] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 01:14:36 2019][1666175.675602] Lustre: Skipped 4 previous similar messages [Wed Feb 6 01:14:54 2019][1666193.668976] LustreError: 9174:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549444232, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff91284d0e8240/0x21ed935181034d67 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 46 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935181034d6e expref: -99 pid: 9174 timeout: 0 lvb_type: 0 [Wed Feb 6 01:14:54 2019][1666193.709054] LustreError: 9174:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 19 previous similar messages [Wed Feb 6 01:14:59 2019][1666198.654098] LustreError: 15218:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.114.15@tcp) failed to reply to blocking AST (req@ffff91241cdd7b00 x1623682051842416 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff9150b93c2ac0/0x21ed93518068d476 lrc: 4/0,0 mode: PR/PR res: [0x2c0001689:0x4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.114.15@tcp remote: 0x5a89b021a3479fdd expref: 9 pid: 6804 timeout: 1666305 lvb_type: 0 [Wed Feb 6 01:14:59 2019][1666198.696608] LustreError: 15218:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 9 previous similar messages [Wed Feb 6 01:14:59 2019][1666198.706871] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.114.15@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 01:14:59 2019][1666198.719573] LustreError: Skipped 9 previous similar messages [Wed Feb 6 01:14:59 2019][1666198.725432] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.114.15@tcp ns: mdt-fir-MDT0002_UUID lock: ffff9150b93c2ac0/0x21ed93518068d476 lrc: 3/0,0 mode: PR/PR res: [0x2c0001689:0x4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.114.15@tcp remote: 0x5a89b021a3479fdd expref: 10 pid: 6804 timeout: 0 lvb_type: 0 [Wed Feb 6 01:14:59 2019][1666198.762519] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 9 previous similar messages [Wed Feb 6 01:14:59 2019][1666198.772484] Lustre: 15218:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:647s); client may timeout. req@ffff912c09beda00 x1624702076472672/t0(0) o36->0161d142-b9e6-5768-fef0-f19a8b36b0dc@10.10.114.10@tcp:425/0 lens 520/424 e 0 to 0 dl 1549443890 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 01:15:19 2019][1666218.621608] LustreError: dumping log to /tmp/lustre-log.1549444557.15221 [Wed Feb 6 01:15:41 2019][1666240.613150] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 01:16:12 2019][1666271.870939] LustreError: dumping log to /tmp/lustre-log.1549444610.6615 [Wed Feb 6 01:16:41 2019][1666300.543649] LustreError: dumping log to /tmp/lustre-log.1549444639.6174 [Wed Feb 6 01:16:53 2019][1666312.831998] LustreError: dumping log to /tmp/lustre-log.1549444651.15281 [Wed Feb 6 01:17:10 2019][1666329.748076] Lustre: fir-MDT0002: Connection restored to 9ac98536-561e-ef99-7124-2ad408fdb185 (at 10.9.101.7@o2ib4) [Wed Feb 6 01:17:10 2019][1666329.758595] Lustre: Skipped 54 previous similar messages [Wed Feb 6 01:17:26 2019][1666345.600841] LustreError: dumping log to /tmp/lustre-log.1549444684.53421 [Wed Feb 6 01:17:30 2019][1666349.658119] LustreError: 9171:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9122f64b6000 ns: mdt-fir-MDT0002_UUID lock: ffff911b18dd7980/0x21ed9351812c10a7 lrc: 1/0,0 mode: --/PW res: [0x2c0001689:0x6:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x54a01000000000 nid: 10.10.114.10@tcp remote: 0xd3460787bfacff68 expref: 4 pid: 9171 timeout: 0 lvb_type: 0 [Wed Feb 6 01:17:30 2019][1666349.658231] LNet: Service thread pid 8573 completed after 1401.98s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 01:17:30 2019][1666349.658233] LNet: Skipped 9 previous similar messages [Wed Feb 6 01:17:30 2019][1666349.715921] Lustre: 7029:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:647s); client may timeout. req@ffff9141b42d3c00 x1624705157109024/t0(0) o36->492310fb-8028-83a2-d5b0-7c5235cdad98@10.10.29.8@tcp:576/0 lens 520/424 e 0 to 0 dl 1549444041 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 01:18:13 2019][1666392.818961] LNetError: 15261:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.29.8@tcp from [Wed Feb 6 01:18:13 2019][1666392.829570] LNetError: 15261:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 118305650 previous similar messages [Wed Feb 6 01:18:19 2019][1666398.850119] LustreError: dumping log to /tmp/lustre-log.1549444737.15240 [Wed Feb 6 01:18:21 2019][1666400.841159] Lustre: 15261:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549444739/real 1549444739] req@ffff91209a590900 x1623682051954672/t0(0) o104->fir-MDT0000@10.10.29.8@tcp:15/16 lens 296/224 e 0 to 1 dl 1549444750 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 01:18:21 2019][1666400.868753] Lustre: 15261:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 117896710 previous similar messages [Wed Feb 6 01:18:40 2019][1666419.330720] LustreError: dumping log to /tmp/lustre-log.1549444758.17206 [Wed Feb 6 01:19:17 2019][1666457.091595] Lustre: 6616:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 01:19:17 2019][1666457.091595] req@ffff911d8291d700 x1624673932600720/t0(0) o36->5f9695c8-8953-f70d-cc1c-5dc8656027f2@10.9.101.20@o2ib4:580/0 lens 528/2888 e 0 to 0 dl 1549444800 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 01:19:17 2019][1666457.120926] Lustre: 6616:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 37 previous similar messages [Wed Feb 6 01:19:23 2019][1666462.730732] Lustre: fir-MDT0002: Client 5f9695c8-8953-f70d-cc1c-5dc8656027f2 (at 10.9.101.20@o2ib4) reconnecting [Wed Feb 6 01:19:23 2019][1666462.741082] Lustre: Skipped 35 previous similar messages [Wed Feb 6 01:19:37 2019][1666476.676071] LNet: Service thread pid 8973 was inactive for 1202.40s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 01:19:37 2019][1666476.693177] LNet: Skipped 4 previous similar messages [Wed Feb 6 01:19:37 2019][1666476.698405] Pid: 8973, comm: mdt01_020 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:19:37 2019][1666476.708231] Call Trace: [Wed Feb 6 01:19:37 2019][1666476.710868] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:19:37 2019][1666476.717980] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:19:37 2019][1666476.725258] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:19:37 2019][1666476.732105] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:19:37 2019][1666476.738849] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:19:37 2019][1666476.745244] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:19:37 2019][1666476.751555] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:19:37 2019][1666476.758990] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:19:37 2019][1666476.765899] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:19:37 2019][1666476.774117] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:19:37 2019][1666476.780511] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:19:37 2019][1666476.786732] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:19:37 2019][1666476.793466] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:19:37 2019][1666476.799343] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:19:37 2019][1666476.806459] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:19:37 2019][1666476.814356] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:19:37 2019][1666476.820855] [] kthread+0xd1/0xe0 [Wed Feb 6 01:19:37 2019][1666476.825943] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:19:37 2019][1666476.832589] [] 0xffffffffffffffff [Wed Feb 6 01:19:37 2019][1666476.837783] LustreError: dumping log to /tmp/lustre-log.1549444815.8973 [Wed Feb 6 01:19:49 2019][1666488.964378] Pid: 8998, comm: mdt01_023 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:19:49 2019][1666488.974209] Call Trace: [Wed Feb 6 01:19:49 2019][1666488.976855] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:19:49 2019][1666488.983973] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 01:19:49 2019][1666488.991274] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 01:19:49 2019][1666488.998111] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 01:19:49 2019][1666489.004859] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 01:19:49 2019][1666489.011259] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 01:19:49 2019][1666489.017581] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 01:19:49 2019][1666489.025020] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 01:19:49 2019][1666489.031942] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 01:19:49 2019][1666489.040162] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:19:49 2019][1666489.046572] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:19:49 2019][1666489.052796] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:19:49 2019][1666489.059554] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:19:49 2019][1666489.065426] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:19:49 2019][1666489.072553] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:19:49 2019][1666489.080444] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:19:49 2019][1666489.086960] [] kthread+0xd1/0xe0 [Wed Feb 6 01:19:49 2019][1666489.092053] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:19:49 2019][1666489.098720] [] 0xffffffffffffffff [Wed Feb 6 01:19:49 2019][1666489.103916] LustreError: dumping log to /tmp/lustre-log.1549444827.8998 [Wed Feb 6 01:20:18 2019][1666517.637139] Pid: 8982, comm: mdt02_014 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:20:18 2019][1666517.646960] Call Trace: [Wed Feb 6 01:20:18 2019][1666517.649595] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:20:18 2019][1666517.656709] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:20:18 2019][1666517.664067] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:20:18 2019][1666517.672284] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:20:18 2019][1666517.678671] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:20:18 2019][1666517.684885] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:20:18 2019][1666517.691629] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:20:18 2019][1666517.697494] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:20:18 2019][1666517.704618] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:20:18 2019][1666517.712506] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:20:18 2019][1666517.719006] [] kthread+0xd1/0xe0 [Wed Feb 6 01:20:18 2019][1666517.724093] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:20:18 2019][1666517.730811] [] 0xffffffffffffffff [Wed Feb 6 01:20:18 2019][1666517.736030] LustreError: dumping log to /tmp/lustre-log.1549444856.8982 [Wed Feb 6 01:20:47 2019][1666546.309823] Pid: 6760, comm: mdt01_006 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:20:47 2019][1666546.319646] Call Trace: [Wed Feb 6 01:20:47 2019][1666546.322293] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:20:47 2019][1666546.329411] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:20:47 2019][1666546.336779] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:20:47 2019][1666546.345002] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:20:47 2019][1666546.351400] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:20:47 2019][1666546.357621] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:20:47 2019][1666546.364364] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:20:47 2019][1666546.370240] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:20:47 2019][1666546.377356] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:20:47 2019][1666546.385245] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:20:47 2019][1666546.391743] [] kthread+0xd1/0xe0 [Wed Feb 6 01:20:47 2019][1666546.396831] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:20:47 2019][1666546.403477] [] 0xffffffffffffffff [Wed Feb 6 01:20:47 2019][1666546.408672] LustreError: dumping log to /tmp/lustre-log.1549444885.6760 [Wed Feb 6 01:20:55 2019][1666554.502024] Pid: 15406, comm: mdt00_047 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:20:55 2019][1666554.511941] Call Trace: [Wed Feb 6 01:20:55 2019][1666554.514602] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:20:55 2019][1666554.521723] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:20:55 2019][1666554.529088] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:20:55 2019][1666554.537321] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:20:55 2019][1666554.543711] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:20:55 2019][1666554.549923] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:20:55 2019][1666554.556659] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:20:55 2019][1666554.562526] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:20:55 2019][1666554.569650] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:20:55 2019][1666554.577536] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:20:55 2019][1666554.584037] [] kthread+0xd1/0xe0 [Wed Feb 6 01:20:55 2019][1666554.589124] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:20:55 2019][1666554.595772] [] 0xffffffffffffffff [Wed Feb 6 01:20:55 2019][1666554.600981] LustreError: dumping log to /tmp/lustre-log.1549444893.15406 [Wed Feb 6 01:21:30 2019][1666589.861913] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 01:21:40 2019][1666599.559418] LNet: Service thread pid 9009 was inactive for 1201.26s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 01:21:40 2019][1666599.572458] LNet: Skipped 34 previous similar messages [Wed Feb 6 01:21:40 2019][1666599.577781] LustreError: dumping log to /tmp/lustre-log.1549444938.9009 [Wed Feb 6 01:22:07 2019][1666626.510870] LustreError: 9005:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549444665, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9123df48de80/0x21ed9351812c287f lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 227 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 9005 timeout: 0 lvb_type: 0 [Wed Feb 6 01:22:07 2019][1666626.550166] LustreError: 9005:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 11 previous similar messages [Wed Feb 6 01:22:17 2019][1666636.424076] LustreError: dumping log to /tmp/lustre-log.1549444975.8947 [Wed Feb 6 01:22:32 2019][1666651.665619] Lustre: 8998:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:611s); client may timeout. req@ffff91370d889500 x1624701976803536/t0(0) o36->5208f31c-d200-e396-4871-5bc220652223@10.10.115.6@tcp:159/0 lens 520/424 e 0 to 0 dl 1549444379 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 01:22:32 2019][1666651.694282] Lustre: 8998:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2 previous similar messages [Wed Feb 6 01:24:36 2019][1666775.691574] LustreError: dumping log to /tmp/lustre-log.1549445114.38457 [Wed Feb 6 01:24:38 2019][1666777.317702] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 01:24:38 2019][1666777.328751] Lustre: Skipped 4 previous similar messages [Wed Feb 6 01:25:01 2019][1666800.268199] Pid: 15389, comm: mdt01_073 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 01:25:01 2019][1666800.278107] Call Trace: [Wed Feb 6 01:25:01 2019][1666800.280745] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 01:25:01 2019][1666800.287856] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 01:25:01 2019][1666800.295224] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 01:25:01 2019][1666800.303441] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 01:25:01 2019][1666800.309836] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 01:25:01 2019][1666800.316060] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 01:25:01 2019][1666800.322801] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 01:25:01 2019][1666800.328683] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 01:25:01 2019][1666800.335804] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 01:25:01 2019][1666800.343688] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 01:25:01 2019][1666800.350193] [] kthread+0xd1/0xe0 [Wed Feb 6 01:25:01 2019][1666800.355278] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 01:25:01 2019][1666800.361926] [] 0xffffffffffffffff [Wed Feb 6 01:25:01 2019][1666800.367118] LustreError: dumping log to /tmp/lustre-log.1549445139.15389 [Wed Feb 6 01:25:03 2019][1666802.669257] LustreError: 6613:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff91219aa09e00 x1623682052051408 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff913cc7f2cc80/0x21ed9351812bfff2 lrc: 4/0,0 mode: PR/PR res: [0x2c0001679:0x3:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e141b0 expref: 13 pid: 15265 timeout: 1666909 lvb_type: 0 [Wed Feb 6 01:25:03 2019][1666802.711847] LustreError: 6613:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 6 previous similar messages [Wed Feb 6 01:25:03 2019][1666802.722026] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.15@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 01:25:03 2019][1666802.734738] LustreError: Skipped 6 previous similar messages [Wed Feb 6 01:25:03 2019][1666802.740596] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.15@tcp ns: mdt-fir-MDT0002_UUID lock: ffff913cc7f2cc80/0x21ed9351812bfff2 lrc: 3/0,0 mode: PR/PR res: [0x2c0001679:0x3:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e141b0 expref: 14 pid: 15265 timeout: 0 lvb_type: 0 [Wed Feb 6 01:25:03 2019][1666802.777745] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 6 previous similar messages [Wed Feb 6 01:25:12 2019][1666811.953489] LustreError: 15281:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549444850, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff911b83bca640/0x21ed9351812c7f81 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 30 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351812c7f88 expref: -99 pid: 15281 timeout: 0 lvb_type: 0 [Wed Feb 6 01:25:12 2019][1666811.993742] LustreError: 15281:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 17 previous similar messages [Wed Feb 6 01:27:32 2019][1666951.689224] Lustre: fir-MDT0000: Connection restored to 10.0.10.51@o2ib7 (at 0@lo) [Wed Feb 6 01:27:32 2019][1666951.696970] Lustre: Skipped 57 previous similar messages [Wed Feb 6 01:28:13 2019][1666992.834017] LNetError: 17223:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.112.17@tcp from [Wed Feb 6 01:28:13 2019][1666992.834019] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.112.12@tcp from [Wed Feb 6 01:28:13 2019][1666992.834023] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 113241531 previous similar messages [Wed Feb 6 01:28:13 2019][1666992.866869] LNetError: 17223:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 4429 previous similar messages [Wed Feb 6 01:28:21 2019][1667000.856218] Lustre: 17223:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549445339/real 1549445339] req@ffff915999fa4500 x1623682052165072/t0(0) o104->fir-MDT0000@10.10.112.17@tcp:15/16 lens 296/224 e 0 to 1 dl 1549445350 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 01:28:21 2019][1667000.883987] Lustre: 17223:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 113112344 previous similar messages [Wed Feb 6 01:29:37 2019][1667076.531133] Lustre: 7415:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 01:29:37 2019][1667076.531133] req@ffff913526b5dd00 x1624672672958912/t0(0) o36->a437cdb1-0624-3608-42e3-2ea1301fd34c@10.9.102.3@o2ib4:445/0 lens 536/2888 e 0 to 0 dl 1549445420 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 01:29:37 2019][1667076.560376] Lustre: 7415:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 31 previous similar messages [Wed Feb 6 01:29:43 2019][1667082.571791] Lustre: fir-MDT0000: Client a437cdb1-0624-3608-42e3-2ea1301fd34c (at 10.9.102.3@o2ib4) reconnecting [Wed Feb 6 01:29:43 2019][1667082.582049] Lustre: Skipped 26 previous similar messages [Wed Feb 6 01:32:34 2019][1667253.689577] LustreError: 8999:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549445292, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff914f00fbcc80/0x21ed9351812d5e70 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 223 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 8999 timeout: 0 lvb_type: 0 [Wed Feb 6 01:32:34 2019][1667253.713570] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 01:32:34 2019][1667253.713572] Lustre: Skipped 1 previous similar message [Wed Feb 6 01:32:34 2019][1667253.749289] LustreError: 8999:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 36 previous similar messages [Wed Feb 6 01:34:43 2019][1667382.638954] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 01:34:43 2019][1667382.649998] Lustre: Skipped 5 previous similar messages [Wed Feb 6 01:35:07 2019][1667406.684413] LustreError: 6175:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.112.17@tcp) failed to reply to blocking AST (req@ffff9142bda4bf00 x1623682052320704 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff913f2efe7980/0x21ed9351813f7aca lrc: 4/0,0 mode: PR/PR res: [0x200001787:0x3:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.112.17@tcp remote: 0xcec6ee3d595b3ae7 expref: 10 pid: 6175 timeout: 1667513 lvb_type: 0 [Wed Feb 6 01:35:07 2019][1667406.726919] LustreError: 6175:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 5 previous similar messages [Wed Feb 6 01:35:07 2019][1667406.737098] LustreError: 138-a: fir-MDT0000: A client on nid 10.10.112.17@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 01:35:07 2019][1667406.749790] LustreError: Skipped 5 previous similar messages [Wed Feb 6 01:35:07 2019][1667406.755655] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.17@tcp ns: mdt-fir-MDT0000_UUID lock: ffff913f2efe7980/0x21ed9351813f7aca lrc: 3/0,0 mode: PR/PR res: [0x200001787:0x3:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.112.17@tcp remote: 0xcec6ee3d595b3ae7 expref: 11 pid: 6175 timeout: 0 lvb_type: 0 [Wed Feb 6 01:35:07 2019][1667406.792717] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages [Wed Feb 6 01:35:22 2019][1667421.676792] LustreError: 15202:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549445460, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff91205ec2b840/0x21ed9351813ef672 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 38 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351813ef679 expref: -99 pid: 15202 timeout: 0 lvb_type: 0 [Wed Feb 6 01:35:22 2019][1667421.717042] LustreError: 15202:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 27 previous similar messages [Wed Feb 6 01:37:35 2019][1667554.508605] Lustre: MGS: Connection restored to 89ea7458-6a00-b656-cb60-c83a923152d6 (at 10.8.7.23@o2ib6) [Wed Feb 6 01:37:35 2019][1667554.518349] Lustre: Skipped 117 previous similar messages [Wed Feb 6 01:38:13 2019][1667592.849075] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.112.12@tcp from [Wed Feb 6 01:38:13 2019][1667592.859857] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 99727930 previous similar messages [Wed Feb 6 01:38:21 2019][1667600.871279] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549445939/real 1549445939] req@ffff9143d4758600 x1623682052444336/t0(0) o104->fir-MDT0002@10.10.112.12@tcp:15/16 lens 296/224 e 0 to 1 dl 1549445950 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 01:38:21 2019][1667600.899049] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 99580062 previous similar messages [Wed Feb 6 01:40:44 2019][1667743.373673] Lustre: fir-MDT0000: haven't heard from client 7f648de4-ccea-3fb5-cc96-0c4479d5ecdb (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff913423a01c00, cur 1549446082 expire 1549445932 last 1549445855 [Wed Feb 6 01:40:44 2019][1667743.395546] Lustre: Skipped 2 previous similar messages [Wed Feb 6 01:42:38 2019][1667857.701734] LustreError: 15230:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549445896, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff912be2915c40/0x21ed935181ea0ddb lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 195 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 15230 timeout: 0 lvb_type: 0 [Wed Feb 6 01:42:38 2019][1667857.741209] LustreError: 15230:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 29 previous similar messages [Wed Feb 6 01:42:38 2019][1667857.749729] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 01:42:38 2019][1667857.749731] Lustre: Skipped 1 previous similar message [Wed Feb 6 01:42:40 2019][1667859.695869] LustreError: 15370:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9152d4e6c800 ns: mdt-fir-MDT0002_UUID lock: ffff913ea2c7ba80/0x21ed93518203dead lrc: 3/0,0 mode: PW/PW res: [0x2c00016e5:0x4:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT flags: 0x50200000000000 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7d324 expref: 42 pid: 15370 timeout: 0 lvb_type: 0 [Wed Feb 6 01:42:40 2019][1667859.730310] LustreError: 15370:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 2 previous similar messages [Wed Feb 6 01:45:09 2019][1668008.718551] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 01:45:09 2019][1668008.729591] Lustre: Skipped 4 previous similar messages [Wed Feb 6 01:45:11 2019][1668010.699570] LustreError: 15169:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.112.13@tcp) failed to reply to blocking AST (req@ffff911ed88c7800 x1623682052559968 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff913a6a353180/0x21ed9351820ae296 lrc: 4/0,0 mode: PR/PR res: [0x2000017d4:0x2:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.112.13@tcp remote: 0x64798b822b61f2e1 expref: 9 pid: 15185 timeout: 1668117 lvb_type: 0 [Wed Feb 6 01:45:11 2019][1668010.699575] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 01:45:11 2019][1668010.699577] LustreError: Skipped 3 previous similar messages [Wed Feb 6 01:45:11 2019][1668010.699599] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.6@tcp ns: mdt-fir-MDT0002_UUID lock: ffff91265ab2dc40/0x21ed9351820af757 lrc: 3/0,0 mode: PR/PR res: [0x2c00016e5:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7d466 expref: 28 pid: 8999 timeout: 0 lvb_type: 0 [Wed Feb 6 01:45:11 2019][1668010.699602] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages [Wed Feb 6 01:45:11 2019][1668010.699742] LustreError: 15185:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff914f1bf07000 ns: mdt-fir-MDT0002_UUID lock: ffff913cfc736c00/0x21ed9351820af773 lrc: 1/0,0 mode: --/PW res: [0x2c00016e5:0x4:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x54a01000000000 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7d474 expref: 7 pid: 15185 timeout: 0 lvb_type: 0 [Wed Feb 6 01:45:11 2019][1668010.841681] LustreError: 15169:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 4 previous similar messages [Wed Feb 6 01:45:23 2019][1668022.584877] LustreError: 15330:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549446061, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff913ce58be540/0x21ed93518204c789 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 42 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518204c790 expref: -99 pid: 15330 timeout: 0 lvb_type: 0 [Wed Feb 6 01:45:23 2019][1668022.625139] LustreError: 15330:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 54 previous similar messages [Wed Feb 6 01:47:40 2019][1668159.794396] Lustre: fir-MDT0000: Connection restored to 10.0.10.51@o2ib7 (at 0@lo) [Wed Feb 6 01:47:40 2019][1668159.802145] Lustre: Skipped 373 previous similar messages [Wed Feb 6 01:47:42 2019][1668161.703435] LustreError: 8829:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff912bc0223000 ns: mdt-fir-MDT0002_UUID lock: ffff913ea2c7f2c0/0x21ed935182116f24 lrc: 3/0,0 mode: PW/PW res: [0x2c00016e5:0x4:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT flags: 0x50200000000000 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7d507 expref: 14 pid: 8829 timeout: 0 lvb_type: 0 [Wed Feb 6 01:48:13 2019][1668192.864131] LNetError: 6169:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.15@tcp from [Wed Feb 6 01:48:13 2019][1668192.874820] LNetError: 6169:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 117640485 previous similar messages [Wed Feb 6 01:48:21 2019][1668200.886335] Lustre: 15370:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549446539/real 1549446539] req@ffff9143d475a700 x1623682052667072/t0(0) o104->fir-MDT0002@10.10.113.15@tcp:15/16 lens 296/224 e 0 to 1 dl 1549446550 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 01:48:21 2019][1668200.914106] Lustre: 15370:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 118314086 previous similar messages [Wed Feb 6 01:52:16 2019][1668435.918243] Lustre: 8999:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 01:52:16 2019][1668435.918243] req@ffff91248f0a7200 x1624697925552720/t0(0) o36->cf0229df-1948-b94a-5e59-bdab7e865d86@10.9.103.11@o2ib4:294/0 lens 528/2888 e 0 to 0 dl 1549446779 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 01:52:16 2019][1668435.947570] Lustre: 8999:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 16 previous similar messages [Wed Feb 6 01:52:22 2019][1668441.417963] Lustre: fir-MDT0002: Client cf0229df-1948-b94a-5e59-bdab7e865d86 (at 10.9.103.11@o2ib4) reconnecting [Wed Feb 6 01:52:22 2019][1668441.428312] Lustre: Skipped 15 previous similar messages [Wed Feb 6 01:52:42 2019][1668461.768887] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 01:52:42 2019][1668461.784002] Lustre: Skipped 1 previous similar message [Wed Feb 6 01:52:44 2019][1668463.711098] LustreError: 17222:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9138494e8c00 ns: mdt-fir-MDT0002_UUID lock: ffff91539da68900/0x21ed93518220aa32 lrc: 3/0,0 mode: PW/PW res: [0x2c00016e5:0x4:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x50200000000000 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7d54d expref: 3 pid: 17222 timeout: 0 lvb_type: 0 [Wed Feb 6 01:52:44 2019][1668463.745478] LustreError: 17222:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message [Wed Feb 6 01:53:18 2019][1668497.728795] LustreError: 15169:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549446536, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9126a5de6540/0x21ed9351821ac4bd lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 204 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 15169 timeout: 0 lvb_type: 0 [Wed Feb 6 01:53:18 2019][1668497.768269] LustreError: 15169:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 31 previous similar messages [Wed Feb 6 01:53:39 2019][1668518.327322] Lustre: 6845:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 01:53:39 2019][1668518.327322] req@ffff91566f2bc200 x1624699630303488/t0(0) o36->ab01728a-7f0a-5d0b-1dba-f4bfe5ee2a67@10.9.105.67@o2ib4:376/0 lens 544/2888 e 0 to 0 dl 1549446861 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 01:53:39 2019][1668518.356653] Lustre: 6845:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 44 previous similar messages [Wed Feb 6 01:53:44 2019][1668523.816891] Lustre: fir-MDT0002: Client ab01728a-7f0a-5d0b-1dba-f4bfe5ee2a67 (at 10.9.105.67@o2ib4) reconnecting [Wed Feb 6 01:53:44 2019][1668523.827244] Lustre: Skipped 24 previous similar messages [Wed Feb 6 01:55:15 2019][1668614.714734] LustreError: 6682:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff9157f8f11800 x1623682052765392 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff9144f1bfbf00/0x21ed93518220cef2 lrc: 4/0,0 mode: PR/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 9 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e149dd expref: 27 pid: 48676 timeout: 1668721 lvb_type: 0 [Wed Feb 6 01:55:15 2019][1668614.714744] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.112.12@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 01:55:15 2019][1668614.714746] LustreError: Skipped 8 previous similar messages [Wed Feb 6 01:55:15 2019][1668614.714770] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.15@tcp ns: mdt-fir-MDT0002_UUID lock: ffff913aec6169c0/0x21ed9351822274d5 lrc: 3/0,0 mode: PR/PR res: [0x2c00016df:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e149eb expref: 28 pid: 15372 timeout: 0 lvb_type: 0 [Wed Feb 6 01:55:15 2019][1668614.714773] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 6 previous similar messages [Wed Feb 6 01:55:15 2019][1668614.823036] LustreError: 6682:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 9 previous similar messages [Wed Feb 6 01:55:19 2019][1668618.421861] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 01:55:19 2019][1668618.432906] Lustre: Skipped 3 previous similar messages [Wed Feb 6 01:55:49 2019][1668648.395445] Lustre: MGS: haven't heard from client 5bf8aaa4-7c14-2e07-6f31-830b68974f4c (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff911f4dc79400, cur 1549446987 expire 1549446837 last 1549446760 [Wed Feb 6 01:55:49 2019][1668648.416537] Lustre: Skipped 2 previous similar messages [Wed Feb 6 01:55:52 2019][1668651.373717] Lustre: fir-MDT0000: haven't heard from client b1ae38e6-4e9c-51ec-0d30-af325c63f3ef (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff912b00bd3c00, cur 1549446990 expire 1549446840 last 1549446763 [Wed Feb 6 01:57:42 2019][1668761.277471] Lustre: 53422:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 01:57:42 2019][1668761.277471] req@ffff91402f51ec00 x1624705304995584/t0(0) o36->cd2fc9d9-2801-0b42-234d-619ca30bf3c7@10.8.9.8@o2ib6:619/0 lens 520/2888 e 0 to 0 dl 1549447104 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 01:57:42 2019][1668761.306631] Lustre: 53422:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 7 previous similar messages [Wed Feb 6 01:57:44 2019][1668763.768473] LustreError: 17223:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549446802, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff915654edec00/0x21ed935182227d25 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 36 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935182227d33 expref: -99 pid: 17223 timeout: 0 lvb_type: 0 [Wed Feb 6 01:57:44 2019][1668763.781545] Lustre: fir-MDT0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) [Wed Feb 6 01:57:44 2019][1668763.781547] Lustre: Skipped 188 previous similar messages [Wed Feb 6 01:57:44 2019][1668763.823074] LustreError: 17223:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 24 previous similar messages [Wed Feb 6 01:58:13 2019][1668792.879183] LNetError: 8968:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.14.7@tcp from [Wed Feb 6 01:58:13 2019][1668792.889704] LNetError: 8968:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 145221694 previous similar messages [Wed Feb 6 01:58:21 2019][1668800.901386] Lustre: 15185:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549447139/real 1549447139] req@ffff913bc8e16300 x1623682052906160/t0(0) o104->fir-MDT0002@10.10.113.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549447150 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 01:58:21 2019][1668800.929085] Lustre: 15185:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 147615298 previous similar messages [Wed Feb 6 01:58:24 2019][1668803.900716] Lustre: fir-MDT0000: Client 3f8f2dad-883d-00e9-c850-0ee136ec2184 (at 10.9.103.13@o2ib4) reconnecting [Wed Feb 6 02:00:17 2019][1668916.722502] Lustre: 6795:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:120s); client may timeout. req@ffff91258e858300 x1624701847883552/t0(0) o36->480f6595-8ac1-2fa1-010d-61ed12a06db6@10.10.112.13@tcp:650/0 lens 528/424 e 0 to 0 dl 1549447135 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 02:00:17 2019][1668916.751238] Lustre: 6795:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [Wed Feb 6 02:02:46 2019][1669065.734042] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 02:02:46 2019][1669065.749155] Lustre: Skipped 1 previous similar message [Wed Feb 6 02:02:48 2019][1669067.726205] LustreError: 15222:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff913555871c00 ns: mdt-fir-MDT0002_UUID lock: ffff912f4b6657c0/0x21ed935182363893 lrc: 3/0,0 mode: --/PW res: [0x2c00016df:0x5:0x0].0x0 bits 0x40/0x0 rrc: 3 type: IBT flags: 0x54a01000000000 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e14b1f expref: 4 pid: 15222 timeout: 0 lvb_type: 0 [Wed Feb 6 02:02:48 2019][1669067.760654] LustreError: 15222:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 11 previous similar messages [Wed Feb 6 02:05:19 2019][1669218.729889] LustreError: 15202:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff91243cfe1500 x1623682053037520 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff913722ce1440/0x21ed93518239d438 lrc: 4/0,0 mode: PR/PR res: [0x2c00016df:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e14b57 expref: 13 pid: 15184 timeout: 1669325 lvb_type: 0 [Wed Feb 6 02:05:19 2019][1669218.729894] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 02:05:19 2019][1669218.729896] LustreError: Skipped 22 previous similar messages [Wed Feb 6 02:05:19 2019][1669218.729920] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.29.8@tcp ns: mdt-fir-MDT0000_UUID lock: ffff9135f16e2640/0x21ed935182361a32 lrc: 3/0,0 mode: PR/PR res: [0x200001760:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d639b expref: 18 pid: 15164 timeout: 0 lvb_type: 0 [Wed Feb 6 02:05:19 2019][1669218.729923] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 17 previous similar messages [Wed Feb 6 02:05:19 2019][1669218.837788] LustreError: 15202:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 25 previous similar messages [Wed Feb 6 02:07:50 2019][1669369.756449] Lustre: fir-MDT0002: Connection restored to c21fe3e0-48ac-9c3c-090a-df5411207a5b (at 10.10.113.6@tcp) [Wed Feb 6 02:07:50 2019][1669369.766889] Lustre: Skipped 41 previous similar messages [Wed Feb 6 02:08:13 2019][1669392.894242] LNetError: 17221:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.15@tcp from [Wed Feb 6 02:08:13 2019][1669392.905025] LNetError: 17221:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 206871490 previous similar messages [Wed Feb 6 02:08:21 2019][1669400.916444] Lustre: 17206:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549447739/real 1549447739] req@ffff9153dfe2f200 x1623682053182416/t0(0) o104->fir-MDT0002@10.10.113.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549447750 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 02:08:21 2019][1669400.944125] Lustre: 17206:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 207890382 previous similar messages [Wed Feb 6 02:12:50 2019][1669669.947205] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 02:12:50 2019][1669669.948172] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 02:12:50 2019][1669669.948175] Lustre: Skipped 4 previous similar messages [Wed Feb 6 02:12:50 2019][1669669.978761] LustreError: 15278:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549447708, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff911ddbd37500/0x21ed93518255bbcd lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 46 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518255bbd4 expref: -99 pid: 15278 timeout: 0 lvb_type: 0 [Wed Feb 6 02:12:50 2019][1669670.015208] LustreError: 17218:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549447708, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9154a4641f80/0x21ed93518255cfe6 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 186 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 17218 timeout: 0 lvb_type: 0 [Wed Feb 6 02:12:50 2019][1669670.015211] LustreError: 17218:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 16 previous similar messages [Wed Feb 6 02:12:50 2019][1669670.069595] LustreError: 15278:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 22 previous similar messages [Wed Feb 6 02:12:52 2019][1669671.741355] LustreError: 38457:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff912710f99c00 ns: mdt-fir-MDT0002_UUID lock: ffff915567e406c0/0x21ed9351825ba276 lrc: 1/0,0 mode: --/PW res: [0x2c00016df:0x5:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x54a01000000000 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e14d9c expref: 5 pid: 38457 timeout: 0 lvb_type: 0 [Wed Feb 6 02:12:52 2019][1669671.775825] LustreError: 38457:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 13 previous similar messages [Wed Feb 6 02:15:23 2019][1669822.745044] LustreError: 17218:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.6@tcp) failed to reply to blocking AST (req@ffff9158a9f8a700 x1623682053319312 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff913d0437aac0/0x21ed935182672cca lrc: 4/0,0 mode: PR/PR res: [0x2c00016e5:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7d8d4 expref: 13 pid: 15331 timeout: 1669929 lvb_type: 0 [Wed Feb 6 02:15:23 2019][1669822.745046] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 02:15:23 2019][1669822.745050] LustreError: Skipped 23 previous similar messages [Wed Feb 6 02:15:23 2019][1669822.745077] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.17@tcp ns: mdt-fir-MDT0002_UUID lock: ffff913d9f35d580/0x21ed9351826786dc lrc: 3/0,0 mode: PR/PR res: [0x2c0001700:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.112.17@tcp remote: 0xcec6ee3d595b46b7 expref: 23 pid: 10287 timeout: 0 lvb_type: 0 [Wed Feb 6 02:15:23 2019][1669822.745079] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 18 previous similar messages [Wed Feb 6 02:15:23 2019][1669822.853090] LustreError: 17218:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 20 previous similar messages [Wed Feb 6 02:16:03 2019][1669863.129054] LNet: Service thread pid 9174 was inactive for 302.79s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 02:16:03 2019][1669863.146077] LNet: Skipped 5 previous similar messages [Wed Feb 6 02:16:03 2019][1669863.151311] Pid: 9174, comm: mdt00_023 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:16:03 2019][1669863.161134] Call Trace: [Wed Feb 6 02:16:03 2019][1669863.163772] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:16:03 2019][1669863.170902] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:16:03 2019][1669863.178180] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:16:03 2019][1669863.185026] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:16:03 2019][1669863.191761] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:16:03 2019][1669863.198150] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:16:03 2019][1669863.204450] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:16:03 2019][1669863.211886] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:16:03 2019][1669863.218802] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:16:03 2019][1669863.227018] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:16:03 2019][1669863.233415] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:16:03 2019][1669863.239637] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:16:03 2019][1669863.246380] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:16:03 2019][1669863.252248] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:16:04 2019][1669863.259374] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:16:04 2019][1669863.267259] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:16:04 2019][1669863.273759] [] kthread+0xd1/0xe0 [Wed Feb 6 02:16:04 2019][1669863.278847] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:16:04 2019][1669863.285496] [] 0xffffffffffffffff [Wed Feb 6 02:16:04 2019][1669863.290698] LustreError: dumping log to /tmp/lustre-log.1549448201.9174 [Wed Feb 6 02:16:19 2019][1669878.489440] Pid: 15164, comm: mdt01_042 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:16:19 2019][1669878.499355] Call Trace: [Wed Feb 6 02:16:19 2019][1669878.501992] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:16:19 2019][1669878.509104] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:16:19 2019][1669878.516383] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:16:19 2019][1669878.523228] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:16:19 2019][1669878.529964] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:16:19 2019][1669878.536359] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:16:19 2019][1669878.542661] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:16:19 2019][1669878.550097] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:16:19 2019][1669878.557005] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:16:19 2019][1669878.565222] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:16:19 2019][1669878.571616] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:16:19 2019][1669878.577838] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:16:19 2019][1669878.584573] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:16:19 2019][1669878.590442] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:16:19 2019][1669878.597559] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:16:19 2019][1669878.605444] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:16:19 2019][1669878.611943] [] kthread+0xd1/0xe0 [Wed Feb 6 02:16:19 2019][1669878.617032] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:16:19 2019][1669878.623678] [] 0xffffffffffffffff [Wed Feb 6 02:16:19 2019][1669878.628873] LustreError: dumping log to /tmp/lustre-log.1549448217.15164 [Wed Feb 6 02:16:35 2019][1669894.873852] Pid: 6827, comm: mdt02_009 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:16:35 2019][1669894.883679] Call Trace: [Wed Feb 6 02:16:35 2019][1669894.886326] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:16:35 2019][1669894.893455] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 02:16:35 2019][1669894.900836] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 02:16:35 2019][1669894.909064] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:16:35 2019][1669894.915463] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:16:35 2019][1669894.921682] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:16:35 2019][1669894.928434] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:16:35 2019][1669894.934320] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:16:35 2019][1669894.941456] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:16:35 2019][1669894.949365] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:16:35 2019][1669894.955880] [] kthread+0xd1/0xe0 [Wed Feb 6 02:16:35 2019][1669894.960965] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:16:35 2019][1669894.967611] [] 0xffffffffffffffff [Wed Feb 6 02:16:35 2019][1669894.972810] LustreError: dumping log to /tmp/lustre-log.1549448233.6827 [Wed Feb 6 02:16:36 2019][1669895.897874] Pid: 15510, comm: mdt02_048 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:16:36 2019][1669895.907791] Call Trace: [Wed Feb 6 02:16:36 2019][1669895.910428] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:16:36 2019][1669895.917556] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:16:36 2019][1669895.924827] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:16:36 2019][1669895.931675] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:16:36 2019][1669895.938416] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:16:36 2019][1669895.944813] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:16:36 2019][1669895.951123] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:16:36 2019][1669895.958560] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:16:36 2019][1669895.965467] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:16:36 2019][1669895.973684] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:16:36 2019][1669895.980079] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:16:36 2019][1669895.986301] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:16:36 2019][1669895.993037] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:16:36 2019][1669895.998912] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:16:36 2019][1669896.006034] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:16:36 2019][1669896.013922] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:16:36 2019][1669896.020424] [] kthread+0xd1/0xe0 [Wed Feb 6 02:16:36 2019][1669896.025513] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:16:36 2019][1669896.032158] [] 0xffffffffffffffff [Wed Feb 6 02:16:36 2019][1669896.037352] LustreError: dumping log to /tmp/lustre-log.1549448234.15510 [Wed Feb 6 02:16:51 2019][1669911.258261] Pid: 15228, comm: mdt00_036 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:16:51 2019][1669911.268173] Call Trace: [Wed Feb 6 02:16:51 2019][1669911.270824] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:16:51 2019][1669911.277966] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 02:16:52 2019][1669911.285339] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 02:16:52 2019][1669911.293564] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:16:52 2019][1669911.299960] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:16:52 2019][1669911.306183] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:16:52 2019][1669911.312926] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:16:52 2019][1669911.318802] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:16:52 2019][1669911.325943] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:16:52 2019][1669911.333850] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:16:52 2019][1669911.340367] [] kthread+0xd1/0xe0 [Wed Feb 6 02:16:52 2019][1669911.345453] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:16:52 2019][1669911.352102] [] 0xffffffffffffffff [Wed Feb 6 02:16:52 2019][1669911.357295] LustreError: dumping log to /tmp/lustre-log.1549448249.15228 [Wed Feb 6 02:17:52 2019][1669971.755884] Lustre: fir-MDT0000: Connection restored to 10.0.10.51@o2ib7 (at 0@lo) [Wed Feb 6 02:17:52 2019][1669971.763632] Lustre: Skipped 24 previous similar messages [Wed Feb 6 02:18:13 2019][1669992.909302] LNetError: 6177:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.112.17@tcp from [Wed Feb 6 02:18:13 2019][1669992.919992] LNetError: 6177:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 174516271 previous similar messages [Wed Feb 6 02:18:21 2019][1670000.931502] Lustre: 15169:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549448339/real 1549448339] req@ffff912146840f00 x1623682053431808/t0(0) o104->fir-MDT0002@10.10.113.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549448350 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 02:18:21 2019][1670000.959185] Lustre: 15169:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 176062608 previous similar messages [Wed Feb 6 02:19:48 2019][1670087.390681] LNet: Service thread pid 15242 was inactive for 566.20s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 02:19:48 2019][1670087.403715] LNet: Skipped 5 previous similar messages [Wed Feb 6 02:19:48 2019][1670087.408953] LustreError: dumping log to /tmp/lustre-log.1549448426.15242 [Wed Feb 6 02:19:50 2019][1670089.438732] LustreError: dumping log to /tmp/lustre-log.1549448428.15260 [Wed Feb 6 02:19:58 2019][1670097.630937] LustreError: dumping log to /tmp/lustre-log.1549448436.15292 [Wed Feb 6 02:20:16 2019][1670115.519442] Lustre: 9008:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 02:20:16 2019][1670115.519442] req@ffff912df54b5100 x1624702055531360/t0(0) o36->169b1d1d-2dc2-d613-36bc-7e8f822bec38@10.9.113.1@o2ib4:464/0 lens 528/2888 e 9 to 0 dl 1549448459 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 02:20:16 2019][1670115.548434] Lustre: 9008:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 11 previous similar messages [Wed Feb 6 02:20:22 2019][1670121.879986] Lustre: fir-MDT0002: Client 0fa57d46-9654-e504-7c22-8305cb443bcd (at 10.9.101.11@o2ib4) reconnecting [Wed Feb 6 02:20:22 2019][1670121.890332] Lustre: Skipped 4 previous similar messages [Wed Feb 6 02:20:25 2019][1670124.753056] Lustre: 15384:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (382:222s); client may timeout. req@ffff912df54b5d00 x1624705157188928/t0(0) o36->492310fb-8028-83a2-d5b0-7c5235cdad98@10.10.29.8@tcp:246/0 lens 520/424 e 0 to 0 dl 1549448241 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 02:20:25 2019][1670124.754018] LNet: Service thread pid 15222 completed after 603.92s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 02:20:25 2019][1670124.754021] LNet: Skipped 35 previous similar messages [Wed Feb 6 02:22:47 2019][1670266.979202] Lustre: 15165:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 02:22:47 2019][1670266.979202] req@ffff915999f39200 x1624673899042656/t0(0) o36->824bd29d-65a6-aa3d-238a-0a9d670705d8@10.9.101.35@o2ib4:615/0 lens 520/2888 e 2 to 0 dl 1549448610 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 02:22:47 2019][1670267.008364] Lustre: 15165:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 33 previous similar messages [Wed Feb 6 02:22:52 2019][1670271.715308] LNet: Service thread pid 7029 was inactive for 599.90s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 02:22:52 2019][1670271.732325] LNet: Skipped 4 previous similar messages [Wed Feb 6 02:22:52 2019][1670271.737553] Pid: 7029, comm: mdt02_010 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:22:52 2019][1670271.747381] Call Trace: [Wed Feb 6 02:22:52 2019][1670271.750013] [] 0xffffffffffffffff [Wed Feb 6 02:22:52 2019][1670271.755214] LustreError: dumping log to /tmp/lustre-log.1549448610.7029 [Wed Feb 6 02:22:52 2019][1670272.221206] Pid: 15331, comm: mdt02_043 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:22:52 2019][1670272.231120] Call Trace: [Wed Feb 6 02:22:52 2019][1670272.233761] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:22:52 2019][1670272.240878] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:22:52 2019][1670272.248158] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:22:52 2019][1670272.255006] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:22:52 2019][1670272.261750] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:22:53 2019][1670272.268143] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:22:53 2019][1670272.274454] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:22:53 2019][1670272.281890] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:22:53 2019][1670272.288808] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:22:53 2019][1670272.297023] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:22:53 2019][1670272.303435] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:22:53 2019][1670272.309660] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:22:53 2019][1670272.316392] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:22:53 2019][1670272.322260] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:22:53 2019][1670272.329375] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:22:53 2019][1670272.337263] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:22:53 2019][1670272.343764] [] kthread+0xd1/0xe0 [Wed Feb 6 02:22:53 2019][1670272.348849] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:22:53 2019][1670272.355498] [] 0xffffffffffffffff [Wed Feb 6 02:22:53 2019][1670272.360693] Pid: 6173, comm: mdt02_000 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:22:53 2019][1670272.370535] Call Trace: [Wed Feb 6 02:22:53 2019][1670272.373168] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:22:53 2019][1670272.380268] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:22:53 2019][1670272.387538] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:22:53 2019][1670272.394378] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:22:53 2019][1670272.401113] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:22:53 2019][1670272.407501] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:22:53 2019][1670272.413799] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:22:53 2019][1670272.421227] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:22:53 2019][1670272.428144] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:22:53 2019][1670272.436375] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:22:53 2019][1670272.442775] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:22:53 2019][1670272.448998] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:22:53 2019][1670272.455731] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:22:53 2019][1670272.461607] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:22:53 2019][1670272.468714] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:22:53 2019][1670272.476603] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:22:53 2019][1670272.483101] [] kthread+0xd1/0xe0 [Wed Feb 6 02:22:53 2019][1670272.488188] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:22:53 2019][1670272.494828] [] 0xffffffffffffffff [Wed Feb 6 02:22:53 2019][1670272.500019] Pid: 15184, comm: mdt02_032 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:22:53 2019][1670272.509926] Call Trace: [Wed Feb 6 02:22:53 2019][1670272.512558] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:22:53 2019][1670272.519667] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:22:53 2019][1670272.526940] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:22:53 2019][1670272.533776] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:22:53 2019][1670272.540510] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:22:53 2019][1670272.546899] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:22:53 2019][1670272.553199] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:22:53 2019][1670272.560627] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:22:53 2019][1670272.567548] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:22:53 2019][1670272.575767] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:22:53 2019][1670272.582165] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:22:53 2019][1670272.588386] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:22:53 2019][1670272.595129] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:22:53 2019][1670272.600996] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:22:53 2019][1670272.608104] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:22:53 2019][1670272.615990] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:22:53 2019][1670272.622489] [] kthread+0xd1/0xe0 [Wed Feb 6 02:22:53 2019][1670272.627570] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:22:53 2019][1670272.634231] [] 0xffffffffffffffff [Wed Feb 6 02:22:53 2019][1670272.853760] Lustre: fir-MDT0002: Client 97d93834-3c87-3fed-069a-aff0074e2ab1 (at 10.10.112.12@tcp) reconnecting [Wed Feb 6 02:22:53 2019][1670272.864021] Lustre: Skipped 21 previous similar messages [Wed Feb 6 02:22:54 2019][1670273.766357] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 02:22:54 2019][1670273.777364] LustreError: 8973:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549448312, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff912b2cb2da00/0x21ed93518283a17e lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 190 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 8973 timeout: 0 lvb_type: 0 [Wed Feb 6 02:22:54 2019][1670273.777367] LustreError: 8973:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 19 previous similar messages [Wed Feb 6 02:22:54 2019][1670273.801461] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 02:22:54 2019][1670273.801463] Lustre: Skipped 5 previous similar messages [Wed Feb 6 02:22:54 2019][1670273.848225] Lustre: Skipped 1 previous similar message [Wed Feb 6 02:22:54 2019][1670273.853549] LustreError: 6683:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549448312, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff912b27103a80/0x21ed93518283a218 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 46 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518283a21f expref: -99 pid: 6683 timeout: 0 lvb_type: 0 [Wed Feb 6 02:22:54 2019][1670273.893623] LustreError: 6683:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 57 previous similar messages [Wed Feb 6 02:22:56 2019][1670275.756613] LustreError: 9199:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9124134d1000 ns: mdt-fir-MDT0002_UUID lock: ffff9125fb6f72c0/0x21ed93518285267c lrc: 3/0,0 mode: --/PW res: [0x2c00016df:0x5:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x54a01000000000 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e15082 expref: 7 pid: 9199 timeout: 0 lvb_type: 0 [Wed Feb 6 02:22:56 2019][1670275.756855] LNet: Service thread pid 7029 completed after 603.94s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 02:22:56 2019][1670275.756857] LNet: Skipped 14 previous similar messages [Wed Feb 6 02:22:56 2019][1670275.811409] Pid: 15267, comm: mdt02_039 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:22:56 2019][1670275.811410] Call Trace: [Wed Feb 6 02:22:56 2019][1670275.811452] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 02:22:56 2019][1670275.811480] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 02:22:56 2019][1670275.811507] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 02:22:56 2019][1670275.811518] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 02:22:56 2019][1670275.811529] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 02:22:56 2019][1670275.811539] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 02:22:56 2019][1670275.811554] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 02:22:56 2019][1670275.811564] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 02:22:56 2019][1670275.811576] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 02:22:56 2019][1670275.811587] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 02:22:56 2019][1670275.811597] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 02:22:56 2019][1670275.811606] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 02:22:56 2019][1670275.811616] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 02:22:56 2019][1670275.811655] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 02:22:56 2019][1670275.811686] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 02:22:56 2019][1670275.811717] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 02:22:56 2019][1670275.811721] [] kthread+0xd1/0xe0 [Wed Feb 6 02:22:56 2019][1670275.811724] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 02:22:56 2019][1670275.811746] [] 0xffffffffffffffff [Wed Feb 6 02:22:56 2019][1670275.811750] LustreError: dumping log to /tmp/lustre-log.1549448614.15267 [Wed Feb 6 02:22:56 2019][1670275.957852] LustreError: 9199:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 11 previous similar messages [Wed Feb 6 02:23:00 2019][1670279.907510] LNet: Service thread pid 15389 was inactive for 605.18s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 02:23:00 2019][1670279.920544] LNet: Skipped 9 previous similar messages [Wed Feb 6 02:23:00 2019][1670279.925770] LustreError: dumping log to /tmp/lustre-log.1549448618.15389 [Wed Feb 6 02:23:02 2019][1670281.955562] LustreError: dumping log to /tmp/lustre-log.1549448620.15187 [Wed Feb 6 02:23:27 2019][1670306.532177] LustreError: dumping log to /tmp/lustre-log.1549448645.9005 [Wed Feb 6 02:23:29 2019][1670308.580227] LustreError: dumping log to /tmp/lustre-log.1549448647.6758 [Wed Feb 6 02:23:35 2019][1670314.724388] LustreError: dumping log to /tmp/lustre-log.1549448653.10287 [Wed Feb 6 02:23:43 2019][1670322.916587] LustreError: dumping log to /tmp/lustre-log.1549448661.17205 [Wed Feb 6 02:25:18 2019][1670417.638978] Lustre: 6684:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 02:25:18 2019][1670417.638978] req@ffff914f37bb5d00 x1624701997821072/t0(0) o36->797f0075-ec92-4d37-f23e-cc9ca768ea89@10.9.113.5@o2ib4:11/0 lens 544/2888 e 1 to 0 dl 1549448761 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 02:25:18 2019][1670417.667877] Lustre: 6684:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 12 previous similar messages [Wed Feb 6 02:25:24 2019][1670423.820550] Lustre: fir-MDT0002: Client cf0229df-1948-b94a-5e59-bdab7e865d86 (at 10.9.103.11@o2ib4) reconnecting [Wed Feb 6 02:25:24 2019][1670423.830893] Lustre: Skipped 22 previous similar messages [Wed Feb 6 02:25:27 2019][1670426.760202] LustreError: 15389:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.29.8@tcp) failed to reply to blocking AST (req@ffff913cd71a1800 x1623682053551904 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff91281a075a00/0x21ed935182861746 lrc: 4/0,0 mode: PR/PR res: [0x200001761:0x2:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d6737 expref: 12 pid: 15260 timeout: 1670533 lvb_type: 0 [Wed Feb 6 02:25:27 2019][1670426.802544] LustreError: 15389:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 13 previous similar messages [Wed Feb 6 02:25:27 2019][1670426.812892] LustreError: 138-a: fir-MDT0000: A client on nid 10.10.29.8@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 02:25:27 2019][1670426.825403] LustreError: Skipped 15 previous similar messages [Wed Feb 6 02:25:27 2019][1670426.831350] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.29.8@tcp ns: mdt-fir-MDT0000_UUID lock: ffff91281a075a00/0x21ed935182861746 lrc: 3/0,0 mode: PR/PR res: [0x200001761:0x2:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d6737 expref: 13 pid: 15260 timeout: 0 lvb_type: 0 [Wed Feb 6 02:25:27 2019][1670426.868136] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 14 previous similar messages [Wed Feb 6 02:25:27 2019][1670426.878358] Lustre: 15389:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (557:195s); client may timeout. req@ffff9135fba42a00 x1624705157199760/t0(0) o36->492310fb-8028-83a2-d5b0-7c5235cdad98@10.10.29.8@tcp:575/0 lens 520/424 e 1 to 0 dl 1549448570 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 02:25:27 2019][1670426.880903] LNet: Service thread pid 15267 completed after 750.79s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 02:25:27 2019][1670426.880905] LNet: Skipped 3 previous similar messages [Wed Feb 6 02:25:27 2019][1670426.928543] Lustre: 15389:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1 previous similar message [Wed Feb 6 02:25:28 2019][1670427.367215] LustreError: dumping log to /tmp/lustre-log.1549448765.15259 [Wed Feb 6 02:27:54 2019][1670574.242340] Lustre: fir-MDT0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) [Wed Feb 6 02:27:54 2019][1670574.251129] Lustre: Skipped 66 previous similar messages [Wed Feb 6 02:27:56 2019][1670575.850946] Lustre: 17219:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-2), not sending early reply [Wed Feb 6 02:27:56 2019][1670575.850946] req@ffff9155c7b7a700 x1624697923656368/t0(0) o36->e78a174a-ae69-cb32-9b42-0306c7153992@10.9.103.7@o2ib4:169/0 lens 528/2888 e 0 to 0 dl 1549448919 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 02:27:56 2019][1670575.880103] Lustre: 17219:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [Wed Feb 6 02:28:02 2019][1670581.824168] Lustre: fir-MDT0002: Client f05f50bb-5878-44e8-e806-fa2968d0f6ca (at 10.9.105.1@o2ib4) reconnecting [Wed Feb 6 02:28:02 2019][1670581.834427] Lustre: Skipped 1 previous similar message [Wed Feb 6 02:28:13 2019][1670592.924357] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 02:28:13 2019][1670592.935052] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 173716668 previous similar messages [Wed Feb 6 02:28:21 2019][1670600.946558] Lustre: 8967:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549448939/real 1549448939] req@ffff911e1a1ba400 x1623682053674128/t0(0) o104->fir-MDT0002@10.10.113.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549448950 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 02:28:21 2019][1670600.974158] Lustre: 8967:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 174701398 previous similar messages [Wed Feb 6 02:32:56 2019][1670875.890475] Lustre: 17220:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 02:32:56 2019][1670875.890475] req@ffff915997a2b900 x1624669465202336/t0(0) o36->d5a74680-e7af-ebfb-7dfd-72e2645d277b@10.9.101.51@o2ib4:469/0 lens 520/2888 e 0 to 0 dl 1549449219 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 02:32:56 2019][1670875.919889] Lustre: 17220:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 23 previous similar messages [Wed Feb 6 02:33:02 2019][1670881.913567] Lustre: fir-MDT0002: Client 173446cc-39b1-333f-81fc-6684fb678e20 (at 10.8.3.19@o2ib6) reconnecting [Wed Feb 6 02:33:02 2019][1670881.923741] Lustre: Skipped 19 previous similar messages [Wed Feb 6 02:33:07 2019][1670886.774740] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 02:33:07 2019][1670886.789852] Lustre: Skipped 1 previous similar message [Wed Feb 6 02:33:07 2019][1670886.795183] LustreError: 6795:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549448925, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff913874c80900/0x21ed935182b6fe1e lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 49 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935182b6fe25 expref: -99 pid: 6795 timeout: 0 lvb_type: 0 [Wed Feb 6 02:33:07 2019][1670886.795263] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Feb 6 02:33:07 2019][1670886.795265] Lustre: Skipped 5 previous similar messages [Wed Feb 6 02:33:07 2019][1670886.850650] LustreError: 6795:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 45 previous similar messages [Wed Feb 6 02:33:07 2019][1670887.101750] LustreError: 9009:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549448925, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9134b10df500/0x21ed935182b7b1a8 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 197 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 9009 timeout: 0 lvb_type: 0 [Wed Feb 6 02:33:07 2019][1670887.141047] LustreError: 9009:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 16 previous similar messages [Wed Feb 6 02:33:09 2019][1670888.772173] LustreError: 6172:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9136a5ab8400 ns: mdt-fir-MDT0002_UUID lock: ffff912e9ae70fc0/0x21ed935182c79c65 lrc: 1/0,0 mode: EX/EX res: [0x2c00016df:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e153fb expref: 4 pid: 6172 timeout: 0 lvb_type: 3 [Wed Feb 6 02:33:09 2019][1670888.806359] LustreError: 6172:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 5 previous similar messages [Wed Feb 6 02:35:40 2019][1671039.808932] Lustre: 15370:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:160s); client may timeout. req@ffff9153bef16000 x1624702067094064/t0(0) o36->db7578a7-294b-6d6d-61e8-bb06998c6482@10.10.113.6@tcp:468/0 lens 544/424 e 0 to 0 dl 1549449218 ref 1 fl Complete:/0/0 rc -2/-2 [Wed Feb 6 02:35:40 2019][1671039.837670] Lustre: 15370:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1 previous similar message [Wed Feb 6 02:38:03 2019][1671182.964408] Lustre: fir-MDT0002: Connection restored to 97a504ab-e280-8a5f-853e-e38376475af1 (at 10.9.101.9@o2ib4) [Wed Feb 6 02:38:03 2019][1671182.974932] Lustre: Skipped 64 previous similar messages [Wed Feb 6 02:38:11 2019][1671190.779372] LustreError: 15257:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff91512db9b000 x1623682053842144 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff9133f70f8480/0x21ed935182d25bea lrc: 4/0,0 mode: PR/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 29 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e15425 expref: 13 pid: 6172 timeout: 1671297 lvb_type: 0 [Wed Feb 6 02:38:11 2019][1671190.779376] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.15@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 02:38:11 2019][1671190.779379] LustreError: Skipped 10 previous similar messages [Wed Feb 6 02:38:11 2019][1671190.779399] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.15@tcp ns: mdt-fir-MDT0002_UUID lock: ffff9135b2531f80/0x21ed935182dd0087 lrc: 3/0,0 mode: PR/PR res: [0x2c00016df:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e15433 expref: 14 pid: 6172 timeout: 0 lvb_type: 0 [Wed Feb 6 02:38:11 2019][1671190.779401] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 7 previous similar messages [Wed Feb 6 02:38:11 2019][1671190.887782] LustreError: 15257:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 11 previous similar messages [Wed Feb 6 02:38:13 2019][1671192.939414] LNetError: 6172:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.15@tcp from [Wed Feb 6 02:38:13 2019][1671192.939417] LNetError: 15267:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.15@tcp from [Wed Feb 6 02:38:13 2019][1671192.939422] LNetError: 15267:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 141980125 previous similar messages [Wed Feb 6 02:38:13 2019][1671192.972196] LNetError: 6172:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 6466 previous similar messages [Wed Feb 6 02:38:21 2019][1671200.961615] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549449539/real 1549449539] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549449550 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 02:38:21 2019][1671200.989296] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 142471172 previous similar messages [Wed Feb 6 02:43:11 2019][1671490.950905] LustreError: 15266:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549449529, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff91499bf05a00/0x21ed935182ebac8c lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 48 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935182ebac93 expref: -99 pid: 15266 timeout: 0 lvb_type: 0 [Wed Feb 6 02:43:11 2019][1671490.976906] LustreError: 15169:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549449529, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9120668e0480/0x21ed935182ebb55a lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 182 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 15169 timeout: 0 lvb_type: 0 [Wed Feb 6 02:43:11 2019][1671490.976909] LustreError: 15169:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 10 previous similar messages [Wed Feb 6 02:43:11 2019][1671491.041753] LustreError: 15266:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 33 previous similar messages [Wed Feb 6 02:43:13 2019][1671492.787332] LustreError: 15186:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff912a7436a400 ns: mdt-fir-MDT0000_UUID lock: ffff911f0c3dcc80/0x21ed935182fbe24a lrc: 1/0,0 mode: EX/EX res: [0x200001807:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d69c9 expref: 4 pid: 15186 timeout: 0 lvb_type: 3 [Wed Feb 6 02:43:13 2019][1671492.821521] LustreError: 15186:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 2 previous similar messages [Wed Feb 6 02:45:42 2019][1671641.896687] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 02:45:42 2019][1671641.907332] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 02:45:42 2019][1671641.907335] Lustre: Skipped 4 previous similar messages [Wed Feb 6 02:45:42 2019][1671641.928233] Lustre: Skipped 1 previous similar message [Wed Feb 6 02:48:10 2019][1671789.321390] LNet: Service thread pid 6615 was inactive for 1201.62s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 02:48:10 2019][1671789.338498] LNet: Skipped 4 previous similar messages [Wed Feb 6 02:48:10 2019][1671789.343732] Pid: 6615, comm: mdt00_004 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 02:48:10 2019][1671789.353558] Call Trace: [Wed Feb 6 02:48:10 2019][1671789.356195] [] 0xffffffffffffffff [Wed Feb 6 02:48:10 2019][1671789.361402] LustreError: dumping log to /tmp/lustre-log.1549450127.6615 [Wed Feb 6 02:48:13 2019][1671792.954468] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 02:48:13 2019][1671792.965165] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 195808080 previous similar messages [Wed Feb 6 02:48:15 2019][1671794.794532] LustreError: 8829:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.29.8@tcp) failed to reply to blocking AST (req@ffff913d43ad2d00 x1623682054114816 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff9138ba5f1200/0x21ed9351832a0af2 lrc: 4/0,0 mode: CR/CR res: [0x200001809:0x1:0x0].0x0 bits 0x9/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d6aa2 expref: 15 pid: 15243 timeout: 1671901 lvb_type: 0 [Wed Feb 6 02:48:15 2019][1671794.794549] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.112.12@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 02:48:15 2019][1671794.794551] LustreError: Skipped 8 previous similar messages [Wed Feb 6 02:48:15 2019][1671794.794577] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.12@tcp ns: mdt-fir-MDT0002_UUID lock: ffff9150dd6be300/0x21ed9351832a091d lrc: 3/0,0 mode: CR/CR res: [0x2c00016ef:0x1:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT flags: 0x60000400000020 nid: 10.10.112.12@tcp remote: 0xefba423e1327f08e expref: 8 pid: 6176 timeout: 0 lvb_type: 3 [Wed Feb 6 02:48:15 2019][1671794.794580] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 7 previous similar messages [Wed Feb 6 02:48:15 2019][1671794.823911] Lustre: fir-MDT0002: Connection restored to 034162e2-29a3-b06b-c62e-a56d63789e61 (at 10.10.112.12@tcp) [Wed Feb 6 02:48:15 2019][1671794.823913] Lustre: Skipped 28 previous similar messages [Wed Feb 6 02:48:15 2019][1671794.918134] LustreError: 8829:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 9 previous similar messages [Wed Feb 6 02:48:21 2019][1671800.976674] Lustre: 15386:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549450139/real 1549450139] req@ffff911cd96c5700 x1623682054176912/t0(0) o104->fir-MDT0000@10.10.29.8@tcp:15/16 lens 296/224 e 0 to 1 dl 1549450150 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 02:48:21 2019][1671801.004267] Lustre: 15386:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 198693396 previous similar messages [Wed Feb 6 02:53:15 2019][1672094.938061] LustreError: 9182:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549450133, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff911faba10b40/0x21ed93518335fc47 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 200 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 9182 timeout: 0 lvb_type: 0 [Wed Feb 6 02:53:15 2019][1672094.977506] LustreError: 9182:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 46 previous similar messages [Wed Feb 6 02:53:16 2019][1672095.818086] LustreError: 9199:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549450134, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff913e0e460d80/0x21ed935183361f71 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 41 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935183361f7f expref: -99 pid: 9199 timeout: 0 lvb_type: 0 [Wed Feb 6 02:53:16 2019][1672095.858163] LustreError: 9199:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 54 previous similar messages [Wed Feb 6 02:53:17 2019][1672096.802236] LustreError: 15382:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9137ec5de800 ns: mdt-fir-MDT0002_UUID lock: ffff912c898e7500/0x21ed935183521a93 lrc: 1/0,0 mode: --/PW res: [0x2c00016bf:0x3:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x54a01000000000 nid: 10.10.112.12@tcp remote: 0xefba423e1327f247 expref: 4 pid: 15382 timeout: 0 lvb_type: 0 [Wed Feb 6 02:53:17 2019][1672096.836683] LustreError: 15382:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 5 previous similar messages [Wed Feb 6 02:53:20 2019][1672099.673884] Lustre: fir-MDT0002: Client 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) reconnecting [Wed Feb 6 02:53:20 2019][1672099.684061] Lustre: Skipped 38 previous similar messages [Wed Feb 6 02:55:47 2019][1672246.823891] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 02:55:47 2019][1672246.839013] Lustre: Skipped 1 previous similar message [Wed Feb 6 02:55:47 2019][1672246.840296] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 02:55:47 2019][1672246.840298] Lustre: Skipped 5 previous similar messages [Wed Feb 6 02:58:12 2019][1672391.360504] Lustre: 17222:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 02:58:12 2019][1672391.360504] req@ffff91537db27800 x1624699474990112/t0(0) o36->64e61a89-bb20-a6be-0cca-fdf4cefe7623@10.9.105.11@o2ib4:474/0 lens 520/2888 e 2 to 0 dl 1549450734 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 02:58:12 2019][1672391.389658] Lustre: 17222:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 50 previous similar messages [Wed Feb 6 02:58:13 2019][1672392.969525] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 02:58:13 2019][1672392.980220] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 190878299 previous similar messages [Wed Feb 6 02:58:18 2019][1672397.901471] Lustre: fir-MDT0000: Client 64e61a89-bb20-a6be-0cca-fdf4cefe7623 (at 10.9.105.11@o2ib4) reconnecting [Wed Feb 6 02:58:18 2019][1672397.911838] Lustre: fir-MDT0000: Connection restored to 43d58128-0571-5875-44a2-2bdc857dd6a9 (at 10.9.105.11@o2ib4) [Wed Feb 6 02:58:18 2019][1672397.922446] Lustre: Skipped 21 previous similar messages [Wed Feb 6 02:58:19 2019][1672398.809693] LustreError: 6883:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.112.12@tcp) failed to reply to blocking AST (req@ffff912af276e300 x1623682054351280 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff914c1a3906c0/0x21ed93518372a81a lrc: 4/0,0 mode: PR/PR res: [0x2c00016bf:0x3:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.112.12@tcp remote: 0xefba423e1327f2e8 expref: 14 pid: 17223 timeout: 1672505 lvb_type: 0 [Wed Feb 6 02:58:19 2019][1672398.852293] LustreError: 6883:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 12 previous similar messages [Wed Feb 6 02:58:19 2019][1672398.862561] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.112.12@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 02:58:19 2019][1672398.875265] LustreError: Skipped 13 previous similar messages [Wed Feb 6 02:58:19 2019][1672398.881213] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.12@tcp ns: mdt-fir-MDT0002_UUID lock: ffff914c1a3906c0/0x21ed93518372a81a lrc: 3/0,0 mode: PR/PR res: [0x2c00016bf:0x3:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.112.12@tcp remote: 0xefba423e1327f2e8 expref: 15 pid: 17223 timeout: 0 lvb_type: 0 [Wed Feb 6 02:58:19 2019][1672398.918363] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 10 previous similar messages [Wed Feb 6 02:58:20 2019][1672399.834106] Lustre: 15182:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:2s); client may timeout. req@ffff91384f98b000 x1624669686360032/t13600081818(0) o36->fda7a4af-47c0-0068-cddf-309c3a9c784c@10.9.101.13@o2ib4:476/0 lens 520/424 e 2 to 0 dl 1549450736 ref 1 fl Complete:/0/0 rc 0/0 [Wed Feb 6 02:58:21 2019][1672400.991735] Lustre: 6684:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549450739/real 1549450739] req@ffff91215498fb00 x1623682054402880/t0(0) o104->fir-MDT0002@10.10.113.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549450750 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 02:58:21 2019][1672401.019328] Lustre: 6684:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 191486556 previous similar messages [Wed Feb 6 03:03:20 2019][1672699.824246] LustreError: 17052:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549450738, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff91409fbd4a40/0x21ed9351837bcd0e lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 49 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351837bcd31 expref: -99 pid: 17052 timeout: 0 lvb_type: 0 [Wed Feb 6 03:03:20 2019][1672699.864499] LustreError: 17052:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 37 previous similar messages [Wed Feb 6 03:03:20 2019][1672699.874240] LustreError: 8573:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549450738, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9124084cad00/0x21ed9351837be8dd lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 170 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 8573 timeout: 0 lvb_type: 0 [Wed Feb 6 03:03:20 2019][1672699.874242] LustreError: 8573:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 14 previous similar messages [Wed Feb 6 03:03:22 2019][1672701.817397] LustreError: 15510:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9157b1210800 ns: mdt-fir-MDT0002_UUID lock: ffff913c521f5c40/0x21ed9351838d8cb6 lrc: 1/0,0 mode: --/PW res: [0x2c0001712:0x4:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT flags: 0x54a01000000000 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e15b8e expref: 6 pid: 15510 timeout: 0 lvb_type: 0 [Wed Feb 6 03:03:22 2019][1672701.851850] LustreError: 15510:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 6 previous similar messages [Wed Feb 6 03:05:56 2019][1672856.274847] Lustre: fir-MDT0002: Client 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) reconnecting [Wed Feb 6 03:05:56 2019][1672856.285025] Lustre: Skipped 6 previous similar messages [Wed Feb 6 03:08:13 2019][1672992.984583] LNetError: 9027:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.15@tcp from [Wed Feb 6 03:08:13 2019][1672992.995280] LNetError: 9027:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 193567921 previous similar messages [Wed Feb 6 03:08:21 2019][1673001.006786] Lustre: 15389:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549451339/real 1549451339] req@ffff913650d5ec00 x1623682054602304/t0(0) o104->fir-MDT0000@10.10.29.8@tcp:15/16 lens 296/224 e 0 to 1 dl 1549451350 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 03:08:21 2019][1673001.034382] Lustre: 15389:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 196134274 previous similar messages [Wed Feb 6 03:08:24 2019][1673003.824887] LustreError: 9027:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff9128c7104800 x1623682054580336 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff913ae37e86c0/0x21ed935183cac03e lrc: 4/0,0 mode: PR/PR res: [0x2c0001712:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e15bc6 expref: 13 pid: 53422 timeout: 1673110 lvb_type: 0 [Wed Feb 6 03:08:24 2019][1673003.824892] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.15@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 03:08:24 2019][1673003.824894] LustreError: Skipped 12 previous similar messages [Wed Feb 6 03:08:24 2019][1673003.824919] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.15@tcp ns: mdt-fir-MDT0002_UUID lock: ffff913b95811200/0x21ed935183ae0694 lrc: 3/0,0 mode: PR/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 37 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e15bb8 expref: 14 pid: 8829 timeout: 0 lvb_type: 0 [Wed Feb 6 03:08:24 2019][1673003.824922] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 9 previous similar messages [Wed Feb 6 03:08:24 2019][1673003.933274] LustreError: 9027:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 13 previous similar messages [Wed Feb 6 03:08:24 2019][1673003.943978] Lustre: fir-MDT0002: Connection restored to e6807a0b-87d6-292a-3e89-235f77ec28d8 (at 10.10.113.15@tcp) [Wed Feb 6 03:08:24 2019][1673003.954500] Lustre: Skipped 25 previous similar messages [Wed Feb 6 03:08:30 2019][1673010.315167] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 03:08:30 2019][1673010.326207] Lustre: Skipped 5 previous similar messages [Wed Feb 6 03:10:53 2019][1673152.855616] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 03:10:53 2019][1673152.870731] Lustre: Skipped 1 previous similar message [Wed Feb 6 03:11:15 2019][1673174.829295] LustreError: 15260:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9132e7006f00 x1623682054739344/t0(0) o104->fir-MDT0002@10.10.112.17@tcp:15/16 lens 296/224 e 0 to 1 dl 1549451524 ref 1 fl Rpc:eX/2/ffffffff rc -5/-1 [Wed Feb 6 03:13:25 2019][1673305.245439] LustreError: 6804:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549451343, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff91433e3b5580/0x21ed935183eb6821 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 62 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935183eb6828 expref: -99 pid: 6804 timeout: 0 lvb_type: 0 [Wed Feb 6 03:13:25 2019][1673305.285516] LustreError: 6804:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 65 previous similar messages [Wed Feb 6 03:13:26 2019][1673305.799444] LustreError: 15266:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549451344, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff913e06928b40/0x21ed935183ebb8af lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 164 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 15266 timeout: 0 lvb_type: 0 [Wed Feb 6 03:13:26 2019][1673305.832531] LustreError: 7191:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff91456232c800 ns: mdt-fir-MDT0002_UUID lock: ffff91222681c140/0x21ed935184031c5d lrc: 3/0,0 mode: PW/PW res: [0x2c000171f:0x4:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT flags: 0x50200000000000 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7ed3a expref: 42 pid: 7191 timeout: 0 lvb_type: 0 [Wed Feb 6 03:13:26 2019][1673305.832533] LustreError: 7191:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 7 previous similar messages [Wed Feb 6 03:13:26 2019][1673305.883347] LustreError: 15266:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 13 previous similar messages [Wed Feb 6 03:15:42 2019][1673441.492036] Lustre: fir-MDT0002: haven't heard from client 8dd25978-9a6e-9f02-c6e4-dd789624ff48 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff912394264800, cur 1549451780 expire 1549451630 last 1549451553 [Wed Feb 6 03:15:42 2019][1673441.513908] Lustre: Skipped 1 previous similar message [Wed Feb 6 03:18:13 2019][1673592.999641] LNetError: 9019:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.29.8@tcp from [Wed Feb 6 03:18:13 2019][1673593.010165] LNetError: 9019:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 211185348 previous similar messages [Wed Feb 6 03:18:21 2019][1673601.021842] Lustre: 8973:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549451939/real 1549451939] req@ffff9132e7000c00 x1623682054875024/t0(0) o104->fir-MDT0002@10.10.113.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549451950 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 03:18:21 2019][1673601.049433] Lustre: 8973:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 213989686 previous similar messages [Wed Feb 6 03:18:26 2019][1673605.912441] Lustre: fir-MDT0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) [Wed Feb 6 03:18:26 2019][1673605.921229] Lustre: Skipped 20 previous similar messages [Wed Feb 6 03:18:28 2019][1673607.840033] LustreError: 8973:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.6@tcp) failed to reply to blocking AST (req@ffff9132e7000c00 x1623682054875024 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff91248fe6e780/0x21ed9351841a2f53 lrc: 4/0,0 mode: PR/PR res: [0x2c000171f:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7ee1a expref: 27 pid: 8985 timeout: 1673714 lvb_type: 0 [Wed Feb 6 03:18:28 2019][1673607.840040] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 03:18:28 2019][1673607.840043] LustreError: Skipped 16 previous similar messages [Wed Feb 6 03:18:28 2019][1673607.840064] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.6@tcp ns: mdt-fir-MDT0002_UUID lock: ffff911edd0a6540/0x21ed9351840c26bd lrc: 3/0,0 mode: PR/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 36 type: IBT flags: 0x60200400000020 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7ee0c expref: 28 pid: 15219 timeout: 0 lvb_type: 0 [Wed Feb 6 03:18:28 2019][1673607.840067] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 13 previous similar messages [Wed Feb 6 03:18:28 2019][1673607.948079] LustreError: 8973:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 16 previous similar messages [Wed Feb 6 03:18:32 2019][1673611.898819] Lustre: fir-MDT0002: Client 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) reconnecting [Wed Feb 6 03:20:59 2019][1673758.843976] LustreError: 15329:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff914ee3a86000 x1623682055019872/t0(0) o104->fir-MDT0002@10.10.113.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549452108 ref 1 fl Rpc:eX/2/ffffffff rc -5/-1 [Wed Feb 6 03:23:30 2019][1673909.847897] LustreError: 15169:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9126d57a5c00 ns: mdt-fir-MDT0002_UUID lock: ffff9122ca7c4ec0/0x21ed93518466c4a8 lrc: 1/0,0 mode: EX/EX res: [0x2c0001712:0x3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e160b9 expref: 4 pid: 15169 timeout: 0 lvb_type: 3 [Wed Feb 6 03:23:30 2019][1673909.882275] LustreError: 15169:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 12 previous similar messages [Wed Feb 6 03:23:51 2019][1673930.848192] LustreError: 8984:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff913d2f05d700 x1623682055427744/t0(0) o104->fir-MDT0002@10.10.112.17@tcp:15/16 lens 296/224 e 0 to 1 dl 1549452280 ref 1 fl Rpc:eX/2/ffffffff rc -5/-1 [Wed Feb 6 03:28:13 2019][1674193.014697] LNetError: 47983:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.15@tcp from [Wed Feb 6 03:28:13 2019][1674193.025472] LNetError: 47983:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 230078824 previous similar messages [Wed Feb 6 03:28:21 2019][1674201.036904] Lustre: 6694:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549452539/real 1549452539] req@ffff913d20d50900 x1623682055957744/t0(0) o104->fir-MDT0000@10.10.29.8@tcp:15/16 lens 296/224 e 0 to 1 dl 1549452550 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 03:28:21 2019][1674201.064427] Lustre: 6694:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 233484952 previous similar messages [Wed Feb 6 03:28:32 2019][1674211.855185] LustreError: 6694:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.29.8@tcp) failed to reply to blocking AST (req@ffff913d20d50900 x1623682055957744 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff9135f6d421c0/0x21ed9351850cfcf9 lrc: 4/0,0 mode: CR/CR res: [0x200001831:0x1:0x0].0x0 bits 0x9/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d72f2 expref: 27 pid: 6171 timeout: 1674318 lvb_type: 0 [Wed Feb 6 03:28:32 2019][1674211.855192] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.15@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 03:28:32 2019][1674211.855195] LustreError: Skipped 19 previous similar messages [Wed Feb 6 03:28:32 2019][1674211.855218] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.15@tcp ns: mdt-fir-MDT0002_UUID lock: ffff9139bcce3a80/0x21ed9351850cd94a lrc: 3/0,0 mode: PR/PR res: [0x2c0001712:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e161ca expref: 18 pid: 15231 timeout: 0 lvb_type: 0 [Wed Feb 6 03:28:32 2019][1674211.855221] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 16 previous similar messages [Wed Feb 6 03:28:32 2019][1674211.895121] Lustre: fir-MDT0002: Connection restored to e6807a0b-87d6-292a-3e89-235f77ec28d8 (at 10.10.113.15@tcp) [Wed Feb 6 03:28:32 2019][1674211.895124] Lustre: Skipped 18 previous similar messages [Wed Feb 6 03:28:32 2019][1674211.978885] LustreError: 6694:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 19 previous similar messages [Wed Feb 6 03:31:08 2019][1674367.498903] Lustre: fir-MDT0002: Client 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) reconnecting [Wed Feb 6 03:33:32 2019][1674512.322245] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 03:33:32 2019][1674512.333283] Lustre: Skipped 5 previous similar messages [Wed Feb 6 03:33:33 2019][1674512.571726] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 03:33:33 2019][1674512.586844] Lustre: Skipped 1 previous similar message [Wed Feb 6 03:33:33 2019][1674512.592180] LustreError: 9027:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549452551, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff91222faffbc0/0x21ed9351852d9ed1 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 53 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351852d9ed8 expref: -99 pid: 9027 timeout: 0 lvb_type: 0 [Wed Feb 6 03:33:33 2019][1674512.632258] LustreError: 9027:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 27 previous similar messages [Wed Feb 6 03:33:34 2019][1674513.862990] LustreError: 15231:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff912414416800 ns: mdt-fir-MDT0000_UUID lock: ffff9137f307c380/0x21ed9351858042ec lrc: 1/0,0 mode: EX/EX res: [0x200001832:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d73d2 expref: 4 pid: 15231 timeout: 0 lvb_type: 3 [Wed Feb 6 03:33:34 2019][1674513.897170] LustreError: 15231:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 6 previous similar messages [Wed Feb 6 03:33:34 2019][1674514.253769] LNet: Service thread pid 15181 was inactive for 302.38s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 03:33:34 2019][1674514.270881] Pid: 15181, comm: mdt02_029 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:33:34 2019][1674514.280795] Call Trace: [Wed Feb 6 03:33:34 2019][1674514.283428] [] 0xffffffffffffffff [Wed Feb 6 03:33:34 2019][1674514.288626] LustreError: dumping log to /tmp/lustre-log.1549452852.15181 [Wed Feb 6 03:33:36 2019][1674515.449373] Pid: 9027, comm: mdt00_020 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:33:36 2019][1674515.459202] Call Trace: [Wed Feb 6 03:33:36 2019][1674515.461841] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:33:36 2019][1674515.468971] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:33:36 2019][1674515.476258] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:33:36 2019][1674515.483114] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:33:36 2019][1674515.489846] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:33:36 2019][1674515.496254] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:33:36 2019][1674515.502550] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:33:36 2019][1674515.509991] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:33:36 2019][1674515.516898] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:33:36 2019][1674515.525112] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:33:36 2019][1674515.531509] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:33:36 2019][1674515.537734] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:33:36 2019][1674515.544465] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:33:36 2019][1674515.550344] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:33:36 2019][1674515.557470] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:33:36 2019][1674515.565375] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:33:36 2019][1674515.571890] [] kthread+0xd1/0xe0 [Wed Feb 6 03:33:36 2019][1674515.576988] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:33:36 2019][1674515.583643] [] 0xffffffffffffffff [Wed Feb 6 03:33:36 2019][1674516.301824] LNet: Service thread pid 7500 was inactive for 302.73s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 03:33:36 2019][1674516.318847] LNet: Skipped 1 previous similar message [Wed Feb 6 03:33:36 2019][1674516.323995] Pid: 7500, comm: mdt00_011 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:33:36 2019][1674516.333821] Call Trace: [Wed Feb 6 03:33:36 2019][1674516.336459] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:33:36 2019][1674516.343587] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:33:36 2019][1674516.350868] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:33:36 2019][1674516.357714] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:33:36 2019][1674516.364450] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:33:37 2019][1674516.370836] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:33:37 2019][1674516.377143] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:33:37 2019][1674516.384584] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:33:37 2019][1674516.391498] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:33:37 2019][1674516.399713] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:33:37 2019][1674516.406108] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:33:37 2019][1674516.412333] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:33:37 2019][1674516.419078] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:33:37 2019][1674516.424952] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:33:37 2019][1674516.432075] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:33:37 2019][1674516.439964] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:33:37 2019][1674516.446464] [] kthread+0xd1/0xe0 [Wed Feb 6 03:33:37 2019][1674516.451549] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:33:37 2019][1674516.458197] [] 0xffffffffffffffff [Wed Feb 6 03:33:37 2019][1674516.463391] LustreError: dumping log to /tmp/lustre-log.1549452854.7500 [Wed Feb 6 03:33:46 2019][1674525.518050] LNet: Service thread pid 15293 was inactive for 302.83s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 03:33:46 2019][1674525.535159] Pid: 15293, comm: mdt01_066 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:33:46 2019][1674525.545072] Call Trace: [Wed Feb 6 03:33:46 2019][1674525.547715] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:33:46 2019][1674525.554826] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:33:46 2019][1674525.562106] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:33:46 2019][1674525.568953] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:33:46 2019][1674525.575695] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:33:46 2019][1674525.582092] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:33:46 2019][1674525.588401] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:33:46 2019][1674525.595838] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:33:46 2019][1674525.602756] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:33:46 2019][1674525.610971] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:33:46 2019][1674525.617365] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:33:46 2019][1674525.623588] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:33:46 2019][1674525.630334] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:33:46 2019][1674525.636209] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:33:46 2019][1674525.643325] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:33:46 2019][1674525.651213] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:33:46 2019][1674525.657712] [] kthread+0xd1/0xe0 [Wed Feb 6 03:33:46 2019][1674525.662798] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:33:46 2019][1674525.669446] [] 0xffffffffffffffff [Wed Feb 6 03:33:46 2019][1674525.674640] LustreError: dumping log to /tmp/lustre-log.1549452864.15293 [Wed Feb 6 03:34:02 2019][1674541.902463] LNet: Service thread pid 15184 was inactive for 302.65s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 03:34:02 2019][1674541.919576] Pid: 15184, comm: mdt02_032 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:34:02 2019][1674541.929491] Call Trace: [Wed Feb 6 03:34:02 2019][1674541.932134] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:34:02 2019][1674541.939253] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:34:02 2019][1674541.946552] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:34:02 2019][1674541.953413] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:34:02 2019][1674541.960159] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:34:02 2019][1674541.966555] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:34:02 2019][1674541.972879] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:34:02 2019][1674541.980325] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:34:02 2019][1674541.987243] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:34:02 2019][1674541.995474] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:34:02 2019][1674542.001874] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:34:02 2019][1674542.008118] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:34:02 2019][1674542.014866] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:34:02 2019][1674542.020761] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:34:02 2019][1674542.027891] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:34:02 2019][1674542.035809] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:34:02 2019][1674542.042311] [] kthread+0xd1/0xe0 [Wed Feb 6 03:34:02 2019][1674542.047399] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:34:02 2019][1674542.054066] [] 0xffffffffffffffff [Wed Feb 6 03:34:02 2019][1674542.059282] LustreError: dumping log to /tmp/lustre-log.1549452880.15184 [Wed Feb 6 03:34:05 2019][1674544.974543] LNet: Service thread pid 8984 was inactive for 302.81s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 03:34:05 2019][1674544.987486] LNet: Skipped 10 previous similar messages [Wed Feb 6 03:34:05 2019][1674544.992808] LustreError: dumping log to /tmp/lustre-log.1549452883.8984 [Wed Feb 6 03:34:08 2019][1674548.046617] LustreError: dumping log to /tmp/lustre-log.1549452886.6758 [Wed Feb 6 03:34:12 2019][1674552.142717] LustreError: dumping log to /tmp/lustre-log.1549452890.6952 [Wed Feb 6 03:34:13 2019][1674553.166748] LustreError: dumping log to /tmp/lustre-log.1549452891.15372 [Wed Feb 6 03:34:14 2019][1674554.190771] LustreError: dumping log to /tmp/lustre-log.1549452892.15220 [Wed Feb 6 03:34:18 2019][1674558.286873] LustreError: dumping log to /tmp/lustre-log.1549452896.9182 [Wed Feb 6 03:34:27 2019][1674566.480080] LNet: Service thread pid 8968 was inactive for 302.80s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 03:34:27 2019][1674566.493030] LNet: Skipped 5 previous similar messages [Wed Feb 6 03:34:27 2019][1674566.498265] LustreError: dumping log to /tmp/lustre-log.1549452904.8968 [Wed Feb 6 03:34:38 2019][1674577.743359] LustreError: dumping log to /tmp/lustre-log.1549452916.6806 [Wed Feb 6 03:34:45 2019][1674584.911541] LustreError: dumping log to /tmp/lustre-log.1549452923.9016 [Wed Feb 6 03:34:48 2019][1674587.983618] LustreError: dumping log to /tmp/lustre-log.1549452926.15248 [Wed Feb 6 03:34:54 2019][1674594.127773] LustreError: dumping log to /tmp/lustre-log.1549452932.8946 [Wed Feb 6 03:35:01 2019][1674601.295958] LustreError: dumping log to /tmp/lustre-log.1549452939.48636 [Wed Feb 6 03:35:24 2019][1674623.824522] LNet: Service thread pid 8969 was inactive for 302.52s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 03:35:24 2019][1674623.837473] LNet: Skipped 5 previous similar messages [Wed Feb 6 03:35:24 2019][1674623.842708] LustreError: dumping log to /tmp/lustre-log.1549452962.8969 [Wed Feb 6 03:35:42 2019][1674642.256981] LustreError: dumping log to /tmp/lustre-log.1549452980.9199 [Wed Feb 6 03:35:49 2019][1674648.401134] LustreError: dumping log to /tmp/lustre-log.1549452986.17204 [Wed Feb 6 03:35:50 2019][1674649.425164] LustreError: dumping log to /tmp/lustre-log.1549452987.15393 [Wed Feb 6 03:35:51 2019][1674650.449188] LustreError: dumping log to /tmp/lustre-log.1549452988.6813 [Wed Feb 6 03:36:01 2019][1674660.689444] LustreError: dumping log to /tmp/lustre-log.1549452999.8971 [Wed Feb 6 03:36:05 2019][1674664.785546] LustreError: dumping log to /tmp/lustre-log.1549453003.9006 [Wed Feb 6 03:36:06 2019][1674665.809572] LustreError: dumping log to /tmp/lustre-log.1549453004.15258 [Wed Feb 6 03:36:07 2019][1674666.833597] LustreError: dumping log to /tmp/lustre-log.1549453005.15218 [Wed Feb 6 03:36:08 2019][1674667.857627] LustreError: dumping log to /tmp/lustre-log.1549453006.6173 [Wed Feb 6 03:36:11 2019][1674670.929698] LustreError: dumping log to /tmp/lustre-log.1549453009.8973 [Wed Feb 6 03:36:13 2019][1674672.977752] LustreError: dumping log to /tmp/lustre-log.1549453011.15389 [Wed Feb 6 03:36:16 2019][1674676.049829] LustreError: dumping log to /tmp/lustre-log.1549453014.8985 [Wed Feb 6 03:36:19 2019][1674679.121910] LustreError: dumping log to /tmp/lustre-log.1549453017.15239 [Wed Feb 6 03:36:21 2019][1674681.169958] LustreError: dumping log to /tmp/lustre-log.1549453019.53421 [Wed Feb 6 03:36:30 2019][1674689.362161] LustreError: dumping log to /tmp/lustre-log.1549453027.15261 [Wed Feb 6 03:36:36 2019][1674695.506316] LustreError: dumping log to /tmp/lustre-log.1549453033.15242 [Wed Feb 6 03:38:12 2019][1674791.518731] LustreError: 15509:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549452829, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff91452ce1de80/0x21ed935185c0eeef lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 145 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 15509 timeout: 0 lvb_type: 0 [Wed Feb 6 03:38:12 2019][1674791.558201] LustreError: 15509:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 6 previous similar messages [Wed Feb 6 03:38:13 2019][1674793.029759] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 03:38:13 2019][1674793.040453] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 381619699 previous similar messages [Wed Feb 6 03:38:21 2019][1674801.051958] Lustre: 9014:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549453139/real 1549453139] req@ffff912090798300 x1623682056556784/t0(0) o104->fir-MDT0000@10.10.29.8@tcp:15/16 lens 296/224 e 0 to 1 dl 1549453150 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 03:38:21 2019][1674801.079471] Lustre: 9014:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 151765800 previous similar messages [Wed Feb 6 03:38:27 2019][1674806.997128] Lustre: 15510:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 03:38:27 2019][1674806.997128] req@ffff91450aab0600 x1624697977400448/t13600424486(0) o36->d78401ad-d7f9-70f0-e382-808abf23c9bd@10.9.103.10@o2ib4:625/0 lens 528/2888 e 17 to 0 dl 1549453150 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 03:38:27 2019][1674807.027236] Lustre: 15510:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Feb 6 03:38:34 2019][1674813.927781] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 03:38:34 2019][1674813.938388] Lustre: Skipped 16 previous similar messages [Wed Feb 6 03:38:36 2019][1674815.870348] LustreError: 9014:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.29.8@tcp) failed to reply to blocking AST (req@ffff912090798300 x1623682056556784 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff91330d28f980/0x21ed9351864bd513 lrc: 4/0,0 mode: CR/CR res: [0x200001833:0x1:0x0].0x0 bits 0x9/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d7529 expref: 25 pid: 9014 timeout: 1674922 lvb_type: 0 [Wed Feb 6 03:38:36 2019][1674815.912424] LustreError: 9014:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages [Wed Feb 6 03:38:36 2019][1674815.922600] LustreError: 138-a: fir-MDT0000: A client on nid 10.10.29.8@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 03:38:36 2019][1674815.935129] LustreError: Skipped 4 previous similar messages [Wed Feb 6 03:38:36 2019][1674815.940992] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.29.8@tcp ns: mdt-fir-MDT0000_UUID lock: ffff91330d28f980/0x21ed9351864bd513 lrc: 3/0,0 mode: CR/CR res: [0x200001833:0x1:0x0].0x0 bits 0x9/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d7529 expref: 26 pid: 9014 timeout: 0 lvb_type: 0 [Wed Feb 6 03:38:36 2019][1674815.977614] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 4 previous similar messages [Wed Feb 6 03:38:38 2019][1674818.005404] Lustre: 9014:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 03:38:38 2019][1674818.005404] req@ffff9135a450f500 x1624671643885360/t0(0) o36->88ddc3f9-1f21-4f81-e1f1-3f396b007308@10.9.101.54@o2ib4:636/0 lens 544/2888 e 17 to 0 dl 1549453161 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 03:38:38 2019][1674818.034562] Lustre: 9014:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [Wed Feb 6 03:38:57 2019][1674837.053878] Lustre: 15265:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 03:38:57 2019][1674837.053878] req@ffff91432e3dc800 x1624712305624448/t0(0) o36->3e106753-f830-b044-4cf1-87da4380b4a5@10.8.6.12@o2ib6:655/0 lens 544/2888 e 9 to 0 dl 1549453180 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 03:38:57 2019][1674837.082864] Lustre: 15265:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Feb 6 03:39:02 2019][1674841.941995] LNet: Service thread pid 6687 was inactive for 436.72s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 03:39:02 2019][1674841.959017] Pid: 6687, comm: mdt02_004 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:39:02 2019][1674841.968847] Call Trace: [Wed Feb 6 03:39:02 2019][1674841.971510] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:39:02 2019][1674841.978632] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:39:02 2019][1674841.985940] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:39:02 2019][1674841.992789] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:39:02 2019][1674841.999546] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:39:02 2019][1674842.005951] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:39:02 2019][1674842.012282] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:39:02 2019][1674842.019722] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:39:02 2019][1674842.026648] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:39:02 2019][1674842.034873] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:39:02 2019][1674842.041264] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:39:02 2019][1674842.047496] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:39:02 2019][1674842.054233] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:39:02 2019][1674842.060118] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:39:02 2019][1674842.067252] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:39:02 2019][1674842.075141] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:39:02 2019][1674842.081661] [] kthread+0xd1/0xe0 [Wed Feb 6 03:39:02 2019][1674842.086750] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:39:02 2019][1674842.093418] [] 0xffffffffffffffff [Wed Feb 6 03:39:02 2019][1674842.098619] LustreError: dumping log to /tmp/lustre-log.1549453180.6687 [Wed Feb 6 03:39:04 2019][1674843.456132] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 03:39:04 2019][1674843.467169] Lustre: Skipped 2 previous similar messages [Wed Feb 6 03:39:07 2019][1674847.062120] Pid: 15406, comm: mdt00_047 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:39:07 2019][1674847.072033] Call Trace: [Wed Feb 6 03:39:07 2019][1674847.074679] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:39:07 2019][1674847.081804] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:39:07 2019][1674847.089110] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:39:07 2019][1674847.095960] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:39:07 2019][1674847.102713] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:39:07 2019][1674847.109116] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:39:07 2019][1674847.115432] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:39:07 2019][1674847.122871] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:39:07 2019][1674847.129792] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:39:07 2019][1674847.138010] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:39:07 2019][1674847.144422] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:39:07 2019][1674847.150654] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:39:07 2019][1674847.157412] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:39:07 2019][1674847.163283] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:39:07 2019][1674847.170412] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:39:07 2019][1674847.178305] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:39:07 2019][1674847.184818] [] kthread+0xd1/0xe0 [Wed Feb 6 03:39:07 2019][1674847.189908] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:39:07 2019][1674847.196569] [] 0xffffffffffffffff [Wed Feb 6 03:39:07 2019][1674847.201766] LustreError: dumping log to /tmp/lustre-log.1549453185.15406 [Wed Feb 6 03:39:08 2019][1674847.871141] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 03:39:15 2019][1674855.254328] Pid: 15188, comm: mdt01_045 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:39:15 2019][1674855.264241] Call Trace: [Wed Feb 6 03:39:15 2019][1674855.266884] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:39:15 2019][1674855.274003] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:39:15 2019][1674855.281281] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:39:15 2019][1674855.288130] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:39:15 2019][1674855.294872] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:39:15 2019][1674855.301267] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:39:15 2019][1674855.307578] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:39:15 2019][1674855.315015] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:39:15 2019][1674855.321930] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:39:15 2019][1674855.330147] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:39:15 2019][1674855.336542] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:39:15 2019][1674855.342766] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:39:15 2019][1674855.349508] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:39:15 2019][1674855.355384] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:39:15 2019][1674855.362502] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:39:15 2019][1674855.370388] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:39:15 2019][1674855.376887] [] kthread+0xd1/0xe0 [Wed Feb 6 03:39:15 2019][1674855.381974] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:39:16 2019][1674855.388621] [] 0xffffffffffffffff [Wed Feb 6 03:39:16 2019][1674855.393815] LustreError: dumping log to /tmp/lustre-log.1549453193.15188 [Wed Feb 6 03:39:17 2019][1674857.302379] Pid: 7029, comm: mdt02_010 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:39:17 2019][1674857.312209] Call Trace: [Wed Feb 6 03:39:17 2019][1674857.314857] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:39:17 2019][1674857.321983] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:39:17 2019][1674857.329271] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:39:17 2019][1674857.336120] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:39:17 2019][1674857.342870] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:39:17 2019][1674857.349275] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:39:17 2019][1674857.355593] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:39:17 2019][1674857.363030] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:39:17 2019][1674857.369954] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:39:17 2019][1674857.378170] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:39:17 2019][1674857.384566] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:39:17 2019][1674857.390789] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:39:18 2019][1674857.397531] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:39:18 2019][1674857.403407] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:39:18 2019][1674857.410532] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:39:18 2019][1674857.418428] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:39:18 2019][1674857.424936] [] kthread+0xd1/0xe0 [Wed Feb 6 03:39:18 2019][1674857.430026] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:39:18 2019][1674857.436681] [] 0xffffffffffffffff [Wed Feb 6 03:39:18 2019][1674857.441882] LustreError: dumping log to /tmp/lustre-log.1549453195.7029 [Wed Feb 6 03:39:38 2019][1674878.166912] Lustre: 9015:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 03:39:38 2019][1674878.166912] req@ffff9139b1f36c00 x1624705155767072/t0(0) o36->173446cc-39b1-333f-81fc-6684fb678e20@10.8.3.19@o2ib6:696/0 lens 544/2888 e 6 to 0 dl 1549453221 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 03:39:38 2019][1674878.195808] Lustre: 9015:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 7 previous similar messages [Wed Feb 6 03:39:46 2019][1674885.975094] LNet: Service thread pid 24899 was inactive for 436.89s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 03:39:46 2019][1674885.992201] LNet: Skipped 3 previous similar messages [Wed Feb 6 03:39:46 2019][1674885.997435] Pid: 24899, comm: mdt03_032 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:39:46 2019][1674886.007347] Call Trace: [Wed Feb 6 03:39:46 2019][1674886.009989] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:39:46 2019][1674886.017104] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:39:46 2019][1674886.024383] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:39:46 2019][1674886.031231] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:39:46 2019][1674886.037974] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:39:46 2019][1674886.044367] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:39:46 2019][1674886.050679] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:39:46 2019][1674886.058115] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:39:46 2019][1674886.065030] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:39:46 2019][1674886.073247] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:39:46 2019][1674886.079645] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:39:46 2019][1674886.085868] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:39:46 2019][1674886.092608] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:39:46 2019][1674886.098477] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:39:46 2019][1674886.105601] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:39:46 2019][1674886.113487] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:39:46 2019][1674886.119995] [] kthread+0xd1/0xe0 [Wed Feb 6 03:39:46 2019][1674886.125085] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:39:46 2019][1674886.131731] [] 0xffffffffffffffff [Wed Feb 6 03:39:46 2019][1674886.136925] LustreError: dumping log to /tmp/lustre-log.1549453224.24899 [Wed Feb 6 03:40:27 2019][1674926.936123] LNet: Service thread pid 32929 was inactive for 487.40s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 03:40:27 2019][1674926.949164] LNet: Skipped 22 previous similar messages [Wed Feb 6 03:40:27 2019][1674926.954486] LustreError: dumping log to /tmp/lustre-log.1549453265.32929 [Wed Feb 6 03:40:31 2019][1674931.032228] LustreError: dumping log to /tmp/lustre-log.1549453269.8998 [Wed Feb 6 03:40:33 2019][1674933.071297] LustreError: 8981:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549452971, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff913dacccf980/0x21ed93518636b809 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 151 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 8981 timeout: 0 lvb_type: 0 [Wed Feb 6 03:40:33 2019][1674933.080282] LustreError: dumping log to /tmp/lustre-log.1549453271.15230 [Wed Feb 6 03:40:33 2019][1674933.117466] LustreError: 8981:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Feb 6 03:40:53 2019][1674953.288796] Lustre: 48673:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 03:40:53 2019][1674953.288796] req@ffff9139edf01450 x1624673940548032/t0(0) o36->ea827f53-1edf-c98c-6637-0c731c4e9044@10.9.101.50@o2ib4:16/0 lens 544/2888 e 3 to 0 dl 1549453296 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 03:40:53 2019][1674953.317865] Lustre: 48673:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 9 previous similar messages [Wed Feb 6 03:41:11 2019][1674970.876240] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 03:41:11 2019][1674970.886328] Lustre: Skipped 33 previous similar messages [Wed Feb 6 03:41:29 2019][1674988.377669] LustreError: dumping log to /tmp/lustre-log.1549453326.6617 [Wed Feb 6 03:42:50 2019][1675070.300726] LustreError: dumping log to /tmp/lustre-log.1549453408.15329 [Wed Feb 6 03:42:59 2019][1675078.491927] LNet: Service thread pid 15509 was inactive for 586.96s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 03:42:59 2019][1675078.504960] LNet: Skipped 5 previous similar messages [Wed Feb 6 03:42:59 2019][1675078.510190] LustreError: dumping log to /tmp/lustre-log.1549453416.15509 [Wed Feb 6 03:44:03 2019][1675143.133562] Lustre: 15277:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 03:44:03 2019][1675143.133562] req@ffff9140fe650600 x1624711757611312/t0(0) o36->56748acc-0114-c8d8-7a9c-c6362a34cc0f@10.8.4.3@o2ib6:206/0 lens 544/2888 e 1 to 0 dl 1549453486 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 03:44:03 2019][1675143.162480] Lustre: 15277:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 30 previous similar messages [Wed Feb 6 03:44:04 2019][1675144.029569] LNet: Service thread pid 15280 was inactive for 636.80s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 03:44:04 2019][1675144.046677] Pid: 15280, comm: mdt00_043 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:44:04 2019][1675144.056593] Call Trace: [Wed Feb 6 03:44:04 2019][1675144.059231] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:44:04 2019][1675144.066348] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 03:44:04 2019][1675144.073713] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 03:44:04 2019][1675144.081929] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:44:04 2019][1675144.088318] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:44:04 2019][1675144.094530] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:44:04 2019][1675144.101265] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:44:04 2019][1675144.107150] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:44:04 2019][1675144.114275] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:44:04 2019][1675144.122161] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:44:04 2019][1675144.128662] [] kthread+0xd1/0xe0 [Wed Feb 6 03:44:04 2019][1675144.133749] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:44:04 2019][1675144.140400] [] 0xffffffffffffffff [Wed Feb 6 03:44:04 2019][1675144.145616] LustreError: dumping log to /tmp/lustre-log.1549453482.15280 [Wed Feb 6 03:44:22 2019][1675161.975616] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 03:44:22 2019][1675161.986658] Lustre: Skipped 1 previous similar message [Wed Feb 6 03:45:34 2019][1675234.143832] Pid: 6167, comm: mdt00_000 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:45:34 2019][1675234.153662] Call Trace: [Wed Feb 6 03:45:34 2019][1675234.156307] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:45:34 2019][1675234.163435] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:45:34 2019][1675234.170706] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:45:34 2019][1675234.177544] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:45:34 2019][1675234.184276] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:45:34 2019][1675234.190664] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:45:34 2019][1675234.196964] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:45:34 2019][1675234.204402] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:45:34 2019][1675234.211310] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:45:34 2019][1675234.219526] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:45:34 2019][1675234.225912] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:45:34 2019][1675234.232128] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:45:34 2019][1675234.238862] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:45:34 2019][1675234.244730] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:45:34 2019][1675234.251854] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:45:34 2019][1675234.259759] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:45:34 2019][1675234.266259] [] kthread+0xd1/0xe0 [Wed Feb 6 03:45:34 2019][1675234.271345] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:45:34 2019][1675234.277991] [] 0xffffffffffffffff [Wed Feb 6 03:45:34 2019][1675234.283187] LustreError: dumping log to /tmp/lustre-log.1549453572.6167 [Wed Feb 6 03:45:36 2019][1675236.191895] Pid: 15409, comm: mdt00_050 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:45:36 2019][1675236.201813] Call Trace: [Wed Feb 6 03:45:36 2019][1675236.204462] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:45:36 2019][1675236.211596] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:45:36 2019][1675236.218884] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:45:36 2019][1675236.225739] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:45:36 2019][1675236.232484] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:45:36 2019][1675236.238877] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:45:36 2019][1675236.245179] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:45:36 2019][1675236.252616] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:45:36 2019][1675236.259530] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:45:36 2019][1675236.267775] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:45:36 2019][1675236.274170] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:45:36 2019][1675236.280392] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:45:36 2019][1675236.287136] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:45:36 2019][1675236.293013] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:45:36 2019][1675236.300153] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:45:36 2019][1675236.308059] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:45:36 2019][1675236.314568] [] kthread+0xd1/0xe0 [Wed Feb 6 03:45:36 2019][1675236.319654] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:45:36 2019][1675236.326301] [] 0xffffffffffffffff [Wed Feb 6 03:45:36 2019][1675236.331504] LustreError: dumping log to /tmp/lustre-log.1549453574.15409 [Wed Feb 6 03:45:38 2019][1675238.239938] Pid: 15260, comm: mdt01_061 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:45:38 2019][1675238.249851] Call Trace: [Wed Feb 6 03:45:38 2019][1675238.252497] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:45:38 2019][1675238.259605] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:45:38 2019][1675238.266886] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:45:38 2019][1675238.273734] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:45:38 2019][1675238.280476] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:45:38 2019][1675238.286873] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:45:38 2019][1675238.293182] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:45:38 2019][1675238.300618] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:45:38 2019][1675238.307525] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:45:38 2019][1675238.315741] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:45:38 2019][1675238.322139] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:45:38 2019][1675238.328362] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:45:38 2019][1675238.335095] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:45:38 2019][1675238.340962] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:45:38 2019][1675238.348078] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:45:38 2019][1675238.355965] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:45:38 2019][1675238.362465] [] kthread+0xd1/0xe0 [Wed Feb 6 03:45:38 2019][1675238.367553] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:45:39 2019][1675238.374199] [] 0xffffffffffffffff [Wed Feb 6 03:45:39 2019][1675238.379393] LustreError: dumping log to /tmp/lustre-log.1549453576.15260 [Wed Feb 6 03:46:56 2019][1675316.065896] LNet: Service thread pid 15392 was inactive for 737.22s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 03:46:56 2019][1675316.083004] LNet: Skipped 3 previous similar messages [Wed Feb 6 03:46:56 2019][1675316.088245] Pid: 15392, comm: mdt01_075 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:46:56 2019][1675316.098163] Call Trace: [Wed Feb 6 03:46:56 2019][1675316.100808] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:46:56 2019][1675316.107934] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:46:56 2019][1675316.115225] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:46:56 2019][1675316.122078] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:46:56 2019][1675316.128823] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:46:56 2019][1675316.135227] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:46:56 2019][1675316.141535] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:46:56 2019][1675316.148980] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:46:56 2019][1675316.155897] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:46:56 2019][1675316.164122] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:46:56 2019][1675316.170528] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:46:56 2019][1675316.176750] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:46:56 2019][1675316.183492] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:46:56 2019][1675316.189377] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:46:56 2019][1675316.196522] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:46:56 2019][1675316.204431] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:46:56 2019][1675316.210949] [] kthread+0xd1/0xe0 [Wed Feb 6 03:46:56 2019][1675316.216037] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:46:56 2019][1675316.222683] [] 0xffffffffffffffff [Wed Feb 6 03:46:56 2019][1675316.227886] LustreError: dumping log to /tmp/lustre-log.1549453654.15392 [Wed Feb 6 03:47:07 2019][1675327.237176] LustreError: 8983:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549453365, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff913d101eee40/0x21ed9351871f10d7 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 168 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 8983 timeout: 0 lvb_type: 0 [Wed Feb 6 03:47:07 2019][1675327.276469] LustreError: 8983:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages [Wed Feb 6 03:47:45 2019][1675365.219126] LustreError: dumping log to /tmp/lustre-log.1549453703.15386 [Wed Feb 6 03:47:56 2019][1675375.459383] LustreError: dumping log to /tmp/lustre-log.1549453713.6694 [Wed Feb 6 03:48:12 2019][1675391.843791] LNet: Service thread pid 6760 was inactive for 789.33s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 03:48:12 2019][1675391.856736] LNet: Skipped 4 previous similar messages [Wed Feb 6 03:48:12 2019][1675391.861973] LustreError: dumping log to /tmp/lustre-log.1549453730.6760 [Wed Feb 6 03:48:13 2019][1675393.044814] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 03:48:13 2019][1675393.055508] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 150595860 previous similar messages [Wed Feb 6 03:48:21 2019][1675401.067015] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549453739/real 1549453739] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549453750 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 03:48:21 2019][1675401.094699] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 151371792 previous similar messages [Wed Feb 6 03:48:35 2019][1675415.020707] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 03:48:35 2019][1675415.031319] Lustre: Skipped 76 previous similar messages [Wed Feb 6 03:48:40 2019][1675419.885501] LustreError: 15294:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.29.8@tcp) failed to reply to blocking AST (req@ffff912d35c1c200 x1623682056768048 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff913766288fc0/0x21ed935187762af5 lrc: 4/0,0 mode: CR/CR res: [0x200001835:0x1:0x0].0x0 bits 0x9/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d77f3 expref: 21 pid: 15294 timeout: 1675526 lvb_type: 0 [Wed Feb 6 03:48:40 2019][1675419.927760] LustreError: 15294:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages [Wed Feb 6 03:48:40 2019][1675419.938022] LustreError: 138-a: fir-MDT0000: A client on nid 10.10.29.8@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 03:48:40 2019][1675419.950532] LustreError: Skipped 3 previous similar messages [Wed Feb 6 03:48:40 2019][1675419.956398] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.29.8@tcp ns: mdt-fir-MDT0000_UUID lock: ffff913766288fc0/0x21ed935187762af5 lrc: 3/0,0 mode: CR/CR res: [0x200001835:0x1:0x0].0x0 bits 0x9/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d77f3 expref: 22 pid: 15294 timeout: 0 lvb_type: 0 [Wed Feb 6 03:48:40 2019][1675419.993096] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages [Wed Feb 6 03:48:40 2019][1675420.003200] LustreError: 15294:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff913d429bd400 ns: mdt-fir-MDT0000_UUID lock: ffff91376628a400/0x21ed935187762b03 lrc: 1/0,0 mode: EX/EX res: [0x200001835:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d77fa expref: 4 pid: 15294 timeout: 0 lvb_type: 3 [Wed Feb 6 03:49:11 2019][1675451.237284] Pid: 15511, comm: mdt02_049 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:49:11 2019][1675451.247203] Call Trace: [Wed Feb 6 03:49:11 2019][1675451.249846] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:49:11 2019][1675451.256957] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:49:11 2019][1675451.264238] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:49:11 2019][1675451.271086] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:49:11 2019][1675451.277827] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:49:11 2019][1675451.284223] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:49:11 2019][1675451.290534] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:49:11 2019][1675451.297970] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:49:11 2019][1675451.304887] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:49:11 2019][1675451.313105] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:49:11 2019][1675451.319507] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:49:11 2019][1675451.325729] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:49:11 2019][1675451.332464] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:49:11 2019][1675451.338331] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:49:11 2019][1675451.345447] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:49:11 2019][1675451.353333] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:49:11 2019][1675451.359834] [] kthread+0xd1/0xe0 [Wed Feb 6 03:49:11 2019][1675451.364923] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:49:12 2019][1675451.371568] [] 0xffffffffffffffff [Wed Feb 6 03:49:12 2019][1675451.376762] LustreError: dumping log to /tmp/lustre-log.1549453789.15511 [Wed Feb 6 03:49:22 2019][1675461.396536] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 03:49:22 2019][1675461.411661] LustreError: 15222:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549453499, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff9139d42dda00/0x21ed935187559ca3 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 64 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed935187559caa expref: -99 pid: 15222 timeout: 0 lvb_type: 0 [Wed Feb 6 03:49:22 2019][1675461.451909] LustreError: 15222:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 60 previous similar messages [Wed Feb 6 03:49:32 2019][1675471.717798] Pid: 17203, comm: mdt03_018 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:49:32 2019][1675471.727709] Call Trace: [Wed Feb 6 03:49:32 2019][1675471.730348] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:49:32 2019][1675471.737458] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 03:49:32 2019][1675471.744825] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 03:49:32 2019][1675471.753040] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:49:32 2019][1675471.759438] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:49:32 2019][1675471.765650] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:49:32 2019][1675471.772386] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:49:32 2019][1675471.778253] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:49:32 2019][1675471.785369] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:49:32 2019][1675471.793255] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:49:32 2019][1675471.799755] [] kthread+0xd1/0xe0 [Wed Feb 6 03:49:32 2019][1675471.804842] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:49:32 2019][1675471.811489] [] 0xffffffffffffffff [Wed Feb 6 03:49:32 2019][1675471.816683] LustreError: dumping log to /tmp/lustre-log.1549453810.17203 [Wed Feb 6 03:49:32 2019][1675471.908435] Pid: 15164, comm: mdt01_042 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:49:32 2019][1675471.918347] Call Trace: [Wed Feb 6 03:49:32 2019][1675471.920984] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:49:32 2019][1675471.928095] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 03:49:32 2019][1675471.935452] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 03:49:32 2019][1675471.943669] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:49:32 2019][1675471.950056] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:49:32 2019][1675471.956271] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:49:32 2019][1675471.963006] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:49:32 2019][1675471.968871] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:49:32 2019][1675471.975988] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:49:32 2019][1675471.983874] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:49:32 2019][1675471.990375] [] kthread+0xd1/0xe0 [Wed Feb 6 03:49:32 2019][1675471.995462] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:49:32 2019][1675472.002101] [] 0xffffffffffffffff [Wed Feb 6 03:49:57 2019][1675496.693002] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 03:49:57 2019][1675496.704040] Lustre: Skipped 2 previous similar messages [Wed Feb 6 03:50:21 2019][1675520.871030] Pid: 8981, comm: mdt02_013 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:50:21 2019][1675520.880857] Call Trace: [Wed Feb 6 03:50:21 2019][1675520.883494] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:50:21 2019][1675520.890606] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 03:50:21 2019][1675520.897975] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 03:50:21 2019][1675520.906189] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:50:21 2019][1675520.912587] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:50:21 2019][1675520.918810] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:50:21 2019][1675520.925545] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:50:21 2019][1675520.931420] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:50:21 2019][1675520.938544] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:50:21 2019][1675520.946430] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:50:21 2019][1675520.952930] [] kthread+0xd1/0xe0 [Wed Feb 6 03:50:21 2019][1675520.958019] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:50:21 2019][1675520.964665] [] 0xffffffffffffffff [Wed Feb 6 03:50:21 2019][1675520.969860] LustreError: dumping log to /tmp/lustre-log.1549453859.8981 [Wed Feb 6 03:50:37 2019][1675537.255442] Pid: 6171, comm: mdt01_001 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 03:50:37 2019][1675537.265266] Call Trace: [Wed Feb 6 03:50:37 2019][1675537.267905] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 03:50:37 2019][1675537.275013] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 03:50:37 2019][1675537.282293] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 03:50:37 2019][1675537.289140] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 03:50:37 2019][1675537.295875] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 03:50:37 2019][1675537.302272] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 03:50:37 2019][1675537.308582] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 03:50:37 2019][1675537.316018] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 03:50:37 2019][1675537.322925] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 03:50:37 2019][1675537.331140] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 03:50:37 2019][1675537.337536] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 03:50:37 2019][1675537.343759] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 03:50:37 2019][1675537.350503] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 03:50:37 2019][1675537.356377] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 03:50:37 2019][1675537.363492] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 03:50:37 2019][1675537.371381] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 03:50:37 2019][1675537.377881] [] kthread+0xd1/0xe0 [Wed Feb 6 03:50:37 2019][1675537.382971] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 03:50:37 2019][1675537.389616] [] 0xffffffffffffffff [Wed Feb 6 03:50:37 2019][1675537.394809] LustreError: dumping log to /tmp/lustre-log.1549453875.6171 [Wed Feb 6 03:51:12 2019][1675571.969274] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 03:51:12 2019][1675571.979365] Lustre: Skipped 67 previous similar messages [Wed Feb 6 03:54:33 2019][1675773.225359] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 03:54:37 2019][1675776.585452] Lustre: 15178:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 03:54:37 2019][1675776.585452] req@ffff913babe7e000 x1624701456212544/t0(0) o36->bdfd7f2b-f57d-9f82-b295-bd88659acc70@10.9.105.24@o2ib4:85/0 lens 528/2888 e 0 to 0 dl 1549454120 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 03:54:37 2019][1675776.614780] Lustre: 15178:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 12 previous similar messages [Wed Feb 6 03:55:35 2019][1675834.692908] LustreError: 7081:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549453873, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff912bc7adb600/0x21ed935187d15a0b lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 174 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 7081 timeout: 0 lvb_type: 0 [Wed Feb 6 03:55:35 2019][1675834.732209] LustreError: 7081:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 3 previous similar messages [Wed Feb 6 03:58:13 2019][1675993.059869] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 03:58:13 2019][1675993.070564] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 128963261 previous similar messages [Wed Feb 6 03:58:21 2019][1676001.082073] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549454339/real 1549454339] req@ffff913afee69e00 x1623682056381040/t0(0) o105->fir-MDT0002@10.10.113.6@tcp:15/16 lens 304/224 e 0 to 1 dl 1549454350 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 03:58:21 2019][1676001.109927] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 128706017 previous similar messages [Wed Feb 6 03:58:36 2019][1676016.113981] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 03:58:36 2019][1676016.124589] Lustre: Skipped 86 previous similar messages [Wed Feb 6 04:01:13 2019][1676173.062501] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 04:01:13 2019][1676173.072587] Lustre: Skipped 72 previous similar messages [Wed Feb 6 04:02:10 2019][1676229.496813] LNet: Service thread pid 8983 was inactive for 1202.23s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 04:02:10 2019][1676229.513982] LNet: Skipped 5 previous similar messages [Wed Feb 6 04:02:10 2019][1676229.519214] Pid: 8983, comm: mdt02_015 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:02:10 2019][1676229.529039] Call Trace: [Wed Feb 6 04:02:10 2019][1676229.531675] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:02:10 2019][1676229.538794] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 04:02:10 2019][1676229.546162] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 04:02:10 2019][1676229.554386] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:02:10 2019][1676229.560782] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:02:10 2019][1676229.567006] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:02:10 2019][1676229.573748] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:02:10 2019][1676229.579625] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:02:10 2019][1676229.586740] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:02:10 2019][1676229.594628] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:02:10 2019][1676229.601127] [] kthread+0xd1/0xe0 [Wed Feb 6 04:02:10 2019][1676229.606215] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:02:10 2019][1676229.612864] [] 0xffffffffffffffff [Wed Feb 6 04:02:10 2019][1676229.618063] LustreError: dumping log to /tmp/lustre-log.1549454568.8983 [Wed Feb 6 04:02:30 2019][1676249.977325] Pid: 10287, comm: mdt03_012 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:02:30 2019][1676249.987234] Call Trace: [Wed Feb 6 04:02:30 2019][1676249.989870] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:02:30 2019][1676249.996984] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 04:02:30 2019][1676250.004352] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 04:02:30 2019][1676250.012566] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:02:30 2019][1676250.018963] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:02:30 2019][1676250.025185] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:02:30 2019][1676250.031919] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:02:30 2019][1676250.037786] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:02:30 2019][1676250.044911] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:02:30 2019][1676250.052798] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:02:30 2019][1676250.059297] [] kthread+0xd1/0xe0 [Wed Feb 6 04:02:30 2019][1676250.064387] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:02:30 2019][1676250.071035] [] 0xffffffffffffffff [Wed Feb 6 04:02:30 2019][1676250.076227] LustreError: dumping log to /tmp/lustre-log.1549454588.10287 [Wed Feb 6 04:03:11 2019][1676290.938355] LNet: Service thread pid 9028 was inactive for 1202.63s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 04:03:11 2019][1676290.955461] LNet: Skipped 1 previous similar message [Wed Feb 6 04:03:11 2019][1676290.960610] Pid: 9028, comm: mdt00_021 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:03:11 2019][1676290.970454] Call Trace: [Wed Feb 6 04:03:11 2019][1676290.973095] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:03:11 2019][1676290.980210] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 04:03:11 2019][1676290.987583] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 04:03:11 2019][1676290.995802] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:03:11 2019][1676291.002210] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:03:11 2019][1676291.008428] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:03:11 2019][1676291.015176] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:03:11 2019][1676291.021047] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:03:11 2019][1676291.028186] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:03:11 2019][1676291.036077] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:03:11 2019][1676291.042589] [] kthread+0xd1/0xe0 [Wed Feb 6 04:03:11 2019][1676291.047682] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:03:11 2019][1676291.054329] [] 0xffffffffffffffff [Wed Feb 6 04:03:11 2019][1676291.059538] LustreError: dumping log to /tmp/lustre-log.1549454629.9028 [Wed Feb 6 04:03:44 2019][1676323.707176] Pid: 8829, comm: mdt02_011 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:03:44 2019][1676323.717001] Call Trace: [Wed Feb 6 04:03:44 2019][1676323.719648] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:03:44 2019][1676323.726786] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 04:03:44 2019][1676323.734183] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 04:03:44 2019][1676323.742402] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:03:44 2019][1676323.748797] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:03:44 2019][1676323.755020] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:03:44 2019][1676323.761777] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:03:44 2019][1676323.767647] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:03:44 2019][1676323.774787] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:03:44 2019][1676323.782677] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:03:44 2019][1676323.789210] [] kthread+0xd1/0xe0 [Wed Feb 6 04:03:44 2019][1676323.794299] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:03:44 2019][1676323.800960] [] 0xffffffffffffffff [Wed Feb 6 04:03:44 2019][1676323.806155] LustreError: dumping log to /tmp/lustre-log.1549454662.8829 [Wed Feb 6 04:04:25 2019][1676364.668205] LNet: Service thread pid 15222 was inactive for 1203.24s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 04:04:25 2019][1676364.685419] LNet: Skipped 1 previous similar message [Wed Feb 6 04:04:25 2019][1676364.690563] Pid: 15222, comm: mdt01_046 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:04:25 2019][1676364.700494] Call Trace: [Wed Feb 6 04:04:25 2019][1676364.703138] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:04:25 2019][1676364.710257] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:04:25 2019][1676364.717554] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:04:25 2019][1676364.724404] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:04:25 2019][1676364.731144] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:04:25 2019][1676364.737541] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:04:25 2019][1676364.743858] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:04:25 2019][1676364.751296] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:04:25 2019][1676364.758227] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:04:25 2019][1676364.766448] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:04:25 2019][1676364.772841] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:04:25 2019][1676364.779064] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:04:25 2019][1676364.785813] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:04:25 2019][1676364.791692] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:04:25 2019][1676364.798825] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:04:25 2019][1676364.806714] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:04:25 2019][1676364.813228] [] kthread+0xd1/0xe0 [Wed Feb 6 04:04:25 2019][1676364.818316] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:04:25 2019][1676364.824978] [] 0xffffffffffffffff [Wed Feb 6 04:04:25 2019][1676364.830175] LustreError: dumping log to /tmp/lustre-log.1549454703.15222 [Wed Feb 6 04:05:55 2019][1676455.277962] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 04:05:55 2019][1676455.289003] Lustre: Skipped 2 previous similar messages [Wed Feb 6 04:08:13 2019][1676593.074932] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 04:08:13 2019][1676593.085631] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 122858566 previous similar messages [Wed Feb 6 04:08:21 2019][1676601.097130] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549454939/real 1549454939] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549454950 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 04:08:21 2019][1676601.124828] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 122921018 previous similar messages [Wed Feb 6 04:08:37 2019][1676617.209094] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 04:08:37 2019][1676617.219700] Lustre: Skipped 72 previous similar messages [Wed Feb 6 04:09:36 2019][1676675.972012] LNet: Service thread pid 15232 was inactive for 1202.72s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 04:09:36 2019][1676675.989214] Pid: 15232, comm: mdt00_037 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:09:36 2019][1676675.999131] Call Trace: [Wed Feb 6 04:09:36 2019][1676676.001776] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:09:36 2019][1676676.008894] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:09:36 2019][1676676.016215] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:09:36 2019][1676676.023066] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:09:36 2019][1676676.029813] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:09:36 2019][1676676.036202] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:09:36 2019][1676676.042505] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:09:36 2019][1676676.049941] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:09:36 2019][1676676.056864] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:09:36 2019][1676676.065082] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:09:36 2019][1676676.071494] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:09:36 2019][1676676.077735] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:09:36 2019][1676676.084491] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:09:36 2019][1676676.090364] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:09:36 2019][1676676.097529] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:09:36 2019][1676676.105420] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:09:36 2019][1676676.111942] [] kthread+0xd1/0xe0 [Wed Feb 6 04:09:36 2019][1676676.117032] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:09:36 2019][1676676.123704] [] 0xffffffffffffffff [Wed Feb 6 04:09:36 2019][1676676.128908] LustreError: dumping log to /tmp/lustre-log.1549455014.15232 [Wed Feb 6 04:09:56 2019][1676695.940515] Pid: 24923, comm: mdt03_034 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:09:56 2019][1676695.950428] Call Trace: [Wed Feb 6 04:09:56 2019][1676695.953071] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 04:09:56 2019][1676695.960202] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 04:09:56 2019][1676695.967568] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 04:09:56 2019][1676695.975785] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:09:56 2019][1676695.982190] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:09:56 2019][1676695.988413] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:09:56 2019][1676695.995165] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:09:56 2019][1676696.001048] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:09:56 2019][1676696.008172] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:09:56 2019][1676696.016058] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:09:56 2019][1676696.022560] [] kthread+0xd1/0xe0 [Wed Feb 6 04:09:56 2019][1676696.027647] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:09:56 2019][1676696.034296] [] 0xffffffffffffffff [Wed Feb 6 04:09:56 2019][1676696.039515] LustreError: dumping log to /tmp/lustre-log.1549455034.24923 [Wed Feb 6 04:10:38 2019][1676737.413559] Pid: 7081, comm: mdt01_010 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:10:38 2019][1676737.423386] Call Trace: [Wed Feb 6 04:10:38 2019][1676737.426023] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:10:38 2019][1676737.433144] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 04:10:38 2019][1676737.440512] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 04:10:38 2019][1676737.448737] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:10:38 2019][1676737.455131] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:10:38 2019][1676737.461354] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:10:38 2019][1676737.468097] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:10:38 2019][1676737.473964] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:10:38 2019][1676737.481082] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:10:38 2019][1676737.488968] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:10:38 2019][1676737.495467] [] kthread+0xd1/0xe0 [Wed Feb 6 04:10:38 2019][1676737.500555] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:10:38 2019][1676737.507203] [] 0xffffffffffffffff [Wed Feb 6 04:10:38 2019][1676737.512396] LustreError: dumping log to /tmp/lustre-log.1549455075.7081 [Wed Feb 6 04:11:14 2019][1676774.155634] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 04:11:14 2019][1676774.165725] Lustre: Skipped 74 previous similar messages [Wed Feb 6 04:11:21 2019][1676780.576766] Lustre: fir-MDT0002: haven't heard from client c5acf83e-6f6d-4eb5-1fed-175307bafafe (at 10.8.10.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9119d119c000, cur 1549455119 expire 1549454969 last 1549454892 [Wed Feb 6 04:11:21 2019][1676780.598637] Lustre: Skipped 2 previous similar messages [Wed Feb 6 04:11:36 2019][1676795.547019] LustreError: 24923:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549454833, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9155a3ba0240/0x21ed935189bb5fbb lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 180 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 24923 timeout: 0 lvb_type: 0 [Wed Feb 6 04:18:13 2019][1677193.089990] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.6@tcp from [Wed Feb 6 04:18:13 2019][1677193.100688] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 122623279 previous similar messages [Wed Feb 6 04:18:21 2019][1677201.112184] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549455539/real 1549455539] req@ffff913afee69e00 x1623682056381040/t0(0) o105->fir-MDT0002@10.10.113.6@tcp:15/16 lens 304/224 e 0 to 1 dl 1549455550 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 04:18:21 2019][1677201.140041] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 122667088 previous similar messages [Wed Feb 6 04:18:38 2019][1677218.301910] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 04:18:38 2019][1677218.312519] Lustre: Skipped 74 previous similar messages [Wed Feb 6 04:19:05 2019][1677244.900296] Lustre: 6845:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 04:19:05 2019][1677244.900296] req@ffff9150f72a1200 x1624674960613424/t0(0) o36->6ff78f59-4d7f-d121-1724-154727e4138f@10.9.0.64@o2ib4:43/0 lens 544/2888 e 0 to 0 dl 1549455588 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 04:19:05 2019][1677244.929368] Lustre: 6845:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 6 previous similar messages [Wed Feb 6 04:21:15 2019][1677375.248350] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 04:21:15 2019][1677375.258440] Lustre: Skipped 72 previous similar messages [Wed Feb 6 04:28:13 2019][1677793.105040] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 04:28:13 2019][1677793.115739] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 122346972 previous similar messages [Wed Feb 6 04:28:21 2019][1677801.127244] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549456139/real 1549456139] req@ffff913afee69e00 x1623682056381040/t0(0) o105->fir-MDT0002@10.10.113.6@tcp:15/16 lens 304/224 e 0 to 1 dl 1549456150 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 04:28:21 2019][1677801.155100] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 122377539 previous similar messages [Wed Feb 6 04:28:39 2019][1677819.394265] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 04:28:39 2019][1677819.404875] Lustre: Skipped 81 previous similar messages [Wed Feb 6 04:31:16 2019][1677976.340686] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 04:31:16 2019][1677976.350774] Lustre: Skipped 71 previous similar messages [Wed Feb 6 04:32:36 2019][1678055.544634] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 04:32:36 2019][1678055.559766] LustreError: 47983:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549456093, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff913baa5c8fc0/0x21ed93518d421841 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 66 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93518d421848 expref: -99 pid: 47983 timeout: 0 lvb_type: 0 [Wed Feb 6 04:32:36 2019][1678055.600015] LustreError: 47983:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Feb 6 04:32:36 2019][1678055.611099] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Feb 6 04:37:30 2019][1678349.896030] Lustre: 17222:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 04:37:30 2019][1678349.896030] req@ffff915691b26600 x1624675946940832/t0(0) o36->ba1e1fbb-a72d-16b5-8ad9-93cddc13d3a5@10.9.0.63@o2ib4:393/0 lens 520/2888 e 1 to 0 dl 1549456693 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 04:38:13 2019][1678393.120098] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 04:38:13 2019][1678393.130800] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 121745957 previous similar messages [Wed Feb 6 04:38:21 2019][1678401.142303] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549456739/real 1549456739] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549456750 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 04:38:21 2019][1678401.169987] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 121785275 previous similar messages [Wed Feb 6 04:38:41 2019][1678420.486339] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 04:38:41 2019][1678420.496950] Lustre: Skipped 76 previous similar messages [Wed Feb 6 04:40:15 2019][1678514.918180] Lustre: fir-OST0001-osc-MDT0002: Connection to fir-OST0001 (at 10.0.10.102@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 04:40:36 2019][1678535.527667] NMI watchdog: BUG: soft lockup - CPU#43 stuck for 21s! [ldlm_cn03_008:23309] [Wed Feb 6 04:40:36 2019][1678535.535929] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 04:40:36 2019][1678535.642701] CPU: 43 PID: 23309 Comm: ldlm_cn03_008 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 04:40:36 2019][1678535.655464] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 04:40:36 2019][1678535.663203] task: ffff9159aa3da080 ti: ffff9158f562c000 task.ti: ffff9158f562c000 [Wed Feb 6 04:40:36 2019][1678535.670855] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 04:40:36 2019][1678535.680970] RSP: 0018:ffff9158f562fc38 EFLAGS: 00000246 [Wed Feb 6 04:40:36 2019][1678535.686458] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001590000 [Wed Feb 6 04:40:36 2019][1678535.693763] RDX: ffff9129fef5b780 RSI: 0000000000a10001 RDI: ffff913af6b0975c [Wed Feb 6 04:40:36 2019][1678535.701068] RBP: ffff9158f562fc38 R08: ffff915a3f69b780 R09: 0000000000000000 [Wed Feb 6 04:40:36 2019][1678535.708375] R10: ffff911b3fc07600 R11: ffffdbb2fc649400 R12: ffff9158f562fbe0 [Wed Feb 6 04:40:36 2019][1678535.715680] R13: ffff912248f6d980 R14: ffff9158f562fba0 R15: ffffffffc0ac8378 [Wed Feb 6 04:40:36 2019][1678535.722987] FS: 00007fccefbfb840(0000) GS:ffff915a3f680000(0000) knlGS:0000000000000000 [Wed Feb 6 04:40:36 2019][1678535.731246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 04:40:36 2019][1678535.737167] CR2: 000055fdd4177300 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 04:40:36 2019][1678535.744474] Call Trace: [Wed Feb 6 04:40:36 2019][1678535.747105] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 04:40:36 2019][1678535.753550] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 04:40:36 2019][1678535.759249] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 04:40:36 2019][1678535.765981] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 04:40:36 2019][1678535.772718] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 04:40:36 2019][1678535.779798] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 04:40:36 2019][1678535.786706] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 04:40:36 2019][1678535.793793] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:40:36 2019][1678535.801660] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 04:40:36 2019][1678535.808621] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 04:40:36 2019][1678535.815058] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 04:40:36 2019][1678535.821099] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:40:36 2019][1678535.827572] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 04:40:36 2019][1678535.835140] [] kthread+0xd1/0xe0 [Wed Feb 6 04:40:36 2019][1678535.840190] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 04:40:36 2019][1678535.846459] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:40:36 2019][1678535.853071] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 04:40:36 2019][1678535.859334] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 04:41:17 2019][1678577.432682] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 04:41:17 2019][1678577.442769] Lustre: Skipped 77 previous similar messages [Wed Feb 6 04:41:42 2019][1678601.965363] LustreError: 15170:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.29.8@tcp) failed to reply to blocking AST (req@ffff912d3d6a5700 x1623682057827040 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff9136104cf500/0x21ed935187da9c9c lrc: 4/0,0 mode: PR/PR res: [0x20000173d:0x4e4a:0x0].0x0 bits 0x13/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d7871 expref: 8 pid: 15170 timeout: 1678708 lvb_type: 0 [Wed Feb 6 04:41:42 2019][1678602.007870] LustreError: 15170:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message [Wed Feb 6 04:41:42 2019][1678602.018049] LustreError: 138-a: fir-MDT0000: A client on nid 10.10.29.8@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 04:41:42 2019][1678602.030557] LustreError: Skipped 1 previous similar message [Wed Feb 6 04:41:42 2019][1678602.036330] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 150s: evicting client at 10.10.29.8@tcp ns: mdt-fir-MDT0000_UUID lock: ffff9136104cf500/0x21ed935187da9c9c lrc: 3/0,0 mode: PR/PR res: [0x20000173d:0x4e4a:0x0].0x0 bits 0x13/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.10.29.8@tcp remote: 0x87752c3a8e2d7871 expref: 9 pid: 15170 timeout: 0 lvb_type: 0 [Wed Feb 6 04:41:42 2019][1678602.073320] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [Wed Feb 6 04:43:27 2019][1678706.614976] LNet: Service thread pid 8999 was inactive for 200.46s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 04:43:27 2019][1678706.632003] LNet: Skipped 2 previous similar messages [Wed Feb 6 04:43:27 2019][1678706.637239] Pid: 8999, comm: mdt00_019 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:43:27 2019][1678706.647063] Call Trace: [Wed Feb 6 04:43:27 2019][1678706.649691] [] 0xffffffffffffffff [Wed Feb 6 04:43:27 2019][1678706.654890] LustreError: dumping log to /tmp/lustre-log.1549457045.8999 [Wed Feb 6 04:43:30 2019][1678709.636078] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 04:45:15 2019][1678814.649687] LNet: Service thread pid 8929 was inactive for 212.44s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 04:45:15 2019][1678814.666714] Pid: 8929, comm: mdt00_012 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:45:15 2019][1678814.676542] Call Trace: [Wed Feb 6 04:45:15 2019][1678814.679184] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 04:45:15 2019][1678814.686314] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 04:45:15 2019][1678814.693689] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 04:45:15 2019][1678814.701904] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:45:15 2019][1678814.708302] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:45:15 2019][1678814.714523] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:45:15 2019][1678814.721268] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:45:15 2019][1678814.727144] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:45:15 2019][1678814.734286] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:45:15 2019][1678814.742190] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:45:15 2019][1678814.748705] [] kthread+0xd1/0xe0 [Wed Feb 6 04:45:15 2019][1678814.753792] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:45:15 2019][1678814.760441] [] 0xffffffffffffffff [Wed Feb 6 04:45:15 2019][1678814.765645] LustreError: dumping log to /tmp/lustre-log.1549457153.8929 [Wed Feb 6 04:46:42 2019][1678902.208888] LustreError: 8929:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549456940, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff911c0dd0a640/0x21ed93518fb32001 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 189 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 8929 timeout: 0 lvb_type: 0 [Wed Feb 6 04:47:38 2019][1678957.501269] LNet: Service thread pid 47983 was inactive for 1201.93s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 04:47:38 2019][1678957.518462] Pid: 47983, comm: mdt03_014 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:47:38 2019][1678957.528385] Call Trace: [Wed Feb 6 04:47:38 2019][1678957.531028] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:47:38 2019][1678957.538152] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 04:47:38 2019][1678957.545446] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 04:47:38 2019][1678957.552292] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 04:47:38 2019][1678957.559036] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 04:47:38 2019][1678957.565431] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 04:47:38 2019][1678957.571733] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 04:47:38 2019][1678957.579169] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 04:47:38 2019][1678957.586094] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 04:47:38 2019][1678957.594318] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:47:38 2019][1678957.600715] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:47:38 2019][1678957.606937] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:47:38 2019][1678957.613671] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:47:38 2019][1678957.619549] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:47:38 2019][1678957.626681] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:47:38 2019][1678957.634587] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:47:38 2019][1678957.641111] [] kthread+0xd1/0xe0 [Wed Feb 6 04:47:38 2019][1678957.646199] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:47:38 2019][1678957.652846] [] 0xffffffffffffffff [Wed Feb 6 04:47:38 2019][1678957.658040] LustreError: dumping log to /tmp/lustre-log.1549457296.47983 [Wed Feb 6 04:48:13 2019][1678993.135154] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 04:48:13 2019][1678993.145846] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 121774334 previous similar messages [Wed Feb 6 04:48:21 2019][1679001.157361] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549457339/real 1549457339] req@ffff913afee69e00 x1623682056381040/t0(0) o105->fir-MDT0002@10.10.113.6@tcp:15/16 lens 304/224 e 0 to 1 dl 1549457350 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 04:48:21 2019][1679001.185264] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 122059114 previous similar messages [Wed Feb 6 04:48:42 2019][1679021.578022] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 04:48:42 2019][1679021.588631] Lustre: Skipped 89 previous similar messages [Wed Feb 6 04:50:01 2019][1679101.208886] Lustre: 6613:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 04:50:01 2019][1679101.208886] req@ffff9122c49ec500 x1624699247657312/t13600535239(0) o36->090b1219-2d44-d095-6331-57a66bc20e45@10.9.105.65@o2ib4:389/0 lens 504/2888 e 24 to 0 dl 1549457444 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 04:51:19 2019][1679178.524129] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 04:51:19 2019][1679178.534219] Lustre: Skipped 75 previous similar messages [Wed Feb 6 04:51:38 2019][1679197.513301] Lustre: 9181:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 04:51:38 2019][1679197.513301] req@ffff9122227eb300 x1624705329706240/t0(0) o36->a650fb27-cfe4-742f-7959-475c51cb8a54@10.8.9.9@o2ib6:485/0 lens 528/2888 e 6 to 0 dl 1549457540 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 04:54:43 2019][1679382.742942] LustreError: 15187:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549457421, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff912c8376de80/0x21ed93519176054b lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 190 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 15187 timeout: 0 lvb_type: 0 [Wed Feb 6 04:54:56 2019][1679395.784268] LNet: Service thread pid 15187 was inactive for 313.04s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 04:54:56 2019][1679395.801374] Pid: 15187, comm: mdt01_044 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 04:54:56 2019][1679395.811287] Call Trace: [Wed Feb 6 04:54:56 2019][1679395.813928] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 04:54:56 2019][1679395.821034] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 04:54:56 2019][1679395.828402] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 04:54:56 2019][1679395.836618] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 04:54:56 2019][1679395.843012] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 04:54:56 2019][1679395.849227] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 04:54:56 2019][1679395.855962] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 04:54:56 2019][1679395.861830] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 04:54:56 2019][1679395.868945] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 04:54:56 2019][1679395.876832] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 04:54:56 2019][1679395.883340] [] kthread+0xd1/0xe0 [Wed Feb 6 04:54:56 2019][1679395.888428] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 04:54:56 2019][1679395.895076] [] 0xffffffffffffffff [Wed Feb 6 04:54:56 2019][1679395.900270] LustreError: dumping log to /tmp/lustre-log.1549457734.15187 [Wed Feb 6 04:58:13 2019][1679593.150209] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.6@tcp from [Wed Feb 6 04:58:13 2019][1679593.160908] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 119422949 previous similar messages [Wed Feb 6 04:58:21 2019][1679601.172413] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549457939/real 1549457939] req@ffff913afee69e00 x1623682056381040/t0(0) o105->fir-MDT0002@10.10.113.6@tcp:15/16 lens 304/224 e 0 to 1 dl 1549457950 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 04:58:21 2019][1679601.200277] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 119516286 previous similar messages [Wed Feb 6 04:58:43 2019][1679622.670063] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 04:58:43 2019][1679622.680668] Lustre: Skipped 74 previous similar messages [Wed Feb 6 04:59:38 2019][1679678.073359] Lustre: 9014:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 04:59:38 2019][1679678.073359] req@ffff912cdd0a2100 x1624705104714112/t0(0) o36->9114d135-37ef-ec55-8f47-397cb880c459@10.8.28.6@o2ib6:211/0 lens 528/2888 e 16 to 0 dl 1549458021 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 05:01:20 2019][1679779.616159] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 05:01:20 2019][1679779.626245] Lustre: Skipped 79 previous similar messages [Wed Feb 6 05:07:47 2019][1680166.677420] LNet: Service thread pid 8999 completed after 1660.49s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 05:07:47 2019][1680166.693753] LNet: Skipped 11 previous similar messages [Wed Feb 6 05:08:13 2019][1680193.165263] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.6@tcp from [Wed Feb 6 05:08:13 2019][1680193.175963] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 118014762 previous similar messages [Wed Feb 6 05:08:21 2019][1680201.187468] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549458539/real 1549458539] req@ffff913afee69e00 x1623682056381040/t0(0) o105->fir-MDT0002@10.10.113.6@tcp:15/16 lens 304/224 e 0 to 1 dl 1549458550 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 05:08:21 2019][1680201.215322] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 118107929 previous similar messages [Wed Feb 6 05:08:44 2019][1680223.763505] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 05:08:44 2019][1680223.774117] Lustre: Skipped 76 previous similar messages [Wed Feb 6 05:11:21 2019][1680380.709048] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 05:11:21 2019][1680380.719144] Lustre: Skipped 74 previous similar messages [Wed Feb 6 05:18:13 2019][1680793.180324] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.6@tcp from [Wed Feb 6 05:18:13 2019][1680793.191021] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 120136411 previous similar messages [Wed Feb 6 05:18:21 2019][1680801.202528] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549459139/real 1549459139] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549459150 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 05:18:21 2019][1680801.230221] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 120182653 previous similar messages [Wed Feb 6 05:18:45 2019][1680824.856571] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 05:18:45 2019][1680824.867183] Lustre: Skipped 83 previous similar messages [Wed Feb 6 05:21:22 2019][1680981.802217] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 05:21:22 2019][1680981.812305] Lustre: Skipped 75 previous similar messages [Wed Feb 6 05:28:13 2019][1681393.195388] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.6@tcp from [Wed Feb 6 05:28:13 2019][1681393.206085] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 120144679 previous similar messages [Wed Feb 6 05:28:21 2019][1681401.217585] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549459739/real 1549459739] req@ffff913afee69e00 x1623682056381040/t0(0) o105->fir-MDT0002@10.10.113.6@tcp:15/16 lens 304/224 e 0 to 1 dl 1549459750 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 05:28:21 2019][1681401.245442] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 120203222 previous similar messages [Wed Feb 6 05:28:46 2019][1681425.949851] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 05:28:46 2019][1681425.960463] Lustre: Skipped 77 previous similar messages [Wed Feb 6 05:31:23 2019][1681582.895438] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 05:31:23 2019][1681582.905527] Lustre: Skipped 77 previous similar messages [Wed Feb 6 05:35:35 2019][1681834.911924] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 05:38:13 2019][1681993.210439] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 05:38:13 2019][1681993.221134] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 120826376 previous similar messages [Wed Feb 6 05:38:21 2019][1682001.232642] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549460339/real 1549460339] req@ffff913afee69e00 x1623682056381040/t0(0) o105->fir-MDT0002@10.10.113.6@tcp:15/16 lens 304/224 e 0 to 1 dl 1549460350 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 05:38:21 2019][1682001.260602] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 120868796 previous similar messages [Wed Feb 6 05:38:47 2019][1682027.042911] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 05:38:47 2019][1682027.053523] Lustre: Skipped 85 previous similar messages [Wed Feb 6 05:41:24 2019][1682183.988377] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 05:41:24 2019][1682183.998468] Lustre: Skipped 77 previous similar messages [Wed Feb 6 05:45:51 2019][1682450.452919] LNet: Service thread pid 6951 was inactive for 200.48s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 05:45:51 2019][1682450.469941] Pid: 6951, comm: mdt03_009 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 05:45:51 2019][1682450.479768] Call Trace: [Wed Feb 6 05:45:51 2019][1682450.482412] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 05:45:51 2019][1682450.489532] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 05:45:51 2019][1682450.496830] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 05:45:51 2019][1682450.503677] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 05:45:51 2019][1682450.510419] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 05:45:51 2019][1682450.516818] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 05:45:51 2019][1682450.523126] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 05:45:51 2019][1682450.530578] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 05:45:51 2019][1682450.537496] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 05:45:51 2019][1682450.545711] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 05:45:51 2019][1682450.552122] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 05:45:51 2019][1682450.558348] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 05:45:51 2019][1682450.565090] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 05:45:51 2019][1682450.570968] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 05:45:51 2019][1682450.578093] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 05:45:51 2019][1682450.586031] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 05:45:51 2019][1682450.592529] [] kthread+0xd1/0xe0 [Wed Feb 6 05:45:51 2019][1682450.597618] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 05:45:51 2019][1682450.604265] [] 0xffffffffffffffff [Wed Feb 6 05:45:51 2019][1682450.609468] LustreError: dumping log to /tmp/lustre-log.1549460788.6951 [Wed Feb 6 05:47:30 2019][1682549.967417] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 05:47:30 2019][1682549.982527] Lustre: Skipped 3 previous similar messages [Wed Feb 6 05:47:30 2019][1682549.987950] LustreError: 6951:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549460588, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff9159956a0480/0x21ed93519d669985 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 67 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93519d66998c expref: -99 pid: 6951 timeout: 0 lvb_type: 0 [Wed Feb 6 05:47:30 2019][1682550.028417] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Feb 6 05:48:13 2019][1682593.225494] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.6@tcp from [Wed Feb 6 05:48:13 2019][1682593.236192] LNetError: 15181:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 121028108 previous similar messages [Wed Feb 6 05:48:21 2019][1682601.247701] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549460939/real 1549460939] req@ffff913afee69e00 x1623682056381040/t0(0) o105->fir-MDT0002@10.10.113.6@tcp:15/16 lens 304/224 e 0 to 1 dl 1549460950 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 05:48:21 2019][1682601.275559] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 121066069 previous similar messages [Wed Feb 6 05:48:48 2019][1682628.136054] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 05:48:48 2019][1682628.146657] Lustre: Skipped 79 previous similar messages [Wed Feb 6 05:51:25 2019][1682785.080989] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 05:51:25 2019][1682785.091076] Lustre: Skipped 78 previous similar messages [Wed Feb 6 05:52:25 2019][1682845.318834] Lustre: 17220:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 05:52:25 2019][1682845.318834] req@ffff91564f6ce000 x1624669502927232/t0(0) o36->657c4193-574d-7ac0-e927-0684806e81d2@10.9.101.42@o2ib4:358/0 lens 520/2888 e 24 to 0 dl 1549461188 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 05:58:13 2019][1683193.240556] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 05:58:13 2019][1683193.251255] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 120851247 previous similar messages [Wed Feb 6 05:58:21 2019][1683201.262750] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549461539/real 1549461539] req@ffff913afee69e00 x1623682056381040/t0(0) o105->fir-MDT0002@10.10.113.6@tcp:15/16 lens 304/224 e 0 to 1 dl 1549461550 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 05:58:21 2019][1683201.290607] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 120908039 previous similar messages [Wed Feb 6 05:58:49 2019][1683229.228865] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 05:58:49 2019][1683229.239472] Lustre: Skipped 76 previous similar messages [Wed Feb 6 05:59:32 2019][1683271.721528] LNet: Service thread pid 15219 was inactive for 200.58s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 05:59:32 2019][1683271.738638] Pid: 15219, comm: mdt00_033 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 05:59:32 2019][1683271.748553] Call Trace: [Wed Feb 6 05:59:32 2019][1683271.751196] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 05:59:32 2019][1683271.758333] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Feb 6 05:59:32 2019][1683271.765612] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Feb 6 05:59:32 2019][1683271.772460] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Feb 6 05:59:32 2019][1683271.779206] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Feb 6 05:59:32 2019][1683271.785610] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Feb 6 05:59:32 2019][1683271.791910] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Feb 6 05:59:32 2019][1683271.799345] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Feb 6 05:59:32 2019][1683271.806260] [] mdt_reint_rename_or_migrate.isra.51+0x362/0x860 [mdt] [Wed Feb 6 05:59:32 2019][1683271.814476] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 05:59:32 2019][1683271.820874] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 05:59:32 2019][1683271.827096] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 05:59:32 2019][1683271.833839] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 05:59:32 2019][1683271.839725] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 05:59:32 2019][1683271.846851] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 05:59:32 2019][1683271.854754] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 05:59:32 2019][1683271.861251] [] kthread+0xd1/0xe0 [Wed Feb 6 05:59:32 2019][1683271.866341] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 05:59:32 2019][1683271.873009] [] 0xffffffffffffffff [Wed Feb 6 05:59:32 2019][1683271.878217] LustreError: dumping log to /tmp/lustre-log.1549461610.15219 [Wed Feb 6 06:01:11 2019][1683371.139024] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 06:01:11 2019][1683371.154153] LustreError: 15219:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549461409, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff911f87659440/0x21ed93519f85d749 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 68 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed93519f85d757 expref: -99 pid: 15219 timeout: 0 lvb_type: 0 [Wed Feb 6 06:01:11 2019][1683371.194504] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Feb 6 06:01:20 2019][1683380.268250] LNet: Service thread pid 15231 was inactive for 200.32s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 06:01:20 2019][1683380.285361] Pid: 15231, comm: mdt01_051 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 06:01:20 2019][1683380.295274] Call Trace: [Wed Feb 6 06:01:20 2019][1683380.297918] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Feb 6 06:01:20 2019][1683380.305023] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 06:01:20 2019][1683380.312406] [] mdt_reint_rename_or_migrate.isra.51+0x67a/0x860 [mdt] [Wed Feb 6 06:01:20 2019][1683380.320623] [] mdt_reint_rename+0x13/0x20 [mdt] [Wed Feb 6 06:01:20 2019][1683380.327018] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 06:01:20 2019][1683380.333242] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 06:01:20 2019][1683380.339982] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 06:01:20 2019][1683380.345860] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 06:01:20 2019][1683380.352976] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 06:01:20 2019][1683380.360862] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 06:01:20 2019][1683380.367362] [] kthread+0xd1/0xe0 [Wed Feb 6 06:01:20 2019][1683380.372449] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 06:01:20 2019][1683380.379113] [] 0xffffffffffffffff [Wed Feb 6 06:01:20 2019][1683380.384316] LustreError: dumping log to /tmp/lustre-log.1549461718.15231 [Wed Feb 6 06:01:26 2019][1683386.172924] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 06:01:26 2019][1683386.183008] Lustre: Skipped 75 previous similar messages [Wed Feb 6 06:03:00 2019][1683479.944760] LustreError: 15231:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549461518, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff912e68e86300/0x21ed93519fd9d9b1 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 195 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 15231 timeout: 0 lvb_type: 0 [Wed Feb 6 06:06:06 2019][1683665.809424] Lustre: 8995:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 06:06:06 2019][1683665.809424] req@ffff912bcb7bf500 x1624669933388976/t0(0) o36->dba0ac03-f756-7e65-44de-ab2d15018d1d@10.9.101.41@o2ib4:424/0 lens 520/2888 e 24 to 0 dl 1549462009 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 06:07:56 2019][1683775.584179] Lustre: 9013:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 06:07:56 2019][1683775.584179] req@ffff9134cf067500 x1624712095711120/t0(0) o36->9cf6d7a8-4898-44eb-2590-b689cf0f2dd8@10.8.6.11@o2ib6:533/0 lens 520/2888 e 24 to 0 dl 1549462118 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 06:08:13 2019][1683793.255608] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 06:08:13 2019][1683793.266302] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 121263528 previous similar messages [Wed Feb 6 06:08:21 2019][1683801.277813] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549462139/real 1549462139] req@ffff913afee69e00 x1623682056381040/t0(0) o105->fir-MDT0002@10.10.113.6@tcp:15/16 lens 304/224 e 0 to 1 dl 1549462150 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 06:08:21 2019][1683801.305675] Lustre: 15181:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 121312269 previous similar messages [Wed Feb 6 06:08:50 2019][1683830.321235] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 06:08:50 2019][1683830.331854] Lustre: Skipped 85 previous similar messages [Wed Feb 6 06:11:27 2019][1683987.265013] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 06:11:27 2019][1683987.275111] Lustre: Skipped 78 previous similar messages [Wed Feb 6 06:18:13 2019][1684393.270670] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 06:18:13 2019][1684393.281366] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 121585119 previous similar messages [Wed Feb 6 06:18:21 2019][1684401.292871] Lustre: 15510:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549462739/real 1549462739] req@ffff913acc23e900 x1623682065425392/t0(0) o104->fir-MDT0002@10.10.113.6@tcp:15/16 lens 296/224 e 0 to 1 dl 1549462750 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 06:18:21 2019][1684401.320556] Lustre: 15510:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 122017808 previous similar messages [Wed Feb 6 06:18:51 2019][1684431.413338] Lustre: fir-MDT0002: Connection restored to 4d607cc8-3374-b84e-8be0-ab12c87ca2c7 (at 10.9.103.10@o2ib4) [Wed Feb 6 06:18:51 2019][1684431.423946] Lustre: Skipped 83 previous similar messages [Wed Feb 6 06:20:34 2019][1684534.114219] LustreError: 15510:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.6@tcp) failed to reply to blocking AST (req@ffff913acc23e900 x1623682065425392 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff915593e51d40/0x21ed93518580334f lrc: 5/0,0 mode: PR/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x2/0x0 rrc: 9 type: IBT flags: 0x60000400000020 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7f0f9 expref: 5 pid: 10287 timeout: 1684640 lvb_type: 0 [Wed Feb 6 06:20:34 2019][1684534.156848] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 06:20:34 2019][1684534.169472] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.6@tcp ns: mdt-fir-MDT0002_UUID lock: ffff915593e51d40/0x21ed93518580334f lrc: 4/0,0 mode: PR/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x2/0x0 rrc: 9 type: IBT flags: 0x60000400000020 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7f0f9 expref: 6 pid: 10287 timeout: 0 lvb_type: 0 [Wed Feb 6 06:20:34 2019][1684534.206729] LNet: Service thread pid 15181 completed after 10322.08s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 06:20:34 2019][1684534.223251] LNet: Skipped 3 previous similar messages [Wed Feb 6 06:21:28 2019][1684588.357755] Lustre: fir-MDT0002: Client 948f385b-b60e-f471-69e7-ab8aac33a9fb (at 10.8.4.7@o2ib6) reconnecting [Wed Feb 6 06:21:28 2019][1684588.367841] Lustre: Skipped 67 previous similar messages [Wed Feb 6 06:23:05 2019][1684685.118007] LustreError: 15330:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff913af0ccc800 x1623682065479328 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff91258b7698c0/0x21ed9351a3df43fa lrc: 4/0,0 mode: CR/CR res: [0x2c0001730:0x2:0x0].0x0 bits 0x9/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e165e4 expref: 28 pid: 15220 timeout: 1684791 lvb_type: 0 [Wed Feb 6 06:23:05 2019][1684685.118016] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.15@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 06:23:05 2019][1684685.118041] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.15@tcp ns: mdt-fir-MDT0002_UUID lock: ffff91380ca85580/0x21ed9351a3df1a55 lrc: 3/0,0 mode: PR/PR res: [0x2c000155d:0x16e7f:0x0].0x0 bits 0x13/0x0 rrc: 29 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e1642b expref: 29 pid: 15258 timeout: 0 lvb_type: 0 [Wed Feb 6 06:23:05 2019][1684685.118580] LNet: Service thread pid 15218 completed after 10320.47s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 06:23:05 2019][1684685.118582] LNet: Skipped 26 previous similar messages [Wed Feb 6 06:23:05 2019][1684685.232692] LustreError: 15330:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message [Wed Feb 6 06:23:05 2019][1684685.243162] LustreError: 15330:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff913980d2b400 ns: mdt-fir-MDT0002_UUID lock: ffff913bc3198480/0x21ed9351a3df449b lrc: 1/0,0 mode: EX/EX res: [0x2c0001730:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e16600 expref: 3 pid: 15330 timeout: 0 lvb_type: 3 [Wed Feb 6 06:24:17 2019][1684757.119815] LustreError: 9029:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.6@tcp) failed to reply to blocking AST (req@ffff91409aee6f00 x1623682065504816 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff91363f4ad580/0x21ed9351a3dfa0c8 lrc: 4/0,0 mode: PR/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x2/0x0 rrc: 11 type: IBT flags: 0x60000400000020 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7f100 expref: 8 pid: 6683 timeout: 1684863 lvb_type: 0 [Wed Feb 6 06:24:17 2019][1684757.162326] LustreError: 9029:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message [Wed Feb 6 06:24:17 2019][1684757.172415] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.6@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 06:24:17 2019][1684757.185017] LustreError: Skipped 2 previous similar messages [Wed Feb 6 06:24:17 2019][1684757.190867] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.6@tcp ns: mdt-fir-MDT0002_UUID lock: ffff91363f4ad580/0x21ed9351a3dfa0c8 lrc: 3/0,0 mode: PR/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x2/0x0 rrc: 11 type: IBT flags: 0x60000400000020 nid: 10.10.113.6@tcp remote: 0x87aefb63f4e7f100 expref: 9 pid: 6683 timeout: 0 lvb_type: 0 [Wed Feb 6 06:24:17 2019][1684757.227951] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [Wed Feb 6 06:25:34 2019][1684834.256742] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 06:25:34 2019][1684834.271875] LustreError: 15185:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549462872, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff914964763f00/0x21ed9351a3dedfb5 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 60 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a3dedfd1 expref: -99 pid: 15185 timeout: 0 lvb_type: 0 [Wed Feb 6 06:25:34 2019][1684834.271976] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Feb 6 06:25:34 2019][1684834.322135] LustreError: 15185:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 6 previous similar messages [Wed Feb 6 06:25:35 2019][1684835.073768] LustreError: 9027:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549462873, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff911d99fe1f80/0x21ed9351a3df656b lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 60 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a3df6572 expref: -99 pid: 9027 timeout: 0 lvb_type: 0 [Wed Feb 6 06:25:35 2019][1684835.113850] LustreError: 9027:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 10 previous similar messages [Wed Feb 6 06:25:36 2019][1684836.132332] LNet: Service thread pid 8983 completed after 9808.65s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 06:25:36 2019][1684836.148712] LNet: Skipped 40 previous similar messages [Wed Feb 6 06:25:48 2019][1684847.782085] LustreError: 8970:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549462886, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff9136d8785c40/0x21ed9351a3dfa0eb lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 60 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a3dfa0f2 expref: -99 pid: 8970 timeout: 0 lvb_type: 0 [Wed Feb 6 06:26:48 2019][1684908.123600] LustreError: 9029:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.112.12@tcp) failed to reply to blocking AST (req@ffff9145066a3c00 x1623682065560560 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff9159a03d57c0/0x21ed9351a4200a13 lrc: 4/0,0 mode: PR/PR res: [0x2c000172e:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.112.12@tcp remote: 0xefba423e132a4000 expref: 27 pid: 17205 timeout: 1685014 lvb_type: 0 [Wed Feb 6 06:26:48 2019][1684908.123603] LustreError: 17205:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff914eaeb40900 x1623682065560624 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff915612f7c140/0x21ed9351a3e3724b lrc: 4/0,0 mode: PR/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 11 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e16662 expref: 16 pid: 7029 timeout: 1685014 lvb_type: 0 [Wed Feb 6 06:26:48 2019][1684908.123607] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.15@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 06:26:48 2019][1684908.123613] LustreError: 17205:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message [Wed Feb 6 06:26:48 2019][1684908.123633] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.15@tcp ns: mdt-fir-MDT0002_UUID lock: ffff915612f7c140/0x21ed9351a3e3724b lrc: 3/0,0 mode: PR/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 11 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e16662 expref: 17 pid: 7029 timeout: 0 lvb_type: 0 [Wed Feb 6 06:26:48 2019][1684908.123702] LustreError: 15391:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff911b1b846800 ns: mdt-fir-MDT0002_UUID lock: ffff912628692f40/0x21ed9351a420136d lrc: 3/0,0 mode: PW/PW res: [0x2c0001730:0x2:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT flags: 0x50200000000000 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e1676c expref: 14 pid: 15391 timeout: 0 lvb_type: 0 [Wed Feb 6 06:26:48 2019][1684908.123705] LustreError: 15391:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message [Wed Feb 6 06:26:48 2019][1684908.314121] LustreError: 9029:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages [Wed Feb 6 06:27:58 2019][1684977.781349] Lustre: fir-MDT0002: haven't heard from client 375c01b4-7d81-9426-7dfa-7b0a1dfe6d49 (at 10.8.6.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff912d57f98800, cur 1549463316 expire 1549463166 last 1549463089 [Wed Feb 6 06:27:58 2019][1684977.803054] Lustre: Skipped 2 previous similar messages [Wed Feb 6 06:28:13 2019][1684993.285723] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 06:28:13 2019][1684993.296420] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 152385413 previous similar messages [Wed Feb 6 06:28:21 2019][1685001.307922] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549463339/real 1549463339] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549463350 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 06:28:21 2019][1685001.335627] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 152935290 previous similar messages [Wed Feb 6 06:29:19 2019][1685059.127392] LustreError: 15391:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff9125e0681e00 x1623682065620096 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff91503a639440/0x21ed9351a4ad2c8c lrc: 4/0,0 mode: PR/PR res: [0x2c0001730:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e167dc expref: 14 pid: 9020 timeout: 1685165 lvb_type: 0 [Wed Feb 6 06:29:19 2019][1685059.127397] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.15@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 06:29:19 2019][1685059.127399] LustreError: Skipped 5 previous similar messages [Wed Feb 6 06:29:19 2019][1685059.127429] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.15@tcp ns: mdt-fir-MDT0002_UUID lock: ffff9156db31f080/0x21ed9351a4ad2bb3 lrc: 3/0,0 mode: PR/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 12 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e167ce expref: 15 pid: 6178 timeout: 0 lvb_type: 0 [Wed Feb 6 06:29:19 2019][1685059.127433] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [Wed Feb 6 06:29:19 2019][1685059.235688] LustreError: 15391:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message [Wed Feb 6 06:29:19 2019][1685059.245901] LustreError: 15391:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9138b6dbc400 ns: mdt-fir-MDT0002_UUID lock: ffff91289b4a86c0/0x21ed9351a4ad2f87 lrc: 3/0,0 mode: PW/PW res: [0x2c0001730:0x2:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x50200000000000 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e167f1 expref: 2 pid: 15391 timeout: 0 lvb_type: 0 [Wed Feb 6 06:29:19 2019][1685059.280355] LustreError: 15391:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message [Wed Feb 6 06:29:19 2019][1685059.303164] Lustre: fir-MDT0002: Connection restored to e6807a0b-87d6-292a-3e89-235f77ec28d8 (at 10.10.113.15@tcp) [Wed Feb 6 06:29:19 2019][1685059.313687] Lustre: Skipped 56 previous similar messages [Wed Feb 6 06:31:48 2019][1685208.151138] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 06:31:48 2019][1685208.166265] LustreError: 15185:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549463246, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff9141bbe9a640/0x21ed9351a4ad1d97 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 56 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a4ad1da5 expref: -99 pid: 15185 timeout: 0 lvb_type: 0 [Wed Feb 6 06:31:48 2019][1685208.166404] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Feb 6 06:31:48 2019][1685208.166406] Lustre: Skipped 2 previous similar messages [Wed Feb 6 06:31:48 2019][1685208.221914] LustreError: 15185:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 18 previous similar messages [Wed Feb 6 06:31:48 2019][1685208.336129] LustreError: 6171:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549463246, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff912efe043600/0x21ed9351a4ad9164 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 160 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 6171 timeout: 0 lvb_type: 0 [Wed Feb 6 06:31:48 2019][1685208.375426] LustreError: 6171:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Feb 6 06:31:50 2019][1685210.131186] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.112.12@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 06:31:50 2019][1685210.131215] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.12@tcp ns: mdt-fir-MDT0002_UUID lock: ffff912e52e40fc0/0x21ed9351a4ad9d5e lrc: 3/0,0 mode: PR/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 13 type: IBT flags: 0x60200400000020 nid: 10.10.112.12@tcp remote: 0xefba423e132a4038 expref: 14 pid: 8996 timeout: 0 lvb_type: 0 [Wed Feb 6 06:31:50 2019][1685210.181289] LustreError: Skipped 2 previous similar messages [Wed Feb 6 06:34:21 2019][1685361.134971] LustreError: 32929:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff9126ce70ce00 x1623682065773968 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff912824ea1680/0x21ed9351a52fb3c9 lrc: 4/0,0 mode: PR/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 15 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e16853 expref: 14 pid: 32929 timeout: 1685467 lvb_type: 0 [Wed Feb 6 06:34:21 2019][1685361.134975] LustreError: 15185:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff913aca09da00 x1623682065773824 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff913af4257740/0x21ed9351a593848c lrc: 4/0,0 mode: PR/PR res: [0x2c0001730:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e16861 expref: 14 pid: 47983 timeout: 1685467 lvb_type: 0 [Wed Feb 6 06:34:21 2019][1685361.134980] LustreError: 15185:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages [Wed Feb 6 06:34:21 2019][1685361.135104] LustreError: 15185:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff91575fa61800 ns: mdt-fir-MDT0002_UUID lock: ffff912875e2aac0/0x21ed9351a5938629 lrc: 5/0,0 mode: PW/PW res: [0x2c0001730:0x2:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x50201000000000 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e16876 expref: 11 pid: 15185 timeout: 0 lvb_type: 0 [Wed Feb 6 06:34:21 2019][1685361.135105] LustreError: 15185:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message [Wed Feb 6 06:36:50 2019][1685510.151704] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 06:36:50 2019][1685510.166831] LustreError: 15184:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549463548, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff913f41e92ac0/0x21ed9351a5938f59 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 59 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a5938f60 expref: -99 pid: 15184 timeout: 0 lvb_type: 0 [Wed Feb 6 06:36:50 2019][1685510.166938] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Feb 6 06:36:50 2019][1685510.166940] Lustre: Skipped 2 previous similar messages [Wed Feb 6 06:36:50 2019][1685510.190706] LustreError: 6173:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549463548, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff913bc001cec0/0x21ed9351a593c385 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 155 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 6173 timeout: 0 lvb_type: 0 [Wed Feb 6 06:36:50 2019][1685510.190710] LustreError: 6173:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Feb 6 06:36:50 2019][1685510.272670] LustreError: 15184:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 54 previous similar messages [Wed Feb 6 06:36:52 2019][1685512.138754] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.15@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 06:36:52 2019][1685512.138787] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.12@tcp ns: mdt-fir-MDT0002_UUID lock: ffff9152d5b70d80/0x21ed9351a60d6307 lrc: 3/0,0 mode: PR/PR res: [0x2c000172e:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.112.12@tcp remote: 0xefba423e132a40d9 expref: 13 pid: 6758 timeout: 0 lvb_type: 0 [Wed Feb 6 06:36:52 2019][1685512.138791] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [Wed Feb 6 06:36:52 2019][1685512.198205] LustreError: Skipped 5 previous similar messages [Wed Feb 6 06:38:13 2019][1685593.300777] LNetError: 15165:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.112.12@tcp from [Wed Feb 6 06:38:13 2019][1685593.311555] LNetError: 15165:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 160436975 previous similar messages [Wed Feb 6 06:38:21 2019][1685601.322983] Lustre: 15183:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549463939/real 1549463939] req@ffff91289ee7da00 x1623682066136928/t0(0) o104->fir-MDT0002@10.10.113.15@tcp:15/16 lens 296/224 e 0 to 1 dl 1549463950 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 06:38:21 2019][1685601.350760] Lustre: 15183:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 161661922 previous similar messages [Wed Feb 6 06:39:15 2019][1685654.798399] Lustre: fir-MDT0002: haven't heard from client afad253f-2143-f49d-3f33-c08f8de9022c (at 10.8.8.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff911c07142000, cur 1549463993 expire 1549463843 last 1549463766 [Wed Feb 6 06:39:15 2019][1685654.820184] Lustre: Skipped 20 previous similar messages [Wed Feb 6 06:39:23 2019][1685663.143375] Lustre: fir-MDT0002: Connection restored to e6807a0b-87d6-292a-3e89-235f77ec28d8 (at 10.10.113.15@tcp) [Wed Feb 6 06:39:23 2019][1685663.153897] Lustre: Skipped 14 previous similar messages [Wed Feb 6 06:40:09 2019][1685708.662794] Lustre: fir-MDT0002: Client 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) reconnecting [Wed Feb 6 06:40:09 2019][1685708.672963] Lustre: Skipped 22 previous similar messages [Wed Feb 6 06:40:31 2019][1685730.800380] Lustre: fir-MDT0002: haven't heard from client a62a683a-0e26-f38d-6315-c77b5607ae54 (at 10.9.107.2@o2ib4) in 163 seconds. I think it's dead, and I am evicting it. exp ffff91352fdfec00, cur 1549464069 expire 1549463919 last 1549463906 [Wed Feb 6 06:40:31 2019][1685730.822251] Lustre: Skipped 14 previous similar messages [Wed Feb 6 06:41:35 2019][1685794.845402] Lustre: MGS: haven't heard from client 312f6082-f4dc-a6a8-e60c-b8f60cb2be68 (at 10.9.107.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff913d91740400, cur 1549464133 expire 1549463983 last 1549463906 [Wed Feb 6 06:41:47 2019][1685806.802185] Lustre: fir-MDT0002: haven't heard from client 582c0bf9-0c65-cb8a-6846-32e5a6d43ca1 (at 10.9.107.43@o2ib4) in 212 seconds. I think it's dead, and I am evicting it. exp ffff91448bff1000, cur 1549464145 expire 1549463995 last 1549463933 [Wed Feb 6 06:41:47 2019][1685806.824150] Lustre: Skipped 11 previous similar messages [Wed Feb 6 06:41:52 2019][1685812.157281] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 06:41:52 2019][1685812.172418] LustreError: 6758:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549463850, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff9124a7e0da00/0x21ed9351a6767fee lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 64 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a6768018 expref: -99 pid: 6758 timeout: 0 lvb_type: 0 [Wed Feb 6 06:41:52 2019][1685812.172422] LustreError: 15509:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549463850, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff914029f957c0/0x21ed9351a6767e19 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 62 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a6767e27 expref: -99 pid: 15509 timeout: 0 lvb_type: 0 [Wed Feb 6 06:41:52 2019][1685812.172426] LustreError: 48636:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549463850, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff9140a489e9c0/0x21ed9351a67682aa lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 65 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a67682b1 expref: -99 pid: 48636 timeout: 0 lvb_type: 0 [Wed Feb 6 06:41:52 2019][1685812.172429] LustreError: 15510:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549463850, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff914029f92f40/0x21ed9351a6768096 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 66 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a676809d expref: -99 pid: 15510 timeout: 0 lvb_type: 0 [Wed Feb 6 06:41:52 2019][1685812.172433] LustreError: 15509:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 17 previous similar messages [Wed Feb 6 06:41:52 2019][1685812.172435] LustreError: 48636:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 20 previous similar messages [Wed Feb 6 06:41:52 2019][1685812.172441] LustreError: 15510:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 20 previous similar messages [Wed Feb 6 06:41:52 2019][1685812.172554] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Feb 6 06:41:52 2019][1685812.172555] Lustre: Skipped 2 previous similar messages [Wed Feb 6 06:41:52 2019][1685812.212521] LustreError: 17207:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549463850, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9159f4e9b840/0x21ed9351a6769bfc lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 153 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 17207 timeout: 0 lvb_type: 0 [Wed Feb 6 06:41:52 2019][1685812.212524] LustreError: 17207:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 4 previous similar messages [Wed Feb 6 06:41:52 2019][1685812.432482] LustreError: 6758:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 28 previous similar messages [Wed Feb 6 06:42:49 2019][1685869.399718] LustreError: 6690:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549463907, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff9120865c2f40/0x21ed9351a69f830e lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 59 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a69f831c expref: -99 pid: 6690 timeout: 0 lvb_type: 0 [Wed Feb 6 06:42:49 2019][1685869.439796] LustreError: 6690:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 3 previous similar messages [Wed Feb 6 06:42:51 2019][1685870.860271] Lustre: MGS: haven't heard from client 93fd3055-c44c-8ba3-0e66-a7b7b3f21418 (at 10.9.107.19@o2ib4) in 158 seconds. I think it's dead, and I am evicting it. exp ffff9143fa81b400, cur 1549464209 expire 1549464059 last 1549464051 [Wed Feb 6 06:42:51 2019][1685870.881540] Lustre: Skipped 4 previous similar messages [Wed Feb 6 06:43:03 2019][1685882.804083] Lustre: fir-MDT0002: haven't heard from client 1ee60be4-911f-02ce-275f-36587bcd7a56 (at 10.9.107.19@o2ib4) in 170 seconds. I think it's dead, and I am evicting it. exp ffff91531c759400, cur 1549464221 expire 1549464071 last 1549464051 [Wed Feb 6 06:43:03 2019][1685882.826050] Lustre: Skipped 1 previous similar message [Wed Feb 6 06:44:24 2019][1685964.561108] LustreError: 15331:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549464002, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff9143a8e93600/0x21ed9351a6d7a106 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 59 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a6d7a10d expref: -99 pid: 15331 timeout: 0 lvb_type: 0 [Wed Feb 6 06:45:03 2019][1686003.151081] LustreError: 6173:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff913aa87bd400 x1623682066291456 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff9140b7b0e9c0/0x21ed9351a737fe41 lrc: 4/0,0 mode: PR/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 9 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e16a67 expref: 14 pid: 9199 timeout: 1686109 lvb_type: 0 [Wed Feb 6 06:45:03 2019][1686003.151201] LustreError: 6758:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff913c53563400 ns: mdt-fir-MDT0002_UUID lock: ffff9154f0365a00/0x21ed9351a74d19e6 lrc: 1/0,0 mode: --/PW res: [0x2c0001730:0x2:0x0].0x0 bits 0x40/0x0 rrc: 3 type: IBT flags: 0x54a01000000000 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e16a8a expref: 9 pid: 6758 timeout: 0 lvb_type: 0 [Wed Feb 6 06:45:03 2019][1686003.151204] LustreError: 6758:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 6 previous similar messages [Wed Feb 6 06:45:03 2019][1686003.238359] LustreError: 6173:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 13 previous similar messages [Wed Feb 6 06:47:32 2019][1686152.168811] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 06:47:32 2019][1686152.183944] LustreError: 6177:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549464190, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff915412e033c0/0x21ed9351a74d3186 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 61 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a74d31c5 expref: -99 pid: 6177 timeout: 0 lvb_type: 0 [Wed Feb 6 06:47:32 2019][1686152.184038] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 06:47:32 2019][1686152.184051] Lustre: Skipped 2 previous similar messages [Wed Feb 6 06:47:32 2019][1686152.199817] LustreError: 9182:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549464190, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff9119eab65580/0x21ed9351a74d452f lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 148 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 9182 timeout: 0 lvb_type: 0 [Wed Feb 6 06:47:32 2019][1686152.199820] LustreError: 9182:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 5 previous similar messages [Wed Feb 6 06:47:32 2019][1686152.290733] LustreError: 6177:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 43 previous similar messages [Wed Feb 6 06:47:34 2019][1686154.154868] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.112.12@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 06:47:34 2019][1686154.154889] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.12@tcp ns: mdt-fir-MDT0002_UUID lock: ffff91571f3d3600/0x21ed9351a7af6de6 lrc: 3/0,0 mode: PR/PR res: [0x2c000172e:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.112.12@tcp remote: 0xefba423e132a4356 expref: 17 pid: 15407 timeout: 0 lvb_type: 0 [Wed Feb 6 06:47:34 2019][1686154.154892] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 9 previous similar messages [Wed Feb 6 06:47:34 2019][1686154.214500] LustreError: Skipped 12 previous similar messages [Wed Feb 6 06:47:44 2019][1686163.811136] Lustre: fir-MDT0002: haven't heard from client ee310e5c-55e4-0a0f-af99-ae0536891531 (at 10.8.27.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff913b0b913400, cur 1549464502 expire 1549464352 last 1549464275 [Wed Feb 6 06:48:13 2019][1686193.315837] LNetError: 15266:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.15@tcp from [Wed Feb 6 06:48:13 2019][1686193.326616] LNetError: 15266:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 174487484 previous similar messages [Wed Feb 6 06:48:21 2019][1686201.338043] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549464539/real 1549464539] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549464550 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 06:48:21 2019][1686201.365728] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 175894598 previous similar messages [Wed Feb 6 06:50:03 2019][1686303.345611] LustreError: 6690:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549464341, 300s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff9124d52f21c0/0x21ed9351a7af86f2 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 42 type: IBT flags: 0x1000001000000 nid: local remote: 0x21ed9351a7af8700 expref: -99 pid: 6690 timeout: 0 lvb_type: 0 [Wed Feb 6 06:50:03 2019][1686303.385743] LustreError: 6690:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 3 previous similar messages [Wed Feb 6 06:50:05 2019][1686305.159177] Lustre: fir-MDT0002: Connection restored to e6807a0b-87d6-292a-3e89-235f77ec28d8 (at 10.10.113.15@tcp) [Wed Feb 6 06:50:05 2019][1686305.169698] Lustre: Skipped 21 previous similar messages [Wed Feb 6 06:52:03 2019][1686422.819669] Lustre: MGS: haven't heard from client 1bf42348-d0c0-542a-e456-6e6923491804 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff915760e47c00, cur 1549464761 expire 1549464611 last 1549464534 [Wed Feb 6 06:52:03 2019][1686422.840981] Lustre: Skipped 2 previous similar messages [Wed Feb 6 06:52:45 2019][1686465.288292] Lustre: fir-MDT0002: Client 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) reconnecting [Wed Feb 6 06:52:45 2019][1686465.298472] Lustre: Skipped 1 previous similar message [Wed Feb 6 06:55:07 2019][1686607.166232] LustreError: 15248:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.112.12@tcp) failed to reply to blocking AST (req@ffff912bc7a8c500 x1623682068044000 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff9156e6ec0240/0x21ed9351a96ab19d lrc: 4/0,0 mode: PR/PR res: [0x2c000172e:0x4:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.112.12@tcp remote: 0xefba423e132a46dd expref: 17 pid: 7135 timeout: 1686713 lvb_type: 0 [Wed Feb 6 06:55:07 2019][1686607.166334] LustreError: 15391:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9143f6a61400 ns: mdt-fir-MDT0002_UUID lock: ffff9128d6d25340/0x21ed9351a96ab33a lrc: 3/0,0 mode: --/PW res: [0x2c0001730:0x2:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x54a01000000000 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e16c9e expref: 11 pid: 15391 timeout: 0 lvb_type: 0 [Wed Feb 6 06:55:07 2019][1686607.166337] LustreError: 15391:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 7 previous similar messages [Wed Feb 6 06:55:07 2019][1686607.253596] LustreError: 15248:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 10 previous similar messages [Wed Feb 6 06:57:38 2019][1686758.170022] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.112.12@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 06:57:38 2019][1686758.170044] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.15@tcp ns: mdt-fir-MDT0002_UUID lock: ffff9125eaebc5c0/0x21ed9351a9e8f4b7 lrc: 3/0,0 mode: PR/PR res: [0x2c0001730:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e16d0e expref: 15 pid: 8995 timeout: 0 lvb_type: 0 [Wed Feb 6 06:57:38 2019][1686758.170047] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 8 previous similar messages [Wed Feb 6 06:57:38 2019][1686758.229571] LustreError: Skipped 9 previous similar messages [Wed Feb 6 06:58:13 2019][1686793.330896] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 06:58:13 2019][1686793.341595] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 158106411 previous similar messages [Wed Feb 6 06:58:21 2019][1686801.353100] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549465139/real 1549465139] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549465150 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 06:58:21 2019][1686801.380784] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 158241674 previous similar messages [Wed Feb 6 07:00:09 2019][1686909.174331] Lustre: fir-MDT0002: Connection restored to 034162e2-29a3-b06b-c62e-a56d63789e61 (at 10.10.112.12@tcp) [Wed Feb 6 07:00:09 2019][1686909.184853] Lustre: Skipped 11 previous similar messages [Wed Feb 6 07:05:11 2019][1687211.181386] LustreError: 24900:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.112.12@tcp) failed to reply to blocking AST (req@ffff91529faf3600 x1623682069212384 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff912d99707bc0/0x21ed9351ab6877d8 lrc: 4/0,0 mode: PR/PR res: [0x2c000173d:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.112.12@tcp remote: 0xefba423e132a4c0f expref: 14 pid: 9171 timeout: 1687317 lvb_type: 0 [Wed Feb 6 07:05:11 2019][1687211.181514] LustreError: 6806:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff912f9e0adc00 ns: mdt-fir-MDT0002_UUID lock: ffff9129ac161680/0x21ed9351ab687848 lrc: 1/0,0 mode: --/PW res: [0x2c0001730:0x2:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT flags: 0x54a01000000000 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e16eb2 expref: 10 pid: 6806 timeout: 0 lvb_type: 0 [Wed Feb 6 07:05:11 2019][1687211.181517] LustreError: 6806:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 7 previous similar messages [Wed Feb 6 07:05:11 2019][1687211.268489] LustreError: 24900:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 10 previous similar messages [Wed Feb 6 07:05:21 2019][1687220.912326] Lustre: fir-MDT0002: Client 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) reconnecting [Wed Feb 6 07:07:42 2019][1687362.185189] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.15@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 07:07:42 2019][1687362.185212] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.15@tcp ns: mdt-fir-MDT0002_UUID lock: ffff9144f6351d40/0x21ed9351aba73987 lrc: 3/0,0 mode: PR/PR res: [0x2c0001730:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e16f22 expref: 15 pid: 9020 timeout: 0 lvb_type: 0 [Wed Feb 6 07:07:42 2019][1687362.185215] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 7 previous similar messages [Wed Feb 6 07:07:42 2019][1687362.244725] LustreError: Skipped 10 previous similar messages [Wed Feb 6 07:08:13 2019][1687393.345957] LNetError: 8973:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.112.12@tcp from [Wed Feb 6 07:08:13 2019][1687393.356651] LNetError: 8973:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 144442567 previous similar messages [Wed Feb 6 07:08:21 2019][1687401.368159] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549465739/real 1549465739] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549465750 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 07:08:21 2019][1687401.395843] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 145126976 previous similar messages [Wed Feb 6 07:10:11 2019][1687510.833643] Lustre: MGS: Connection restored to 73a03089-273f-0ccb-be41-f8441c1f3ecb (at 10.8.7.11@o2ib6) [Wed Feb 6 07:10:11 2019][1687510.843386] Lustre: Skipped 33 previous similar messages [Wed Feb 6 07:15:15 2019][1687815.196550] LustreError: 15293:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff91371e52aa00 x1623682070189120 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff9158e5f9c140/0x21ed9351ac5d7006 lrc: 4/0,0 mode: PR/PR res: [0x2c0001730:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e1716e expref: 14 pid: 6177 timeout: 1687921 lvb_type: 0 [Wed Feb 6 07:15:15 2019][1687815.196655] LustreError: 15163:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff91212bf25400 ns: mdt-fir-MDT0002_UUID lock: ffff9135dd8e9680/0x21ed9351ac5d6f11 lrc: 1/0,0 mode: --/PW res: [0x2c000173d:0x2:0x0].0x0 bits 0x40/0x0 rrc: 3 type: IBT flags: 0x54a01000000000 nid: 10.10.112.12@tcp remote: 0xefba423e132a4db3 expref: 9 pid: 15163 timeout: 0 lvb_type: 0 [Wed Feb 6 07:15:15 2019][1687815.196657] LustreError: 15163:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 6 previous similar messages [Wed Feb 6 07:15:15 2019][1687815.283850] LustreError: 15293:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 7 previous similar messages [Wed Feb 6 07:17:46 2019][1687966.200336] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.112.12@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 07:17:46 2019][1687966.200361] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.15@tcp ns: mdt-fir-MDT0002_UUID lock: ffff912e69804800/0x21ed9351aca6d5cd lrc: 3/0,0 mode: PR/PR res: [0x2c0001730:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e171f3 expref: 15 pid: 15243 timeout: 0 lvb_type: 0 [Wed Feb 6 07:17:46 2019][1687966.200363] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 6 previous similar messages [Wed Feb 6 07:17:46 2019][1687966.259984] LustreError: Skipped 7 previous similar messages [Wed Feb 6 07:17:57 2019][1687977.512159] Lustre: fir-MDT0002: Client 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) reconnecting [Wed Feb 6 07:18:13 2019][1687993.361011] LNetError: 15243:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.112.12@tcp from [Wed Feb 6 07:18:13 2019][1687993.371792] LNetError: 15243:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 149540606 previous similar messages [Wed Feb 6 07:18:21 2019][1688001.383213] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549466339/real 1549466339] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549466350 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 07:18:21 2019][1688001.410892] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 150222126 previous similar messages [Wed Feb 6 07:20:17 2019][1688117.204895] Lustre: fir-MDT0002: Connection restored to e6807a0b-87d6-292a-3e89-235f77ec28d8 (at 10.10.113.15@tcp) [Wed Feb 6 07:20:17 2019][1688117.215423] Lustre: Skipped 29 previous similar messages [Wed Feb 6 07:25:19 2019][1688419.211711] LustreError: 6175:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.112.12@tcp) failed to reply to blocking AST (req@ffff913a9749bc00 x1623682071320768 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff91219a184380/0x21ed9351ada0ef5b lrc: 4/0,0 mode: PR/PR res: [0x2c000173d:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.112.12@tcp remote: 0xefba423e132a4fb2 expref: 14 pid: 15261 timeout: 1688525 lvb_type: 0 [Wed Feb 6 07:25:19 2019][1688419.211802] LustreError: 17050:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff91283f275000 ns: mdt-fir-MDT0002_UUID lock: ffff914fc5253180/0x21ed9351ada0f233 lrc: 3/0,0 mode: --/PW res: [0x2c0001741:0x1:0x0].0x0 bits 0x40/0x0 rrc: 7 type: IBT flags: 0x54a01000000000 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e175f1 expref: 13 pid: 17050 timeout: 0 lvb_type: 0 [Wed Feb 6 07:25:19 2019][1688419.211805] LustreError: 17050:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 7 previous similar messages [Wed Feb 6 07:25:19 2019][1688419.299077] LustreError: 6175:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 10 previous similar messages [Wed Feb 6 07:27:50 2019][1688570.215494] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.112.12@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 07:27:50 2019][1688570.215522] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.113.15@tcp ns: mdt-fir-MDT0002_UUID lock: ffff9126c5cc3cc0/0x21ed9351ade72ab1 lrc: 3/0,0 mode: PR/PR res: [0x2c0001741:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e17661 expref: 15 pid: 6806 timeout: 0 lvb_type: 0 [Wed Feb 6 07:27:50 2019][1688570.215524] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 7 previous similar messages [Wed Feb 6 07:27:50 2019][1688570.275035] LustreError: Skipped 10 previous similar messages [Wed Feb 6 07:28:13 2019][1688593.376064] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 07:28:13 2019][1688593.386762] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 150775603 previous similar messages [Wed Feb 6 07:28:21 2019][1688601.398272] Lustre: 15409:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549466939/real 1549466939] req@ffff911cb39c8c00 x1623682072160384/t0(0) o104->fir-MDT0002@10.10.113.15@tcp:15/16 lens 296/224 e 0 to 1 dl 1549466950 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 07:28:21 2019][1688601.426042] Lustre: 15409:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 151342318 previous similar messages [Wed Feb 6 07:30:20 2019][1688719.963861] Lustre: MGS: Connection restored to 79e6e2fa-2c74-15f5-f673-93328beb2221 (at 10.9.112.1@o2ib4) [Wed Feb 6 07:30:20 2019][1688719.973689] Lustre: Skipped 13 previous similar messages [Wed Feb 6 07:30:33 2019][1688733.137339] Lustre: fir-MDT0002: Client 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) reconnecting [Wed Feb 6 07:35:23 2019][1689023.226876] LustreError: 9171:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.112.12@tcp) failed to reply to blocking AST (req@ffff913a97cf7b00 x1623682072896656 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff911c0586c800/0x21ed9351aefea889 lrc: 4/0,0 mode: PR/PR res: [0x2c000173d:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.112.12@tcp remote: 0xefba423e132a51c6 expref: 14 pid: 8969 timeout: 1689129 lvb_type: 0 [Wed Feb 6 07:35:23 2019][1689023.269381] LustreError: 9171:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 7 previous similar messages [Wed Feb 6 07:35:23 2019][1689023.279665] LustreError: 9171:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff9133bffbd400 ns: mdt-fir-MDT0002_UUID lock: ffff91534f31f080/0x21ed9351aefeaa88 lrc: 1/0,0 mode: --/PW res: [0x2c000173d:0x2:0x0].0x0 bits 0x40/0x0 rrc: 3 type: IBT flags: 0x54a01000000000 nid: 10.10.112.12@tcp remote: 0xefba423e132a51db expref: 9 pid: 9171 timeout: 0 lvb_type: 0 [Wed Feb 6 07:35:23 2019][1689023.313935] LustreError: 9171:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 7 previous similar messages [Wed Feb 6 07:37:54 2019][1689174.230660] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.112.12@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 07:37:54 2019][1689174.243347] LustreError: Skipped 7 previous similar messages [Wed Feb 6 07:37:54 2019][1689174.249209] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.12@tcp ns: mdt-fir-MDT0002_UUID lock: ffff91591cf0c380/0x21ed9351af5e5383 lrc: 3/0,0 mode: PR/PR res: [0x2c000173d:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.112.12@tcp remote: 0xefba423e132a52f3 expref: 15 pid: 17053 timeout: 0 lvb_type: 0 [Wed Feb 6 07:37:54 2019][1689174.286346] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 8 previous similar messages [Wed Feb 6 07:38:13 2019][1689193.391123] LNetError: 17219:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.15@tcp from [Wed Feb 6 07:38:13 2019][1689193.401912] LNetError: 17219:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 146578742 previous similar messages [Wed Feb 6 07:38:21 2019][1689201.413325] Lustre: 15202:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549467539/real 1549467539] req@ffff912690e25700 x1623682073416384/t0(0) o104->fir-MDT0002@10.10.112.12@tcp:15/16 lens 296/224 e 0 to 1 dl 1549467550 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 07:38:21 2019][1689201.441091] Lustre: 15202:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 146840347 previous similar messages [Wed Feb 6 07:40:00 2019][1689300.160812] LNet: Service thread pid 15163 was inactive for 200.02s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 07:40:00 2019][1689300.177927] Pid: 15163, comm: mdt01_041 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 07:40:00 2019][1689300.187838] Call Trace: [Wed Feb 6 07:40:00 2019][1689300.190473] [] 0xffffffffffffffff [Wed Feb 6 07:40:00 2019][1689300.195680] LustreError: dumping log to /tmp/lustre-log.1549467638.15163 [Wed Feb 6 07:40:25 2019][1689325.274497] Lustre: fir-MDT0002: Connection restored to 034162e2-29a3-b06b-c62e-a56d63789e61 (at 10.10.112.12@tcp) [Wed Feb 6 07:40:25 2019][1689325.285015] Lustre: Skipped 15 previous similar messages [Wed Feb 6 07:40:52 2019][1689351.754222] Lustre: fir-OST0028-osc-MDT0002: Connection to fir-OST0028 (at 10.0.10.107@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 07:40:55 2019][1689355.170211] Lustre: fir-OST002a-osc-MDT0002: Connection to fir-OST002a (at 10.0.10.107@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 07:40:58 2019][1689358.046177] Lustre: fir-OST001a-osc-MDT0000: Connection to fir-OST001a (at 10.0.10.105@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 07:40:58 2019][1689358.062423] Lustre: Skipped 3 previous similar messages [Wed Feb 6 07:41:03 2019][1689363.520388] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 21s! [ldlm_cn03_002:6040] [Wed Feb 6 07:41:03 2019][1689363.528559] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:03 2019][1689363.635329] CPU: 11 PID: 6040 Comm: ldlm_cn03_002 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:03 2019][1689363.648006] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:03 2019][1689363.655746] task: ffff9129f304d140 ti: ffff9159f93ac000 task.ti: ffff9159f93ac000 [Wed Feb 6 07:41:03 2019][1689363.663399] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:41:03 2019][1689363.673505] RSP: 0018:ffff9159f93afc38 EFLAGS: 00000246 [Wed Feb 6 07:41:03 2019][1689363.678992] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000590000 [Wed Feb 6 07:41:03 2019][1689363.686297] RDX: ffff9149ff75b780 RSI: 0000000000b10001 RDI: ffff911f85a51a5c [Wed Feb 6 07:41:03 2019][1689363.693604] RBP: ffff9159f93afc38 R08: ffff915a3f49b780 R09: 0000000000000000 [Wed Feb 6 07:41:04 2019][1689363.700909] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: ffff9159f93afbe0 [Wed Feb 6 07:41:04 2019][1689363.708217] R13: ffff913ac8d91740 R14: ffff9159f93afba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:41:04 2019][1689363.715522] FS: 00007f2b3eadb740(0000) GS:ffff915a3f480000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:04 2019][1689363.723782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:04 2019][1689363.729702] CR2: 00007f63fb343000 CR3: 0000004031fb0000 CR4: 00000000003407e0 [Wed Feb 6 07:41:04 2019][1689363.737009] Call Trace: [Wed Feb 6 07:41:04 2019][1689363.739636] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:04 2019][1689363.746077] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:04 2019][1689363.751775] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.758510] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.765249] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.772327] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.779234] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.786324] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.794190] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.799393] NMI watchdog: BUG: soft lockup - CPU#43 stuck for 22s! [ldlm_cn03_019:25318] [Wed Feb 6 07:41:04 2019][1689363.799426] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:04 2019][1689363.799441] CPU: 43 PID: 25318 Comm: ldlm_cn03_019 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:04 2019][1689363.799442] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:04 2019][1689363.799443] task: ffff9139f3766180 ti: ffff912c8410c000 task.ti: ffff912c8410c000 [Wed Feb 6 07:41:04 2019][1689363.799447] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:41:04 2019][1689363.799448] RSP: 0018:ffff912c8410fc38 EFLAGS: 00000246 [Wed Feb 6 07:41:04 2019][1689363.799449] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001590000 [Wed Feb 6 07:41:04 2019][1689363.799450] RDX: ffff915a3f51b780 RSI: 0000000000990001 RDI: ffff911f85a51a5c [Wed Feb 6 07:41:04 2019][1689363.799451] RBP: ffff912c8410fc38 R08: ffff915a3f69b780 R09: 0000000000000000 [Wed Feb 6 07:41:04 2019][1689363.799452] R10: ffff911b3fc07600 R11: ffffdbb2e337ba00 R12: ffff912c8410fbe0 [Wed Feb 6 07:41:04 2019][1689363.799453] R13: ffff913ac8d91740 R14: ffff912c8410fba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:41:04 2019][1689363.799454] FS: 00007f6417cd6700(0000) GS:ffff915a3f680000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:04 2019][1689363.799455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:04 2019][1689363.799456] CR2: 00007f2781327000 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:41:04 2019][1689363.799457] Call Trace: [Wed Feb 6 07:41:04 2019][1689363.799460] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:04 2019][1689363.799462] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:04 2019][1689363.799495] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.799528] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.799565] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.799601] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.799636] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.799677] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.799717] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.799720] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:41:04 2019][1689363.799759] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.799798] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689363.799801] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:04 2019][1689363.799803] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:04 2019][1689363.799805] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:04 2019][1689363.799807] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:04 2019][1689363.799827] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:41:04 2019][1689364.142163] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689364.148637] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:04 2019][1689364.156199] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:04 2019][1689364.161251] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:04 2019][1689364.167519] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:04 2019][1689364.174132] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:04 2019][1689364.180397] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:41:05 2019][1689364.937781] Lustre: fir-OST0022-osc-MDT0000: Connection to fir-OST0022 (at 10.0.10.105@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 07:41:06 2019][1689366.209236] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Wed Feb 6 07:41:06 2019][1689366.219493] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 7 previous similar messages [Wed Feb 6 07:41:06 2019][1689366.229749] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.101@o2ib7 (7): c: 0, oc: 0, rc: 8 [Wed Feb 6 07:41:06 2019][1689366.241825] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 7 previous similar messages [Wed Feb 6 07:41:07 2019][1689367.495488] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [ldlm_cn00_017:14897] [Wed Feb 6 07:41:07 2019][1689367.503487] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [ldlm_cn01_029:28330] [Wed Feb 6 07:41:07 2019][1689367.503539] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:07 2019][1689367.503563] CPU: 9 PID: 28330 Comm: ldlm_cn01_029 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:07 2019][1689367.503563] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:07 2019][1689367.503565] task: ffff9159aa3de180 ti: ffff915863708000 task.ti: ffff915863708000 [Wed Feb 6 07:41:07 2019][1689367.503574] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:41:07 2019][1689367.503575] RSP: 0018:ffff91586370bbb8 EFLAGS: 00000246 [Wed Feb 6 07:41:07 2019][1689367.503576] RAX: 0000000000000000 RBX: ffff91586370bbf0 RCX: 0000000000490000 [Wed Feb 6 07:41:07 2019][1689367.503577] RDX: ffff9129fefdb780 RSI: 0000000000e10000 RDI: ffff911f85a51a5c [Wed Feb 6 07:41:07 2019][1689367.503578] RBP: ffff91586370bbb8 R08: ffff9139ff69b780 R09: 0000000000000000 [Wed Feb 6 07:41:07 2019][1689367.503579] R10: ffff911b3fc07600 R11: ffff913957f6a660 R12: 0000000000000000 [Wed Feb 6 07:41:07 2019][1689367.503579] R13: 0000000000000000 R14: 0000000000000002 R15: ffff9135a3402f40 [Wed Feb 6 07:41:07 2019][1689367.503581] FS: 00007f63fb2d1700(0000) GS:ffff9139ff680000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:07 2019][1689367.503582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:07 2019][1689367.503583] CR2: 00007fd22b9aa000 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:41:08 2019][1689367.503584] Call Trace: [Wed Feb 6 07:41:08 2019][1689367.503591] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:08 2019][1689367.503595] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:08 2019][1689367.503634] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.503668] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.503673] [] ? native_queued_spin_lock_slowpath+0x15e/0x200 [Wed Feb 6 07:41:08 2019][1689367.503706] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.503744] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.503781] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.503816] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.503859] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.503862] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:41:08 2019][1689367.503865] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:41:08 2019][1689367.503905] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.503945] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.503948] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:08 2019][1689367.503949] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:08 2019][1689367.503952] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:08 2019][1689367.503954] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:08 2019][1689367.503974] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:41:08 2019][1689367.780496] NMI watchdog: BUG: soft lockup - CPU#41 stuck for 22s! [ldlm_cn01_030:28331] [Wed Feb 6 07:41:08 2019][1689367.780543] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:08 2019][1689367.780564] CPU: 41 PID: 28331 Comm: ldlm_cn01_030 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:08 2019][1689367.780565] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:08 2019][1689367.780566] task: ffff9159aa3dd140 ti: ffff9156bc290000 task.ti: ffff9156bc290000 [Wed Feb 6 07:41:08 2019][1689367.780572] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:41:08 2019][1689367.780573] RSP: 0018:ffff9156bc293c38 EFLAGS: 00000246 [Wed Feb 6 07:41:08 2019][1689367.780574] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001490000 [Wed Feb 6 07:41:08 2019][1689367.780575] RDX: ffff9149ff85b780 RSI: 0000000001310001 RDI: ffff911f85a51a5c [Wed Feb 6 07:41:08 2019][1689367.780576] RBP: ffff9156bc293c38 R08: ffff9139ff89b780 R09: 0000000000000000 [Wed Feb 6 07:41:08 2019][1689367.780577] R10: ffff911b3fc07600 R11: ffff913957f6a660 R12: ffff9156bc293be0 [Wed Feb 6 07:41:08 2019][1689367.780577] R13: ffff913ac8d91740 R14: ffff9156bc293ba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:41:08 2019][1689367.780579] FS: 00007fe121ad2740(0000) GS:ffff9139ff880000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:08 2019][1689367.780580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:08 2019][1689367.780581] CR2: 0000000001111108 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:41:08 2019][1689367.780582] Call Trace: [Wed Feb 6 07:41:08 2019][1689367.780587] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:08 2019][1689367.780590] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:08 2019][1689367.780626] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.780660] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.780697] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.780734] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.780769] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.780811] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.780815] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:41:08 2019][1689367.780817] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:41:08 2019][1689367.780858] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.780898] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689367.780900] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:08 2019][1689367.780902] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:08 2019][1689367.780904] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:08 2019][1689367.780905] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:08 2019][1689367.780926] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:41:08 2019][1689368.199016] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:08 2019][1689368.305785] CPU: 8 PID: 14897 Comm: ldlm_cn00_017 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:08 2019][1689368.318462] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:08 2019][1689368.326201] task: ffff9135b0d72080 ti: ffff915979284000 task.ti: ffff915979284000 [Wed Feb 6 07:41:08 2019][1689368.333855] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:41:08 2019][1689368.343968] RSP: 0018:ffff915979287c38 EFLAGS: 00000246 [Wed Feb 6 07:41:08 2019][1689368.349455] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000410000 [Wed Feb 6 07:41:08 2019][1689368.356762] RDX: ffff9149ff8db780 RSI: 0000000001710001 RDI: ffff911f85a51a5c [Wed Feb 6 07:41:08 2019][1689368.364068] RBP: ffff915979287c38 R08: ffff9129fee9b780 R09: 0000000000000000 [Wed Feb 6 07:41:08 2019][1689368.371372] R10: ffff911b3fc07600 R11: ffffdbb23642e200 R12: ffff915979287be0 [Wed Feb 6 07:41:08 2019][1689368.378680] R13: ffff913ac8d91740 R14: ffff915979287ba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:41:08 2019][1689368.385988] FS: 00007f63fb2d1700(0000) GS:ffff9129fee80000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:08 2019][1689368.394247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:08 2019][1689368.400164] CR2: 00007f641c871000 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:41:08 2019][1689368.407471] Call Trace: [Wed Feb 6 07:41:08 2019][1689368.410103] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:08 2019][1689368.416541] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:08 2019][1689368.422242] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689368.428972] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689368.435713] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689368.442790] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689368.449698] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689368.456787] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689368.464652] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689368.471608] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:41:08 2019][1689368.477393] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689368.483875] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:08 2019][1689368.491438] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:08 2019][1689368.496491] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:08 2019][1689368.502757] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:08 2019][1689368.509369] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:08 2019][1689368.515634] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:41:09 2019][1689368.739528] Lustre: fir-MDT0000: Client 26c5771f-6bff-b79c-941d-a328fa48123c (at 10.9.102.6@o2ib4) reconnecting [Wed Feb 6 07:41:13 2019][1689373.194329] Lustre: fir-OST000b-osc-MDT0000: Connection to fir-OST000b (at 10.0.10.102@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 07:41:13 2019][1689373.210573] Lustre: Skipped 11 previous similar messages [Wed Feb 6 07:41:21 2019][1689381.393881] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 2 seconds [Wed Feb 6 07:41:21 2019][1689381.404139] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 2 previous similar messages [Wed Feb 6 07:41:21 2019][1689381.414392] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.202@o2ib7 (7): c: 0, oc: 1, rc: 8 [Wed Feb 6 07:41:21 2019][1689381.426462] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 2 previous similar messages [Wed Feb 6 07:41:21 2019][1689381.436283] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.101@o2ib7: 17 seconds [Wed Feb 6 07:41:21 2019][1689381.446620] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 7 previous similar messages [Wed Feb 6 07:41:23 2019][1689382.932688] LNetError: 5875:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) [Wed Feb 6 07:41:26 2019][1689386.490001] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [Wed Feb 6 07:41:26 2019][1689386.499478] Lustre: Skipped 1 previous similar message [Wed Feb 6 07:41:26 2019][1689386.504798] Lustre: 20039:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=30, svcEst=69, delay=0 [Wed Feb 6 07:41:26 2019][1689386.515141] Lustre: 20039:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 1 previous similar message [Wed Feb 6 07:41:26 2019][1689386.524707] Lustre: 20039:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-5s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff9125694e8000 x1624672799360352/t0(0) o103->fa5fc8be-f740-e109-5125-43d4f0bf6a14@10.9.102.33@o2ib4:94/0 lens 328/0 e 0 to 0 dl 1549467719 ref 2 fl New:H/0/ffffffff rc 0/-1 [Wed Feb 6 07:41:26 2019][1689386.555848] Lustre: 20039:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages [Wed Feb 6 07:41:26 2019][1689386.596367] Lustre: 6182:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 12s req@ffff912d92ea8000 x1624705407799056/t0(0) o35->cd2fc9d9-2801-0b42-234d-619ca30bf3c7@10.8.9.8@o2ib6:0/0 lens 392/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 [Wed Feb 6 07:41:26 2019][1689386.620149] Lustre: 6182:0:(service.c:2011:ptlrpc_server_handle_req_in()) Skipped 9 previous similar messages [Wed Feb 6 07:41:29 2019][1689389.454006] Lustre: fir-OST0015-osc-MDT0000: Connection to fir-OST0015 (at 10.0.10.104@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 07:41:29 2019][1689389.470246] Lustre: Skipped 38 previous similar messages [Wed Feb 6 07:41:29 2019][1689389.476115] Lustre: 63246:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43:1s); client may timeout. req@ffff913a4533b900 x1624697938129280/t0(0) o103->62faecbe-6e06-a70f-dde9-8810f97cf707@10.9.103.17@o2ib4:101/0 lens 328/192 e 0 to 0 dl 1549467726 ref 2 fl Complete:H/0/0 rc 0/0 [Wed Feb 6 07:41:29 2019][1689389.505144] LustreError: 63248:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.103.17@o2ib4: deadline 43:1s ago [Wed Feb 6 07:41:29 2019][1689389.505144] req@ffff913a8f04aa00 x1624697938129504/t0(0) o103->62faecbe-6e06-a70f-dde9-8810f97cf707@10.9.103.17@o2ib4:101/0 lens 328/0 e 0 to 0 dl 1549467726 ref 2 fl Interpret:H/0/ffffffff rc 0/-1 [Wed Feb 6 07:41:29 2019][1689389.537502] LustreError: 63248:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 92 previous similar messages [Wed Feb 6 07:41:30 2019][1689390.114234] Lustre: 63239:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43:2s); client may timeout. req@ffff913a9935aa00 x1624697938129392/t0(0) o103->62faecbe-6e06-a70f-dde9-8810f97cf707@10.9.103.17@o2ib4:101/0 lens 328/192 e 0 to 0 dl 1549467726 ref 2 fl Complete:H/0/0 rc 0/0 [Wed Feb 6 07:41:30 2019][1689390.142866] Lustre: 63239:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 96 previous similar messages [Wed Feb 6 07:41:31 2019][1689391.472091] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 21s! [ldlm_cn01_028:28329] [Wed Feb 6 07:41:31 2019][1689391.480267] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE)[1689391.573091] NMI watchdog: BUG: soft lockup - CPU#17 stuck for 21s! [ldlm_cn01_023:28088] [Wed Feb 6 07:41:31 2019][1689391.573093] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:31 2019][1689391.573170] CPU: 17 PID: 28088 Comm: ldlm_cn01_023 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:31 2019][1689391.573170] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:32 2019][1689391.573172] task: ffff915929b4c100 ti: ffff91578b654000 task.ti: ffff91578b654000 [Wed Feb 6 07:41:32 2019][1689391.573173] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:41:32 2019][1689391.573182] RSP: 0018:ffff91578b657bb8 EFLAGS: 00000246 [Wed Feb 6 07:41:32 2019][1689391.573183] RAX: 0000000000000000 RBX: ffff91578b657bf0 RCX: 0000000000890000 [Wed Feb 6 07:41:32 2019][1689391.573184] RDX: ffff9129fee9b780 RSI: 0000000000410000 RDI: ffff911e813fa6dc [Wed Feb 6 07:41:32 2019][1689391.573185] RBP: ffff91578b657bb8 R08: ffff9139ff71b780 R09: 0000000000000000 [Wed Feb 6 07:41:32 2019][1689391.573185] R10: ffff911b3fc07600 R11: ffff913957f6a660 R12: 0000000000000000 [Wed Feb 6 07:41:32 2019][1689391.573186] R13: 0000000000000000 R14: 0000000000000002 R15: ffff91235e303cc0 [Wed Feb 6 07:41:32 2019][1689391.573188] FS: 00007f790e18c700(0000) GS:ffff9139ff700000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:32 2019][1689391.573189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:32 2019][1689391.573189] CR2: 00007f2781326000 CR3: 0000001028b2a000 CR4: 00000000003407e0 [Wed Feb 6 07:41:32 2019][1689391.573190] Call Trace: [Wed Feb 6 07:41:32 2019][1689391.573197] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:32 2019][1689391.573201] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:32 2019][1689391.573240] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689391.573273] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689391.573276] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:41:32 2019][1689391.573309] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689391.573346] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689391.573382] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689391.573418] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689391.573459] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689391.573498] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689391.573501] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:41:32 2019][1689391.573504] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:41:32 2019][1689391.573543] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689391.573581] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689391.573584] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:32 2019][1689391.573586] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:32 2019][1689391.573589] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:32 2019][1689391.573590] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:32 2019][1689391.573591] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:41:32 2019] [Wed Feb 6 07:41:32 2019][1689391.940236] tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:32 2019][1689391.955337] CPU: 5 PID: 28329 Comm: ldlm_cn01_028 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:32 2019][1689391.968011] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:32 2019][1689391.975753] task: ffff914aa94f30c0 ti: ffff9156bd79c000 task.ti: ffff9156bd79c000 [Wed Feb 6 07:41:32 2019][1689391.983404] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:41:32 2019][1689391.993510] RSP: 0018:ffff9156bd79fc38 EFLAGS: 00000246 [Wed Feb 6 07:41:32 2019][1689391.998996] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000290000 [Wed Feb 6 07:41:32 2019][1689392.006302] RDX: ffff915a3f5db780 RSI: 0000000000f90001 RDI: ffff911e813fa6dc [Wed Feb 6 07:41:32 2019][1689392.013610] RBP: ffff9156bd79fc38 R08: ffff9139ff65b780 R09: 0000000000000000 [Wed Feb 6 07:41:32 2019][1689392.020913] R10: ffff911b3fc07600 R11: ffff913957f6a660 R12: ffff9156bd79fbe0 [Wed Feb 6 07:41:32 2019][1689392.028220] R13: ffff91273b6849c0 R14: ffff9156bd79fba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:41:32 2019][1689392.035527] FS: 00007f63fb26e700(0000) GS:ffff9139ff640000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:32 2019][1689392.043786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:32 2019][1689392.049706] CR2: 00007f63fb343000 CR3: 000000103424c000 CR4: 00000000003407e0 [Wed Feb 6 07:41:32 2019][1689392.057013] Call Trace: [Wed Feb 6 07:41:32 2019][1689392.059643] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:32 2019][1689392.066082] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:32 2019][1689392.071773] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689392.078505] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689392.085243] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689392.092323] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689392.099230] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689392.106316] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689392.114150] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:41:32 2019][1689392.120594] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:41:32 2019][1689392.126639] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689392.133121] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:32 2019][1689392.140685] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:32 2019][1689392.145737] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:32 2019][1689392.152003] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:32 2019][1689392.158614] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:32 2019][1689392.164879] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:41:32 2019][1689392.335252] Lustre: 26053:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43:4s); client may timeout. req@ffff913a99359500 x1624697938129424/t0(0) o103->62faecbe-6e06-a70f-dde9-8810f97cf707@10.9.103.17@o2ib4:101/0 lens 328/192 e 0 to 0 dl 1549467726 ref 2 fl Complete:H/0/0 rc 0/0 [Wed Feb 6 07:41:32 2019][1689392.363899] Lustre: 26053:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [Wed Feb 6 07:41:34 2019][1689394.677830] Lustre: 63247:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43:6s); client may timeout. req@ffff913d287dda00 x1624697938127856/t0(0) o103->62faecbe-6e06-a70f-dde9-8810f97cf707@10.9.103.17@o2ib4:101/0 lens 328/192 e 0 to 0 dl 1549467726 ref 2 fl Complete:H/0/0 rc 0/0 [Wed Feb 6 07:41:35 2019][1689394.706467] Lustre: 63247:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2 previous similar messages [Wed Feb 6 07:41:36 2019][1689396.656915] Lustre: mdt_io: This server is not able to keep up with request traffic (cpu-bound). [Wed Feb 6 07:41:36 2019][1689396.665876] Lustre: 27672:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=0 reqQ=0 recA=3, svcEst=34, delay=6700 [Wed Feb 6 07:41:39 2019][1689399.564292] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [ldlm_cn00_024:20043] [Wed Feb 6 07:41:39 2019][1689399.572554] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:39 2019][1689399.679324] CPU: 16 PID: 20043 Comm: ldlm_cn00_024 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:39 2019][1689399.692087] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:39 2019][1689399.699827] task: ffff9139f700a080 ti: ffff9158ffab0000 task.ti: ffff9158ffab0000 [Wed Feb 6 07:41:39 2019][1689399.707482] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:41:39 2019][1689399.717596] RSP: 0018:ffff9158ffab3bb8 EFLAGS: 00000246 [Wed Feb 6 07:41:40 2019][1689399.720295] NMI watchdog: BUG: soft lockup - CPU#34 stuck for 22s! [ldlm_cn02_023:28934] [Wed Feb 6 07:41:40 2019][1689399.720342] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:40 2019][1689399.720363] CPU: 34 PID: 28934 Comm: ldlm_cn02_023 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:40 2019][1689399.720364] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:40 2019][1689399.720366] task: ffff91488911a080 ti: ffff9156a4e78000 task.ti: ffff9156a4e78000 [Wed Feb 6 07:41:40 2019][1689399.720374] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:41:40 2019][1689399.720376] RSP: 0000:ffff9156a4e7bc38 EFLAGS: 00000246 [Wed Feb 6 07:41:40 2019][1689399.720377] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001110000 [Wed Feb 6 07:41:40 2019][1689399.720378] RDX: ffff915a3f59b780 RSI: 0000000000d90000 RDI: ffff911e813fa6dc [Wed Feb 6 07:41:40 2019][1689399.720378] RBP: ffff9156a4e7bc38 R08: ffff9149ff81b780 R09: 0000000000000000 [Wed Feb 6 07:41:40 2019][1689399.720379] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: ffff9156a4e7bbe0 [Wed Feb 6 07:41:40 2019][1689399.720380] R13: ffff91273b6849c0 R14: ffff9156a4e7bba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:41:40 2019][1689399.720382] FS: 00007f7c8de93740(0000) GS:ffff9149ff800000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:40 2019][1689399.720382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:40 2019][1689399.720383] CR2: 00007f7c8d92f210 CR3: 00000020ceef2000 CR4: 00000000003407e0 [Wed Feb 6 07:41:40 2019][1689399.720384] Call Trace: [Wed Feb 6 07:41:40 2019][1689399.720391] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:40 2019][1689399.720395] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:40 2019][1689399.720440] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.720474] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.720513] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.720550] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.720586] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.720629] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.720670] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.720709] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.720749] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.720753] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:40 2019][1689399.720755] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:40 2019][1689399.720758] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:40 2019][1689399.720759] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:40 2019][1689399.720780] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:41:40 2019][1689399.808301] NMI watchdog: BUG: soft lockup - CPU#44 stuck for 23s! [ldlm_cn00_028:20558] [Wed Feb 6 07:41:40 2019][1689399.808350] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:40 2019][1689399.808372] CPU: 44 PID: 20558 Comm: ldlm_cn00_028 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:40 2019][1689399.808372] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:40 2019][1689399.808374] task: ffff9146d2b8a080 ti: ffff9159f5ffc000 task.ti: ffff9159f5ffc000 [Wed Feb 6 07:41:40 2019][1689399.808383] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:41:40 2019][1689399.808384] RSP: 0018:ffff9159f5fffbb8 EFLAGS: 00000246 [Wed Feb 6 07:41:40 2019][1689399.808385] RAX: 0000000000000000 RBX: ffff9159f5fffbf0 RCX: 0000000001610000 [Wed Feb 6 07:41:40 2019][1689399.808386] RDX: ffff9139ff75b780 RSI: 0000000000a90000 RDI: ffff911e813fa6dc [Wed Feb 6 07:41:40 2019][1689399.808387] RBP: ffff9159f5fffbb8 R08: ffff9129ff0db780 R09: 0000000000000000 [Wed Feb 6 07:41:40 2019][1689399.808387] R10: ffff911b3fc07600 R11: ffffdbb24dcd6c00 R12: 0000000000000000 [Wed Feb 6 07:41:40 2019][1689399.808388] R13: 0000000000000000 R14: 0000000000000002 R15: ffff91361d74f980 [Wed Feb 6 07:41:40 2019][1689399.808389] FS: 00007f63fb26e700(0000) GS:ffff9129ff0c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:40 2019][1689399.808390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:40 2019][1689399.808391] CR2: 00007f01048dc1a0 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:41:40 2019][1689399.808392] Call Trace: [Wed Feb 6 07:41:40 2019][1689399.808399] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:40 2019][1689399.808403] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:40 2019][1689399.808447] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.808482] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.808484] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:41:40 2019][1689399.808517] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.808555] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.808592] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.808628] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.808672] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.808712] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.808752] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.808792] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689399.808795] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:40 2019][1689399.808797] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:40 2019][1689399.808800] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:40 2019][1689399.808802] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:40 2019][1689399.808822] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:41:40 2019][1689400.100411] Lustre: 23311:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43:12s); client may timeout. req@ffff913d365c5400 x1624697938127920/t0(0) o103->62faecbe-6e06-a70f-dde9-8810f97cf707@10.9.103.17@o2ib4:101/0 lens 328/192 e 0 to 0 dl 1549467726 ref 2 fl Complete:H/0/0 rc 0/0 [Wed Feb 6 07:41:40 2019][1689400.100415] Lustre: 23311:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2 previous similar messages [Wed Feb 6 07:41:40 2019][1689400.446854] RAX: 0000000000000000 RBX: ffff9158ffab3bf0 RCX: 0000000000810000 [Wed Feb 6 07:41:40 2019][1689400.454159] RDX: ffff9149ff81b780 RSI: 0000000001110000 RDI: ffff911e813fa6dc [Wed Feb 6 07:41:40 2019][1689400.461464] RBP: ffff9158ffab3bb8 R08: ffff9129fef1b780 R09: 0000000000000000 [Wed Feb 6 07:41:40 2019][1689400.468770] R10: ffff911b3fc07600 R11: ffffdbb268b18400 R12: 0000000000000000 [Wed Feb 6 07:41:40 2019][1689400.476076] R13: 0000000000000000 R14: 0000000000000002 R15: ffff912056bb2f40 [Wed Feb 6 07:41:40 2019][1689400.483383] FS: 00007fe2d3435880(0000) GS:ffff9129fef00000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:40 2019][1689400.491644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:40 2019][1689400.497563] CR2: 00007fe2d0ba8050 CR3: 000000403a6c6000 CR4: 00000000003407e0 [Wed Feb 6 07:41:40 2019][1689400.504868] Call Trace: [Wed Feb 6 07:41:40 2019][1689400.507503] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:40 2019][1689400.513948] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:40 2019][1689400.519658] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689400.526388] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689400.533436] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:41:40 2019][1689400.541032] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689400.547765] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689400.554843] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689400.561749] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689400.568837] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689400.576704] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689400.583699] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689400.590182] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:40 2019][1689400.597756] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:40 2019][1689400.602806] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:40 2019][1689400.609072] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:40 2019][1689400.615685] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:40 2019][1689400.621949] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:41:43 2019][1689403.423616] LustreError: 6694:0:(upcall_cache.c:233:upcall_cache_get_entry()) acquire for key 207393: error -110 [Wed Feb 6 07:41:43 2019][1689403.480388] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [ldlm_cn02_015:23318] [Wed Feb 6 07:41:43 2019][1689403.488565] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:43 2019][1689403.595337] CPU: 6 PID: 23318 Comm: ldlm_cn02_015 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:43 2019][1689403.608012] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:43 2019][1689403.615752] task: ffff9159e661b0c0 ti: ffff915853bac000 task.ti: ffff915853bac000 [Wed Feb 6 07:41:43 2019][1689403.623406] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:41:43 2019][1689403.633512] RSP: 0018:ffff915853bafc38 EFLAGS: 00000246 [Wed Feb 6 07:41:43 2019][1689403.638996] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000310000 [Wed Feb 6 07:41:43 2019][1689403.646303] RDX: ffff915a3f41b780 RSI: 0000000000190001 RDI: ffff911e813fa6dc [Wed Feb 6 07:41:43 2019][1689403.653608] RBP: ffff915853bafc38 R08: ffff9149ff65b780 R09: 0000000000000000 [Wed Feb 6 07:41:43 2019][1689403.660916] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: ffff915853bafbe0 [Wed Feb 6 07:41:43 2019][1689403.668222] R13: ffff91273b6849c0 R14: ffff915853bafba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:41:43 2019][1689403.675527] FS: 00007f277d7c4700(0000) GS:ffff9149ff640000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:43 2019][1689403.683788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:43 2019][1689403.689706] CR2: 00007fc91a77880d CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:41:43 2019][1689403.697014] Call Trace: [Wed Feb 6 07:41:43 2019][1689403.699644] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:43 2019][1689403.706090] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:44 2019][1689403.711788] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:44 2019][1689403.718522] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:41:44 2019][1689403.725261] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:44 2019][1689403.732340] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:44 2019][1689403.739247] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:44 2019][1689403.746335] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:44 2019][1689403.754201] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:41:44 2019][1689403.761463] [] ? ptlrpc_nrs_req_add_nolock+0x6a/0x1d0 [ptlrpc] [Wed Feb 6 07:41:44 2019][1689403.769152] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:44 2019][1689403.775624] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:44 2019][1689403.783190] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:44 2019][1689403.788240] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:44 2019][1689403.794509] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:44 2019][1689403.801121] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:44 2019][1689403.807384] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:41:45 2019][1689405.439369] Lustre: MGS: Received new LWP connection from 10.9.105.31@o2ib4, removing former export from same NID [Wed Feb 6 07:41:45 2019][1689405.449798] Lustre: Skipped 2 previous similar messages [Wed Feb 6 07:41:47 2019][1689406.900707] Lustre: MGS: Received new LWP connection from 10.9.103.27@o2ib4, removing former export from same NID [Wed Feb 6 07:41:47 2019][1689406.911141] Lustre: Skipped 6 previous similar messages [Wed Feb 6 07:41:47 2019][1689407.641493] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 21s! [ldlm_cn01_031:28334] [Wed Feb 6 07:41:47 2019][1689407.649753] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:48 2019][1689407.756524] CPU: 25 PID: 28334 Comm: ldlm_cn01_031 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:48 2019][1689407.769286] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:48 2019][1689407.777026] task: ffff9159aa3db0c0 ti: ffff9156ee634000 task.ti: ffff9156ee634000 [Wed Feb 6 07:41:48 2019][1689407.784681] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x128/0x200 [Wed Feb 6 07:41:48 2019][1689407.794793] RSP: 0018:ffff9156ee637bb8 EFLAGS: 00000246 [Wed Feb 6 07:41:48 2019][1689407.800281] RAX: 0000000000000000 RBX: ffff9156ee637bf0 RCX: 0000000000c90000 [Wed Feb 6 07:41:48 2019][1689407.807587] RDX: ffff9129fef1b780 RSI: 0000000000810000 RDI: ffff911e813fa6dc [Wed Feb 6 07:41:48 2019][1689407.814893] RBP: ffff9156ee637bb8 R08: ffff9139ff79b780 R09: 0000000000000000 [Wed Feb 6 07:41:48 2019][1689407.816499] NMI watchdog: BUG: soft lockup - CPU#45 stuck for 24s! [ldlm_cn01_013:19974] [Wed Feb 6 07:41:48 2019][1689407.816546] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:48 2019][1689407.816568] CPU: 45 PID: 19974 Comm: ldlm_cn01_013 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:48 2019][1689407.816569] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:48 2019][1689407.816570] task: ffff9129ee3ca080 ti: ffff91590ceb0000 task.ti: ffff91590ceb0000 [Wed Feb 6 07:41:48 2019][1689407.816579] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:41:48 2019][1689407.816580] RSP: 0018:ffff91590ceb3bb8 EFLAGS: 00000246 [Wed Feb 6 07:41:48 2019][1689407.816581] RAX: 0000000000000000 RBX: ffff91590ceb3bf0 RCX: 0000000001690000 [Wed Feb 6 07:41:48 2019][1689407.816582] RDX: ffff9149ff6db780 RSI: 0000000000710000 RDI: ffff911e813fa6dc [Wed Feb 6 07:41:48 2019][1689407.816583] RBP: ffff91590ceb3bb8 R08: ffff9139ff8db780 R09: 0000000000000000 [Wed Feb 6 07:41:48 2019][1689407.816584] R10: ffff911b3fc07600 R11: ffffdbb27e59c200 R12: 0000000000000000 [Wed Feb 6 07:41:48 2019][1689407.816584] R13: 0000000000000000 R14: 0000000000000002 R15: ffff911d1d5fe9c0 [Wed Feb 6 07:41:48 2019][1689407.816586] FS: 00007f63fb28f700(0000) GS:ffff9139ff8c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:48 2019][1689407.816587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:48 2019][1689407.816588] CR2: 00007f63fb343000 CR3: 000000103424c000 CR4: 00000000003407e0 [Wed Feb 6 07:41:48 2019][1689407.816589] Call Trace: [Wed Feb 6 07:41:48 2019][1689407.816595] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:48 2019][1689407.816599] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:48 2019][1689407.816637] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689407.816671] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689407.816674] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:41:48 2019][1689407.816707] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689407.816745] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689407.816782] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689407.816817] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689407.816860] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689407.816900] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689407.816940] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689407.816979] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689407.816982] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:48 2019][1689407.816984] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:48 2019][1689407.816987] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:48 2019][1689407.816989] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:48 2019][1689407.817009] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:41:48 2019][1689408.171777] R10: ffff911b3fc07600 R11: ffffdbb27e59c200 R12: 0000000000000000 [Wed Feb 6 07:41:48 2019][1689408.179084] R13: 0000000000000000 R14: 0000000000000002 R15: ffff913d170a1200 [Wed Feb 6 07:41:48 2019][1689408.186389] FS: 00007fc9e2d79880(0000) GS:ffff9139ff780000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:48 2019][1689408.194649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:48 2019][1689408.200570] CR2: 00007f63fb343000 CR3: 000000303a262000 CR4: 00000000003407e0 [Wed Feb 6 07:41:48 2019][1689408.207875] Call Trace: [Wed Feb 6 07:41:48 2019][1689408.210507] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:48 2019][1689408.216953] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:48 2019][1689408.222648] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689408.229378] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689408.236425] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:41:48 2019][1689408.244023] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689408.250763] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689408.257841] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689408.264749] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689408.271837] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689408.279702] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689408.286695] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689408.293169] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:48 2019][1689408.300737] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:48 2019][1689408.305786] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:48 2019][1689408.312052] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:48 2019][1689408.318665] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:48 2019][1689408.324930] Code: 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 85 c0 <74> f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 [Wed Feb 6 07:41:51 2019][1689411.437588] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_cn01_001:6033] [Wed Feb 6 07:41:51 2019][1689411.445675] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:51 2019][1689411.552446] CPU: 1 PID: 6033 Comm: ldlm_cn01_001 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:51 2019][1689411.565036] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:51 2019][1689411.572777] task: ffff9129f3d24100 ti: ffff9159f9abc000 task.ti: ffff9159f9abc000 [Wed Feb 6 07:41:51 2019][1689411.580428] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:41:51 2019][1689411.590534] RSP: 0018:ffff9159f9abfbb8 EFLAGS: 00000246 [Wed Feb 6 07:41:51 2019][1689411.596020] RAX: 0000000000000000 RBX: ffff9159f9abfbf0 RCX: 0000000000090000 [Wed Feb 6 07:41:51 2019][1689411.603327] RDX: ffff9129fefdb780 RSI: 0000000000310000 RDI: ffff911e813fa6dc [Wed Feb 6 07:41:51 2019][1689411.610632] RBP: ffff9159f9abfbb8 R08: ffff9139ff61b780 R09: 0000000000000000 [Wed Feb 6 07:41:51 2019][1689411.617938] R10: ffff911b3fc07600 R11: ffffdbb279bf3800 R12: 0000000000000000 [Wed Feb 6 07:41:51 2019][1689411.625245] R13: 0000000000000000 R14: 0000000000000002 R15: ffff91326149de80 [Wed Feb 6 07:41:51 2019][1689411.632552] FS: 00007f63fb2d1700(0000) GS:ffff9139ff600000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:51 2019][1689411.640810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:51 2019][1689411.646731] CR2: 00007f63fb343000 CR3: 000000103424c000 CR4: 00000000003407e0 [Wed Feb 6 07:41:51 2019][1689411.654036] Call Trace: [Wed Feb 6 07:41:51 2019][1689411.656668] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:51 2019][1689411.663106] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:51 2019][1689411.668596] NMI watchdog: BUG: soft lockup - CPU#28 stuck for 21s! [ldlm_cn00_025:20158] [Wed Feb 6 07:41:51 2019][1689411.668645] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:52 2019][1689411.668665] CPU: 28 PID: 20158 Comm: ldlm_cn00_025 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:52 2019][1689411.668666] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:52 2019][1689411.668668] task: ffff9159f1f62080 ti: ffff9158f738c000 task.ti: ffff9158f738c000 [Wed Feb 6 07:41:52 2019][1689411.668676] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:41:52 2019][1689411.668677] RSP: 0018:ffff9158f738fbb8 EFLAGS: 00000246 [Wed Feb 6 07:41:52 2019][1689411.668678] RAX: 0000000000000000 RBX: ffff9158f738fbf0 RCX: 0000000000e10000 [Wed Feb 6 07:41:52 2019][1689411.668679] RDX: ffff9149ff65b780 RSI: 0000000000310000 RDI: ffff911e813fa6dc [Wed Feb 6 07:41:52 2019][1689411.668680] RBP: ffff9158f738fbb8 R08: ffff9129fefdb780 R09: 0000000000000000 [Wed Feb 6 07:41:52 2019][1689411.668681] R10: ffff911b3fc07600 R11: ffffdbb2462ae000 R12: 0000000000000000 [Wed Feb 6 07:41:52 2019][1689411.668681] R13: 0000000000000000 R14: 0000000000000002 R15: ffff913b516faf40 [Wed Feb 6 07:41:52 2019][1689411.668683] FS: 00007f6417cd6700(0000) GS:ffff9129fefc0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:52 2019][1689411.668684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:52 2019][1689411.668685] CR2: 00007f6410016068 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:41:52 2019][1689411.668686] Call Trace: [Wed Feb 6 07:41:52 2019][1689411.668692] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:52 2019][1689411.668695] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:52 2019][1689411.668740] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.668774] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.668779] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:41:52 2019][1689411.668812] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.668851] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.668888] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.668923] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.668968] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.669008] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.669047] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.669087] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.669091] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:52 2019][1689411.669093] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:52 2019][1689411.669096] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:52 2019][1689411.669098] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:52 2019][1689411.669118] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:41:52 2019][1689411.781595] NMI watchdog: BUG: soft lockup - CPU#41 stuck for 22s! [ldlm_cn01_005:6768] [Wed Feb 6 07:41:52 2019][1689411.781626] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:52 2019][1689411.781639] CPU: 41 PID: 6768 Comm: ldlm_cn01_005 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:52 2019][1689411.781640] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:52 2019][1689411.781641] task: ffff9159f33cd140 ti: ffff9159f2724000 task.ti: ffff9159f2724000 [Wed Feb 6 07:41:52 2019][1689411.781644] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:41:52 2019][1689411.781645] RSP: 0018:ffff9159f2727c38 EFLAGS: 00000246 [Wed Feb 6 07:41:52 2019][1689411.781646] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001490000 [Wed Feb 6 07:41:52 2019][1689411.781647] RDX: ffff9139ff6db780 RSI: 0000000000690001 RDI: ffff911e813fa6dc [Wed Feb 6 07:41:52 2019][1689411.781647] RBP: ffff9159f2727c38 R08: ffff9139ff89b780 R09: 0000000000000000 [Wed Feb 6 07:41:52 2019][1689411.781648] R10: ffff911b3fc07600 R11: ffffdbb2f14ef400 R12: ffff9159f2727be0 [Wed Feb 6 07:41:52 2019][1689411.781649] R13: ffff91273b6849c0 R14: ffff9159f2727ba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:41:52 2019][1689411.781650] FS: 00007fe121ad2740(0000) GS:ffff9139ff880000(0000) knlGS:0000000000000000 [Wed Feb 6 07:41:52 2019][1689411.781651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:41:52 2019][1689411.781652] CR2: 0000000001111108 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:41:52 2019][1689411.781652] Call Trace: [Wed Feb 6 07:41:52 2019][1689411.781655] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:41:52 2019][1689411.781657] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:41:52 2019][1689411.781698] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.781731] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.781768] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.781804] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.781839] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.781881] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.781920] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.781958] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.781996] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689411.782000] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:52 2019][1689411.782002] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:52 2019][1689411.782004] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:52 2019][1689411.782006] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:52 2019][1689411.782025] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:41:52 2019][1689412.353182] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689412.359909] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689412.366959] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:41:52 2019][1689412.374555] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689412.381292] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689412.388375] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689412.395280] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689412.402366] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689412.410215] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Wed Feb 6 07:41:52 2019][1689412.417557] [] ? ptlrpc_at_add_timed+0xe5/0x230 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689412.424724] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689412.431199] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:41:52 2019][1689412.438764] [] kthread+0xd1/0xe0 [Wed Feb 6 07:41:52 2019][1689412.443817] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:52 2019][1689412.450083] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:41:52 2019][1689412.456696] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:41:52 2019][1689412.462959] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:41:53 2019][1689413.067692] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 13 seconds [Wed Feb 6 07:41:53 2019][1689413.077861] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 3 previous similar messages [Wed Feb 6 07:41:53 2019][1689413.088117] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.201@o2ib7 (18): c: 0, oc: 1, rc: 8 [Wed Feb 6 07:41:53 2019][1689413.100281] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 3 previous similar messages [Wed Feb 6 07:41:53 2019][1689413.110112] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.101@o2ib7: 1689369 seconds [Wed Feb 6 07:41:53 2019][1689413.120882] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 23 previous similar messages [Wed Feb 6 07:41:58 2019][1689418.700585] Lustre: 6047:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 10s req@ffff913a92ffb850 x1624672980222656/t0(0) o400->e81ecd90-30b3-5dd2-29d5-2e0ca7d2766a@10.9.101.46@o2ib4:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 [Wed Feb 6 07:41:59 2019][1689418.725696] Lustre: MGS: Received new LWP connection from 10.9.101.32@o2ib4, removing former export from same NID [Wed Feb 6 07:41:59 2019][1689419.472793] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [ldlm_cn01_028:28329] [Wed Feb 6 07:41:59 2019][1689419.480972] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE)[1689419.573794] NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [ldlm_cn01_023:28088] [Wed Feb 6 07:41:59 2019][1689419.573795] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:41:59 2019][1689419.573858] CPU: 17 PID: 28088 Comm: ldlm_cn01_023 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:41:59 2019][1689419.573859] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:41:59 2019][1689419.573860] task: ffff915929b4c100 ti: ffff91578b654000 task.ti: ffff91578b654000 [Wed Feb 6 07:41:59 2019][1689419.573861] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:00 2019][1689419.573868] RSP: 0018:ffff91578b657bb8 EFLAGS: 00000246 [Wed Feb 6 07:42:00 2019][1689419.573868] RAX: 0000000000000000 RBX: ffff91578b657bf0 RCX: 0000000000890000 [Wed Feb 6 07:42:00 2019][1689419.573869] RDX: ffff9129ff01b780 RSI: 0000000001010000 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:00 2019][1689419.573870] RBP: ffff91578b657bb8 R08: ffff9139ff71b780 R09: 0000000000000000 [Wed Feb 6 07:42:00 2019][1689419.573871] R10: ffff911b3fc07600 R11: ffff913957f6a660 R12: 0000000000000000 [Wed Feb 6 07:42:00 2019][1689419.573871] R13: 0000000000000000 R14: 0000000000000002 R15: ffff913ca23b1f80 [Wed Feb 6 07:42:00 2019][1689419.573873] FS: 00007f790e18c700(0000) GS:ffff9139ff700000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:00 2019][1689419.573874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:00 2019][1689419.573874] CR2: 00007f2781326000 CR3: 0000001028b2a000 CR4: 00000000003407e0 [Wed Feb 6 07:42:00 2019][1689419.573875] Call Trace: [Wed Feb 6 07:42:00 2019][1689419.573880] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:00 2019][1689419.573883] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:00 2019][1689419.573920] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.573953] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.573955] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:00 2019][1689419.573987] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.574024] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.574059] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.574094] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.574133] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.574171] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.574174] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:00 2019][1689419.574177] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:00 2019][1689419.574214] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.574252] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.574255] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:00 2019][1689419.574256] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:00 2019][1689419.574259] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:00 2019][1689419.574260] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:00 2019][1689419.574261] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:00 2019][1689419.624793] NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ldlm_cn03_000:6038] [Wed Feb 6 07:42:00 2019][1689419.624794] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:00 2019][1689419.624875] CPU: 23 PID: 6038 Comm: ldlm_cn03_000 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:00 2019][1689419.624876] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:00 2019][1689419.624877] task: ffff9129f3049040 ti: ffff9159a86e0000 task.ti: ffff9159a86e0000 [Wed Feb 6 07:42:00 2019][1689419.624878] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:00 2019][1689419.624885] RSP: 0018:ffff9159a86e3c38 EFLAGS: 00000246 [Wed Feb 6 07:42:00 2019][1689419.624886] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000b90000 [Wed Feb 6 07:42:00 2019][1689419.624887] RDX: ffff915a3f59b780 RSI: 0000000000d90001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:00 2019][1689419.624888] RBP: ffff9159a86e3c38 R08: ffff915a3f55b780 R09: 0000000000000000 [Wed Feb 6 07:42:00 2019][1689419.624889] R10: ffff911b3fc07600 R11: ffffdbb283493e00 R12: ffff9159a86e3be0 [Wed Feb 6 07:42:00 2019][1689419.624889] R13: ffff91273b6849c0 R14: ffff9159a86e3ba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:42:00 2019][1689419.624891] FS: 00007fe2d3435880(0000) GS:ffff915a3f540000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:00 2019][1689419.624892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:00 2019][1689419.624893] CR2: 00007fe2d0bac050 CR3: 000000403a6c6000 CR4: 00000000003407e0 [Wed Feb 6 07:42:00 2019][1689419.624894] Call Trace: [Wed Feb 6 07:42:00 2019][1689419.624899] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:00 2019][1689419.624902] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:00 2019][1689419.624945] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.624979] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.625018] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.625055] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.625090] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.625134] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.625174] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.625214] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.625253] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.625256] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:00 2019][1689419.625259] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:00 2019][1689419.625261] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:00 2019][1689419.625263] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:00 2019][1689419.625264] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:00 2019][1689419.694797] NMI watchdog: BUG: soft lockup - CPU#31 stuck for 21s! [ldlm_cn03_022:26176] [Wed Feb 6 07:42:00 2019][1689419.694797] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:00 2019][1689419.694844] CPU: 31 PID: 26176 Comm: ldlm_cn03_022 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:00 2019][1689419.694845] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:00 2019][1689419.694846] task: ffff9129f375d140 ti: ffff911f093c0000 task.ti: ffff911f093c0000 [Wed Feb 6 07:42:00 2019][1689419.694847] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:00 2019][1689419.694852] RSP: 0018:ffff911f093c3bb8 EFLAGS: 00000246 [Wed Feb 6 07:42:00 2019][1689419.694853] RAX: 0000000000000000 RBX: ffff911f093c3bf0 RCX: 0000000000f90000 [Wed Feb 6 07:42:00 2019][1689419.694853] RDX: ffff9139ff81b780 RSI: 0000000001090000 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:00 2019][1689419.694854] RBP: ffff911f093c3bb8 R08: ffff915a3f5db780 R09: 0000000000000000 [Wed Feb 6 07:42:00 2019][1689419.694855] R10: ffff911b3fc07600 R11: 0000000000000000 R12: 0000000000000000 [Wed Feb 6 07:42:00 2019][1689419.694855] R13: 0000000000000000 R14: 0000000000000002 R15: ffff914064e9af40 [Wed Feb 6 07:42:00 2019][1689419.694857] FS: 00007f0ff898a880(0000) GS:ffff915a3f5c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:00 2019][1689419.694858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:00 2019][1689419.694859] CR2: 00007f9200fd81b0 CR3: 000000202de1c000 CR4: 00000000003407e0 [Wed Feb 6 07:42:00 2019][1689419.694860] Call Trace: [Wed Feb 6 07:42:00 2019][1689419.694863] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:00 2019][1689419.694865] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:00 2019][1689419.694898] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.694932] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.694935] [] ? native_queued_spin_lock_slowpath+0x15e/0x200 [Wed Feb 6 07:42:00 2019][1689419.694968] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.695004] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.695041] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.695076] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.695118] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.695158] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.695197] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.695237] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:00 2019][1689419.695239] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:00 2019][1689419.695241] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:00 2019][1689419.695243] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:00 2019][1689419.695244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:00 2019][1689419.695245] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:00 2019][1689419.730800] NMI watchdog: BUG: soft lockup - CPU#35 stuck for 21s! [ldlm_cn03_018:18063] [Wed Feb 6 07:42:00 2019][1689419.730801] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:01 2019][1689419.730880] CPU: 35 PID: 18063 Comm: ldlm_cn03_018 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:01 2019][1689419.730881] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:01 2019][1689419.730882] task: ffff912fe0b62080 ti: ffff9153b0674000 task.ti: ffff9153b0674000 [Wed Feb 6 07:42:01 2019][1689419.730883] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:01 2019][1689419.730890] RSP: 0018:ffff9153b0677bb8 EFLAGS: 00000246 [Wed Feb 6 07:42:01 2019][1689419.730891] RAX: 0000000000000000 RBX: ffff9153b0677bf0 RCX: 0000000001190000 [Wed Feb 6 07:42:01 2019][1689419.730891] RDX: ffff9149ff65b780 RSI: 0000000000310000 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:01 2019][1689419.730892] RBP: ffff9153b0677bb8 R08: ffff915a3f61b780 R09: 0000000000000000 [Wed Feb 6 07:42:01 2019][1689419.730893] R10: 0000000000000096 R11: 000000005c5b0047 R12: 0000000000000000 [Wed Feb 6 07:42:01 2019][1689419.730894] R13: 0000000000000000 R14: 0000000000000002 R15: ffff9123098dba80 [Wed Feb 6 07:42:01 2019][1689419.730895] FS: 00007f277afbf700(0000) GS:ffff915a3f600000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:01 2019][1689419.730896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:01 2019][1689419.730897] CR2: 00007f2781327000 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:42:01 2019][1689419.730898] Call Trace: [Wed Feb 6 07:42:01 2019][1689419.730902] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:01 2019][1689419.730905] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:01 2019][1689419.730949] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.730983] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.730986] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:01 2019][1689419.731021] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.731062] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.731104] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.731144] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.731188] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.731229] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.731270] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.731311] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.731313] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:01 2019][1689419.731315] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:01 2019][1689419.731318] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:01 2019][1689419.731319] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:01 2019][1689419.731320] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:01 2019][1689419.764796] NMI watchdog: BUG: soft lockup - CPU#39 stuck for 22s! [ldlm_cn03_011:23312] [Wed Feb 6 07:42:01 2019][1689419.764797] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:01 2019][1689419.764845] CPU: 39 PID: 23312 Comm: ldlm_cn03_011 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:01 2019][1689419.764846] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:01 2019][1689419.764847] task: ffff9159a4beb0c0 ti: ffff915929b50000 task.ti: ffff915929b50000 [Wed Feb 6 07:42:01 2019][1689419.764848] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:01 2019][1689419.764852] RSP: 0018:ffff915929b53bb8 EFLAGS: 00000246 [Wed Feb 6 07:42:01 2019][1689419.764853] RAX: 0000000000000000 RBX: ffff915929b53bf0 RCX: 0000000001390000 [Wed Feb 6 07:42:01 2019][1689419.764854] RDX: ffff915a3f59b780 RSI: 0000000000d90001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:01 2019][1689419.764855] RBP: ffff915929b53bb8 R08: ffff915a3f65b780 R09: 0000000000000000 [Wed Feb 6 07:42:01 2019][1689419.764856] R10: ffff911b3fc07600 R11: 0000000000000000 R12: 0000000000000000 [Wed Feb 6 07:42:01 2019][1689419.764856] R13: 0000000000000000 R14: 0000000000000002 R15: ffff91524d2269c0 [Wed Feb 6 07:42:01 2019][1689419.764858] FS: 00007f72eaa7c700(0000) GS:ffff915a3f640000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:01 2019][1689419.764859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:01 2019][1689419.764860] CR2: 00007f2781328000 CR3: 000000403bd1e000 CR4: 00000000003407e0 [Wed Feb 6 07:42:01 2019][1689419.764861] Call Trace: [Wed Feb 6 07:42:01 2019][1689419.764864] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:01 2019][1689419.764866] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:01 2019][1689419.764899] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.764933] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.764935] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:01 2019][1689419.764968] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.765005] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.765041] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.765077] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.765118] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.765158] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.765198] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.765237] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689419.765239] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:01 2019][1689419.765240] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:01 2019][1689419.765242] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:01 2019][1689419.765244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:01 2019][1689419.765245] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:42:01 2019] [Wed Feb 6 07:42:01 2019][1689421.342878] tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:01 2019][1689421.357979] CPU: 5 PID: 28329 Comm: ldlm_cn01_028 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:01 2019][1689421.370654] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:01 2019][1689421.378395] task: ffff914aa94f30c0 ti: ffff9156bd79c000 task.ti: ffff9156bd79c000 [Wed Feb 6 07:42:01 2019][1689421.386048] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:01 2019][1689421.396162] RSP: 0018:ffff9156bd79fc38 EFLAGS: 00000246 [Wed Feb 6 07:42:01 2019][1689421.401648] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000290000 [Wed Feb 6 07:42:01 2019][1689421.408955] RDX: ffff9149ff6db780 RSI: 0000000000710001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:01 2019][1689421.416261] RBP: ffff9156bd79fc38 R08: ffff9139ff65b780 R09: 0000000000000000 [Wed Feb 6 07:42:01 2019][1689421.423566] R10: ffff911b3fc07600 R11: ffff913957f6a660 R12: ffff9156bd79fbe0 [Wed Feb 6 07:42:01 2019][1689421.430873] R13: ffff91273b6849c0 R14: ffff9156bd79fba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:42:01 2019][1689421.438179] FS: 00007f63fb26e700(0000) GS:ffff9139ff640000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:01 2019][1689421.446438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:01 2019][1689421.452357] CR2: 00007f63fb343000 CR3: 000000103424c000 CR4: 00000000003407e0 [Wed Feb 6 07:42:01 2019][1689421.459666] Call Trace: [Wed Feb 6 07:42:01 2019][1689421.462298] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:01 2019][1689421.468743] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:01 2019][1689421.474438] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689421.481173] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689421.487912] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689421.494991] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689421.501898] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689421.508986] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689421.516819] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:01 2019][1689421.523254] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:01 2019][1689421.529299] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689421.535773] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:01 2019][1689421.543339] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:01 2019][1689421.548388] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:01 2019][1689421.554656] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:01 2019][1689421.561266] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:01 2019][1689421.567531] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:02 2019][1689421.807953] Lustre: fir-OST000f-osc-MDT0000: Connection to fir-OST000f (at 10.0.10.104@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 07:42:02 2019][1689421.824191] Lustre: Skipped 36 previous similar messages [Wed Feb 6 07:42:03 2019][1689422.977865] LustreError: 27461:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(131072) req@ffff9159a0615450 x1624699200490272/t0(0) o4->dd23da3a-484d-0eb4-af89-89a069ff0621@10.9.104.49@o2ib4:142/0 lens 488/448 e 1 to 0 dl 1549467767 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 07:42:03 2019][1689423.002981] Lustre: fir-MDT0002: Bulk IO write error with dd23da3a-484d-0eb4-af89-89a069ff0621 (at 10.9.104.49@o2ib4), client will retry: rc = -110 [Wed Feb 6 07:42:03 2019][1689423.016362] Lustre: Skipped 5 previous similar messages [Wed Feb 6 07:42:04 2019][1689423.790898] NMI watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [ldlm_cn02_010:9990] [Wed Feb 6 07:42:04 2019][1689423.799072] Modules linked in:[1689423.800901] NMI watchdog: BUG: soft lockup - CPU#43 stuck for 24s! [ldlm_cn03_013:59873] [Wed Feb 6 07:42:04 2019][1689423.800902] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:04 2019][1689423.800977] CPU: 43 PID: 59873 Comm: ldlm_cn03_013 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:04 2019][1689423.800978] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:04 2019][1689423.800979] task: ffff91383a610000 ti: ffff9142785e4000 task.ti: ffff9142785e4000 [Wed Feb 6 07:42:04 2019][1689423.800981] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x156/0x200 [Wed Feb 6 07:42:04 2019][1689423.800988] RSP: 0018:ffff9142785e7c38 EFLAGS: 00000202 [Wed Feb 6 07:42:04 2019][1689423.800988] RAX: 0000000000000001 RBX: 00000001c0e40fd2 RCX: 0000000001590000 [Wed Feb 6 07:42:04 2019][1689423.800989] RDX: 0000000001210001 RSI: 0000000000010001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:04 2019][1689423.800990] RBP: ffff9142785e7c38 R08: ffff915a3f69b780 R09: ffff9139ff81b780 [Wed Feb 6 07:42:04 2019][1689423.800991] R10: ffff911b3fc07600 R11: ffffdbb279f74e00 R12: ffff9142785e7be0 [Wed Feb 6 07:42:04 2019][1689423.800991] R13: ffff91273b6849c0 R14: ffff9142785e7ba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:42:04 2019][1689423.800993] FS: 00007f2b3eadb740(0000) GS:ffff915a3f680000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:04 2019][1689423.800994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:04 2019][1689423.800994] CR2: 00007f2781327000 CR3: 0000004031fb0000 CR4: 00000000003407e0 [Wed Feb 6 07:42:04 2019][1689423.800996] Call Trace: [Wed Feb 6 07:42:04 2019][1689423.801000] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:04 2019][1689423.801003] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:04 2019][1689423.801045] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.801079] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.801117] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.801153] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.801188] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.801230] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.801269] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.801308] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.801346] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.801349] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:04 2019][1689423.801351] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:04 2019][1689423.801354] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:04 2019][1689423.801355] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:04 2019][1689423.801356] Code: 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 <8b> 17 0f b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84 [Wed Feb 6 07:42:04 2019][1689423.825904] NMI watchdog: BUG: soft lockup - CPU#46 stuck for 21s! [ldlm_cn02_016:23319] [Wed Feb 6 07:42:04 2019][1689423.825905] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:04 2019][1689423.825979] CPU: 46 PID: 23319 Comm: ldlm_cn02_016 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:04 2019][1689423.825980] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:04 2019][1689423.825982] task: ffff9159e6619040 ti: ffff9158f7338000 task.ti: ffff9158f7338000 [Wed Feb 6 07:42:04 2019][1689423.825983] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:04 2019][1689423.825992] RSP: 0018:ffff9158f733bc38 EFLAGS: 00000246 [Wed Feb 6 07:42:04 2019][1689423.825993] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001710000 [Wed Feb 6 07:42:04 2019][1689423.825994] RDX: ffff9129fef5b780 RSI: 0000000000a10001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:04 2019][1689423.825995] RBP: ffff9158f733bc38 R08: ffff9149ff8db780 R09: 0000000000000000 [Wed Feb 6 07:42:04 2019][1689423.825996] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: ffff9158f733bbe0 [Wed Feb 6 07:42:04 2019][1689423.825996] R13: ffff91273b6849c0 R14: ffff9158f733bba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:42:04 2019][1689423.825998] FS: 00007f78f341b880(0000) GS:ffff9149ff8c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:04 2019][1689423.825999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:04 2019][1689423.825999] CR2: 00007fd22b9aa000 CR3: 0000004033a9a000 CR4: 00000000003407e0 [Wed Feb 6 07:42:04 2019][1689423.826001] Call Trace: [Wed Feb 6 07:42:04 2019][1689423.826007] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:04 2019][1689423.826011] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:04 2019][1689423.826054] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.826089] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.826127] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.826164] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.826200] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.826243] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.826283] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.826323] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.826362] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:04 2019][1689423.826366] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:04 2019][1689423.826368] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:04 2019][1689423.826371] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:04 2019][1689423.826373] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:04 2019][1689423.826373] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:04 2019][1689423.834899] NMI watchdog: BUG: soft lockup - CPU#47 stuck for 22s! [ldlm_cn03_020:25319] [Wed Feb 6 07:42:04 2019][1689423.834899] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:04 2019][1689423.834969] CPU: 47 PID: 25319 Comm: ldlm_cn03_020 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:04 2019][1689423.834970] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:04 2019][1689423.834971] task: ffff9139f37630c0 ti: ffff91319b3ec000 task.ti: ffff91319b3ec000 [Wed Feb 6 07:42:04 2019][1689423.834972] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:04 2019][1689423.834977] RSP: 0018:ffff91319b3efc38 EFLAGS: 00000246 [Wed Feb 6 07:42:04 2019][1689423.834978] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001790000 [Wed Feb 6 07:42:04 2019][1689423.834978] RDX: ffff915a3f4db780 RSI: 0000000000790001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:04 2019][1689423.834979] RBP: ffff91319b3efc38 R08: ffff915a3f6db780 R09: 0000000000000000 [Wed Feb 6 07:42:04 2019][1689423.834980] R10: ffff911b3fc07600 R11: ffffdbb29c73aa00 R12: ffff91319b3efbe0 [Wed Feb 6 07:42:04 2019][1689423.834981] R13: ffff91273b6849c0 R14: ffff91319b3efba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:42:04 2019][1689423.834982] FS: 00007f72eaa8d700(0000) GS:ffff915a3f6c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:04 2019][1689423.834983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:04 2019][1689423.834983] CR2: 00007effe6f18248 CR3: 000000403bd1e000 CR4: 00000000003407e0 [Wed Feb 6 07:42:04 2019][1689423.834984] Call Trace: [Wed Feb 6 07:42:04 2019][1689423.834987] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:04 2019][1689423.834990] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:05 2019][1689423.835032] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689423.835065] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689423.835102] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689423.835138] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689423.835173] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689423.835214] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689423.835252] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689423.835290] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689423.835293] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:42:05 2019][1689423.835331] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689423.835369] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689423.835371] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:05 2019][1689423.835373] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:05 2019][1689423.835375] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:05 2019][1689423.835376] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:05 2019][1689423.835377] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:42:05 2019] [Wed Feb 6 07:42:05 2019][1689424.834067] osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:05 2019][1689424.939447] CPU: 42 PID: 9990 Comm: ldlm_cn02_010 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:05 2019][1689424.952123] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:05 2019][1689424.959866] task: ffff9149d7abd140 ti: ffff91598baf0000 task.ti: ffff91598baf0000 [Wed Feb 6 07:42:05 2019][1689424.967516] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:05 2019][1689424.977632] RSP: 0018:ffff91598baf3c38 EFLAGS: 00000246 [Wed Feb 6 07:42:05 2019][1689424.983117] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001510000 [Wed Feb 6 07:42:05 2019][1689424.990422] RDX: ffff9149ff69b780 RSI: 0000000000510001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:05 2019][1689424.997728] RBP: ffff91598baf3c38 R08: ffff9149ff89b780 R09: 0000000000000000 [Wed Feb 6 07:42:05 2019][1689425.005034] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: ffff91598baf3be0 [Wed Feb 6 07:42:05 2019][1689425.012341] R13: ffff91273b6849c0 R14: ffff91598baf3ba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:42:05 2019][1689425.019648] FS: 00007fe2d3435880(0000) GS:ffff9149ff880000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:05 2019][1689425.027906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:05 2019][1689425.033828] CR2: 00007fe2d0618b00 CR3: 000000403a6c6000 CR4: 00000000003407e0 [Wed Feb 6 07:42:05 2019][1689425.041134] Call Trace: [Wed Feb 6 07:42:05 2019][1689425.043764] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:05 2019][1689425.050204] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:05 2019][1689425.055903] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689425.062635] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689425.069373] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689425.076452] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689425.083359] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689425.090449] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689425.098314] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689425.105576] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689425.112050] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:05 2019][1689425.119617] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:05 2019][1689425.124675] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:05 2019][1689425.130943] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:05 2019][1689425.137553] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:05 2019][1689425.143819] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:42:05 2019][1689425.457319] LustreError: 6493:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0014-osc-MDT0002: cannot cleanup orphans: rc = -11 [Wed Feb 6 07:42:06 2019][1689425.822561] Lustre: MGS: Received new LWP connection from 10.9.0.64@o2ib4, removing former export from same NID [Wed Feb 6 07:42:06 2019][1689425.832824] Lustre: Skipped 20 previous similar messages [Wed Feb 6 07:42:07 2019][1689427.564993] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 23s! [ldlm_cn00_024:20043] [Wed Feb 6 07:42:07 2019][1689427.573259] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE)[1689427.581993] NMI watchdog: BUG: soft lockup - CPU#18 stuck for 23s! [ldlm_cn02_017:23320] [Wed Feb 6 07:42:07 2019][1689427.581994] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:07 2019][1689427.582060] CPU: 18 PID: 23320 Comm: ldlm_cn02_017 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:08 2019][1689427.582061] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:08 2019][1689427.582062] task: ffff9159e661c100 ti: ffff91580b2b0000 task.ti: ffff91580b2b0000 [Wed Feb 6 07:42:08 2019][1689427.582063] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:08 2019][1689427.582069] RSP: 0018:ffff91580b2b3c38 EFLAGS: 00000246 [Wed Feb 6 07:42:08 2019][1689427.582070] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000910000 [Wed Feb 6 07:42:08 2019][1689427.582071] RDX: ffff9129ff01b780 RSI: 0000000001010001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:08 2019][1689427.582071] RBP: ffff91580b2b3c38 R08: ffff9149ff71b780 R09: 0000000000000000 [Wed Feb 6 07:42:08 2019][1689427.582072] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: ffff91580b2b3be0 [Wed Feb 6 07:42:08 2019][1689427.582073] R13: ffff91273b6849c0 R14: ffff91580b2b3ba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:42:08 2019][1689427.582074] FS: 00007f35005b3880(0000) GS:ffff9149ff700000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:08 2019][1689427.582075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:08 2019][1689427.582076] CR2: 00007ff7f8148248 CR3: 0000001035820000 CR4: 00000000003407e0 [Wed Feb 6 07:42:08 2019][1689427.582077] Call Trace: [Wed Feb 6 07:42:08 2019][1689427.582080] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:08 2019][1689427.582083] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:08 2019][1689427.582124] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.582158] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.582195] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.582231] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.582265] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.582307] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.582345] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.582383] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.582421] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.582424] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:08 2019][1689427.582426] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:08 2019][1689427.582428] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:08 2019][1689427.582430] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:08 2019][1689427.582430] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:42:08 2019][1689427.650993] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [ldlm_cn02_021:23326] [Wed Feb 6 07:42:08 2019][1689427.650994] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:08 2019][1689427.651044] CPU: 26 PID: 23326 Comm: ldlm_cn02_021 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:08 2019][1689427.651044] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:08 2019][1689427.651046] task: ffff91599ff9d140 ti: ffff9159aa650000 task.ti: ffff9159aa650000 [Wed Feb 6 07:42:08 2019][1689427.651047] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:08 2019][1689427.651051] RSP: 0018:ffff9159aa653c38 EFLAGS: 00000246 [Wed Feb 6 07:42:08 2019][1689427.651052] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000d10000 [Wed Feb 6 07:42:08 2019][1689427.651053] RDX: ffff9129fee9b780 RSI: 0000000000410001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:08 2019][1689427.651054] RBP: ffff9159aa653c38 R08: ffff9149ff79b780 R09: 0000000000000000 [Wed Feb 6 07:42:08 2019][1689427.651054] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: ffff9159aa653be0 [Wed Feb 6 07:42:08 2019][1689427.651055] R13: ffff91273b6849c0 R14: ffff9159aa653ba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:42:08 2019][1689427.651057] FS: 00007f35005b3880(0000) GS:ffff9149ff780000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:08 2019][1689427.651058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:08 2019][1689427.651059] CR2: 00007fe833b97ff0 CR3: 0000001035820000 CR4: 00000000003407e0 [Wed Feb 6 07:42:08 2019][1689427.651059] Call Trace: [Wed Feb 6 07:42:08 2019][1689427.651062] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:08 2019][1689427.651065] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:08 2019][1689427.651100] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.651134] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.651174] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.651214] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.651254] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.651298] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.651340] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.651380] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.651420] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.651422] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:08 2019][1689427.651424] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:08 2019][1689427.651426] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:08 2019][1689427.651427] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:08 2019][1689427.651428] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:08 2019][1689427.720995] NMI watchdog: BUG: soft lockup - CPU#34 stuck for 23s! [ldlm_cn02_023:28934] [Wed Feb 6 07:42:08 2019][1689427.720996] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:08 2019][1689427.721041] CPU: 34 PID: 28934 Comm: ldlm_cn02_023 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:08 2019][1689427.721042] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:08 2019][1689427.721043] task: ffff91488911a080 ti: ffff9156a4e78000 task.ti: ffff9156a4e78000 [Wed Feb 6 07:42:08 2019][1689427.721044] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:08 2019][1689427.721048] RSP: 0000:ffff9156a4e7bc38 EFLAGS: 00000246 [Wed Feb 6 07:42:08 2019][1689427.721048] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001110000 [Wed Feb 6 07:42:08 2019][1689427.721049] RDX: ffff9129fef1b780 RSI: 0000000000810001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:08 2019][1689427.721050] RBP: ffff9156a4e7bc38 R08: ffff9149ff81b780 R09: 0000000000000000 [Wed Feb 6 07:42:08 2019][1689427.721051] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: ffff9156a4e7bbe0 [Wed Feb 6 07:42:08 2019][1689427.721051] R13: ffff91273b6849c0 R14: ffff9156a4e7bba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:42:08 2019][1689427.721053] FS: 00007f7c8de93740(0000) GS:ffff9149ff800000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:08 2019][1689427.721054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:08 2019][1689427.721055] CR2: 00007f7c8d92f210 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:42:08 2019][1689427.721055] Call Trace: [Wed Feb 6 07:42:08 2019][1689427.721058] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:08 2019][1689427.721060] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:08 2019][1689427.721093] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.721126] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.721162] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.721198] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.721233] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.721272] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.721311] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.721351] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.721389] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:08 2019][1689427.721390] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:08 2019][1689427.721392] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:08 2019][1689427.721394] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:08 2019][1689427.721396] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:08 2019][1689427.721396] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:08 2019][1689427.755998] NMI watchdog: BUG: soft lockup - CPU#38 stuck for 22s! [ldlm_cn02_008:9191] [Wed Feb 6 07:42:08 2019][1689427.755999] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:08 2019][1689427.756076] CPU: 38 PID: 9191 Comm: ldlm_cn02_008 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:09 2019][1689427.756076] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:09 2019][1689427.756078] task: ffff9159ad54b0c0 ti: ffff9159a6244000 task.ti: ffff9159a6244000 [Wed Feb 6 07:42:09 2019][1689427.756079] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:09 2019][1689427.756087] RSP: 0018:ffff9159a6247bb8 EFLAGS: 00000246 [Wed Feb 6 07:42:09 2019][1689427.756088] RAX: 0000000000000000 RBX: ffff9159a6247bf0 RCX: 0000000001310000 [Wed Feb 6 07:42:09 2019][1689427.756089] RDX: ffff915a3f69b780 RSI: 0000000001590001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:09 2019][1689427.756090] RBP: ffff9159a6247bb8 R08: ffff9149ff85b780 R09: 0000000000000000 [Wed Feb 6 07:42:09 2019][1689427.756090] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: 0000000000000000 [Wed Feb 6 07:42:09 2019][1689427.756091] R13: 0000000000000000 R14: 0000000000000002 R15: ffff914f7cf5c800 [Wed Feb 6 07:42:09 2019][1689427.756093] FS: 00007f5b92ec7740(0000) GS:ffff9149ff840000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:09 2019][1689427.756094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:09 2019][1689427.756094] CR2: 00007f5b929db1e0 CR3: 00000020cd3f4000 CR4: 00000000003407e0 [Wed Feb 6 07:42:09 2019][1689427.756096] Call Trace: [Wed Feb 6 07:42:09 2019][1689427.756102] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:09 2019][1689427.756106] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:09 2019][1689427.756149] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.756183] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.756186] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:09 2019][1689427.756219] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.756257] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.756294] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.756330] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.756376] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.756417] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.756456] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.756496] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.756500] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:09 2019][1689427.756501] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:09 2019][1689427.756505] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:09 2019][1689427.756506] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:09 2019][1689427.756507] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:09 2019][1689427.809002] NMI watchdog: BUG: soft lockup - CPU#44 stuck for 23s! [ldlm_cn00_028:20558] [Wed Feb 6 07:42:09 2019][1689427.809003] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:09 2019][1689427.809082] CPU: 44 PID: 20558 Comm: ldlm_cn00_028 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:09 2019][1689427.809082] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:09 2019][1689427.809084] task: ffff9146d2b8a080 ti: ffff9159f5ffc000 task.ti: ffff9159f5ffc000 [Wed Feb 6 07:42:09 2019][1689427.809085] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:09 2019][1689427.809093] RSP: 0018:ffff9159f5fffbb8 EFLAGS: 00000246 [Wed Feb 6 07:42:09 2019][1689427.809094] RAX: 0000000000000000 RBX: ffff9159f5fffbf0 RCX: 0000000001610000 [Wed Feb 6 07:42:09 2019][1689427.809095] RDX: ffff9149ff89b780 RSI: 0000000001510000 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:09 2019][1689427.809095] RBP: ffff9159f5fffbb8 R08: ffff9129ff0db780 R09: 0000000000000000 [Wed Feb 6 07:42:09 2019][1689427.809096] R10: ffff911b3fc07600 R11: ffffdbb24dcd6c00 R12: 0000000000000000 [Wed Feb 6 07:42:09 2019][1689427.809097] R13: 0000000000000000 R14: 0000000000000002 R15: ffff913ca23b3cc0 [Wed Feb 6 07:42:09 2019][1689427.809098] FS: 00007f63fb26e700(0000) GS:ffff9129ff0c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:09 2019][1689427.809099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:09 2019][1689427.809100] CR2: 00007f01048dc1a0 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:42:09 2019][1689427.809101] Call Trace: [Wed Feb 6 07:42:09 2019][1689427.809106] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:09 2019][1689427.809110] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:09 2019][1689427.809153] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.809186] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.809189] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:09 2019][1689427.809221] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.809259] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.809295] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.809330] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.809372] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.809410] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.809448] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.809486] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689427.809489] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:09 2019][1689427.809491] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:09 2019][1689427.809494] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:09 2019][1689427.809496] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:09 2019][1689427.809496] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:42:09 2019] [Wed Feb 6 07:42:09 2019][1689429.310875] lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:09 2019][1689429.409636] CPU: 16 PID: 20043 Comm: ldlm_cn00_024 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:09 2019][1689429.422399] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:09 2019][1689429.430139] task: ffff9139f700a080 ti: ffff9158ffab0000 task.ti: ffff9158ffab0000 [Wed Feb 6 07:42:09 2019][1689429.437792] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:09 2019][1689429.447906] RSP: 0018:ffff9158ffab3bb8 EFLAGS: 00000246 [Wed Feb 6 07:42:09 2019][1689429.453394] RAX: 0000000000000000 RBX: ffff9158ffab3bf0 RCX: 0000000000810000 [Wed Feb 6 07:42:09 2019][1689429.460698] RDX: ffff9129ff0db780 RSI: 0000000001610000 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:09 2019][1689429.468003] RBP: ffff9158ffab3bb8 R08: ffff9129fef1b780 R09: 0000000000000000 [Wed Feb 6 07:42:09 2019][1689429.475310] R10: ffff911b3fc07600 R11: ffffdbb268b18400 R12: 0000000000000000 [Wed Feb 6 07:42:09 2019][1689429.482616] R13: 0000000000000000 R14: 0000000000000002 R15: ffff9157426a45c0 [Wed Feb 6 07:42:09 2019][1689429.489923] FS: 00007fe2d3435880(0000) GS:ffff9129fef00000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:09 2019][1689429.498182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:09 2019][1689429.504103] CR2: 00007fe2d0ba8050 CR3: 000000403a6c6000 CR4: 00000000003407e0 [Wed Feb 6 07:42:09 2019][1689429.511410] Call Trace: [Wed Feb 6 07:42:09 2019][1689429.514041] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:09 2019][1689429.520488] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:09 2019][1689429.526198] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689429.532928] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689429.539975] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:09 2019][1689429.547573] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689429.554314] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689429.561390] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689429.568298] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689429.575388] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689429.583252] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689429.590248] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689429.596728] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:09 2019][1689429.604295] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:09 2019][1689429.609344] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:09 2019][1689429.615611] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:09 2019][1689429.622223] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:09 2019][1689429.628490] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:42:11 2019][1689431.481095] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [ldlm_cn02_015:23318] [Wed Feb 6 07:42:11 2019][1689431.489270] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:11 2019][1689431.596040] CPU: 6 PID: 23318 Comm: ldlm_cn02_015 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:11 2019][1689431.608716] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:11 2019][1689431.616456] task: ffff9159e661b0c0 ti: ffff915853bac000 task.ti: ffff915853bac000 [Wed Feb 6 07:42:11 2019][1689431.624110] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:11 2019][1689431.634214] RSP: 0018:ffff915853bafc38 EFLAGS: 00000246 [Wed Feb 6 07:42:11 2019][1689431.639700] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000310000 [Wed Feb 6 07:42:11 2019][1689431.647006] RDX: ffff9129fee1b780 RSI: 0000000000010001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:11 2019][1689431.654312] RBP: ffff915853bafc38 R08: ffff9149ff65b780 R09: 0000000000000000 [Wed Feb 6 07:42:11 2019][1689431.661619] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: ffff915853bafbe0 [Wed Feb 6 07:42:11 2019][1689431.668926] R13: ffff91273b6849c0 R14: ffff915853bafba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:42:11 2019][1689431.676233] FS: 00007f277d7c4700(0000) GS:ffff9149ff640000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:11 2019][1689431.684490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:11 2019][1689431.690410] CR2: 00007fc91a77880d CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:42:11 2019][1689431.697716] Call Trace: [Wed Feb 6 07:42:11 2019][1689431.700349] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:11 2019][1689431.706794] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:11 2019][1689431.712493] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:12 2019][1689431.719227] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:12 2019][1689431.725963] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:12 2019][1689431.733044] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:12 2019][1689431.739951] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:12 2019][1689431.747037] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:12 2019][1689431.754906] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:12 2019][1689431.762166] [] ? ptlrpc_nrs_req_add_nolock+0x6a/0x1d0 [ptlrpc] [Wed Feb 6 07:42:12 2019][1689431.769855] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:12 2019][1689431.776328] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:12 2019][1689431.783894] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:12 2019][1689431.788946] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:12 2019][1689431.795212] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:12 2019][1689431.801823] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:12 2019][1689431.808089] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:14 2019][1689434.691873] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 07:42:15 2019][1689435.056849] LustreError: 6687:0:(upcall_cache.c:233:upcall_cache_get_entry()) acquire for key 16489: error -110 [Wed Feb 6 07:42:15 2019][1689435.494682] Lustre: 19973:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (94:4s); client may timeout. req@ffff9131e1326c50 x1624699767849392/t0(0) o103->2cc850be-dc6d-45a9-7198-5a65d8b1618d@10.9.104.39@o2ib4:144/0 lens 3128/192 e 2 to 0 dl 1549467769 ref 2 fl Complete:/0/0 rc 0/0 [Wed Feb 6 07:42:15 2019][1689435.523324] Lustre: 19973:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2 previous similar messages [Wed Feb 6 07:42:17 2019][1689436.809452] Lustre: 28327:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-12s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff9133043c6900 x1624673932936288/t0(0) o103->14d1e2af-bfe3-6fa6-de44-b510e2b94a1a@10.9.101.62@o2ib4:137/0 lens 328/0 e 1 to 0 dl 1549467762 ref 2 fl New:H/0/ffffffff rc 0/-1 [Wed Feb 6 07:42:19 2019][1689439.642298] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 21s! [ldlm_cn01_031:28334] [Wed Feb 6 07:42:19 2019][1689439.650562] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE)[1689439.669297] NMI watchdog: BUG: soft lockup - CPU#28 stuck for 23s! [ldlm_cn00_025:20158] [Wed Feb 6 07:42:19 2019][1689439.669299] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:20 2019][1689439.669373] CPU: 28 PID: 20158 Comm: ldlm_cn00_025 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:20 2019][1689439.669374] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:20 2019][1689439.669375] task: ffff9159f1f62080 ti: ffff9158f738c000 task.ti: ffff9158f738c000 [Wed Feb 6 07:42:20 2019][1689439.669377] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:20 2019][1689439.669385] RSP: 0018:ffff9158f738fbb8 EFLAGS: 00000246 [Wed Feb 6 07:42:20 2019][1689439.669386] RAX: 0000000000000000 RBX: ffff9158f738fbf0 RCX: 0000000000e10000 [Wed Feb 6 07:42:20 2019][1689439.669387] RDX: ffff9139ff81b780 RSI: 0000000001090000 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:20 2019][1689439.669387] RBP: ffff9158f738fbb8 R08: ffff9129fefdb780 R09: 0000000000000000 [Wed Feb 6 07:42:20 2019][1689439.669388] R10: ffff911b3fc07600 R11: ffffdbb2462ae000 R12: 0000000000000000 [Wed Feb 6 07:42:20 2019][1689439.669389] R13: 0000000000000000 R14: 0000000000000002 R15: ffff913ba308cec0 [Wed Feb 6 07:42:20 2019][1689439.669390] FS: 00007f6417cd6700(0000) GS:ffff9129fefc0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:20 2019][1689439.669391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:20 2019][1689439.669392] CR2: 00007f6410016068 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:42:20 2019][1689439.669393] Call Trace: [Wed Feb 6 07:42:20 2019][1689439.669399] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:20 2019][1689439.669403] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:20 2019][1689439.669446] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.669480] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.669482] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:20 2019][1689439.669515] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.669552] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.669589] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.669624] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.669667] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.669706] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.669745] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.669786] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.669789] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:20 2019][1689439.669791] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:20 2019][1689439.669794] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:20 2019][1689439.669795] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:20 2019][1689439.669796] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:20 2019][1689439.782298] NMI watchdog: BUG: soft lockup - CPU#41 stuck for 22s! [ldlm_cn01_005:6768] [Wed Feb 6 07:42:20 2019][1689439.782298] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:20 2019][1689439.782361] CPU: 41 PID: 6768 Comm: ldlm_cn01_005 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:20 2019][1689439.782361] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:20 2019][1689439.782363] task: ffff9159f33cd140 ti: ffff9159f2724000 task.ti: ffff9159f2724000 [Wed Feb 6 07:42:20 2019][1689439.782364] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:20 2019][1689439.782370] RSP: 0018:ffff9159f2727bb8 EFLAGS: 00000246 [Wed Feb 6 07:42:20 2019][1689439.782371] RAX: 0000000000000000 RBX: ffff9159f2727bf0 RCX: 0000000001490000 [Wed Feb 6 07:42:20 2019][1689439.782371] RDX: ffff915a3f41b780 RSI: 0000000000190001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:20 2019][1689439.782372] RBP: ffff9159f2727bb8 R08: ffff9139ff89b780 R09: 0000000000000000 [Wed Feb 6 07:42:20 2019][1689439.782373] R10: ffff911b3fc07600 R11: ffffdbb2f14ef400 R12: 0000000000000000 [Wed Feb 6 07:42:20 2019][1689439.782373] R13: 0000000000000000 R14: 0000000000000002 R15: ffff91242ee521c0 [Wed Feb 6 07:42:20 2019][1689439.782375] FS: 00007fe121ad2740(0000) GS:ffff9139ff880000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:20 2019][1689439.782375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:20 2019][1689439.782376] CR2: 0000000001111108 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:42:20 2019][1689439.782377] Call Trace: [Wed Feb 6 07:42:20 2019][1689439.782381] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:20 2019][1689439.782384] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:20 2019][1689439.782423] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.782456] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.782459] [] ? native_queued_spin_lock_slowpath+0x156/0x200 [Wed Feb 6 07:42:20 2019][1689439.782491] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.782527] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.782563] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.782598] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.782637] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.782675] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.782712] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.782750] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689439.782753] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:20 2019][1689439.782755] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:20 2019][1689439.782757] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:20 2019][1689439.782759] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:20 2019][1689439.782759] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:20 2019] [Wed Feb 6 07:42:20 2019][1689440.377956] obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:20 2019][1689440.467246] CPU: 25 PID: 28334 Comm: ldlm_cn01_031 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:20 2019][1689440.480010] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:20 2019][1689440.487750] task: ffff9159aa3db0c0 ti: ffff9156ee634000 task.ti: ffff9156ee634000 [Wed Feb 6 07:42:20 2019][1689440.495403] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:20 2019][1689440.505507] RSP: 0018:ffff9156ee637bb8 EFLAGS: 00000246 [Wed Feb 6 07:42:20 2019][1689440.510993] RAX: 0000000000000000 RBX: ffff9156ee637bf0 RCX: 0000000000c90000 [Wed Feb 6 07:42:20 2019][1689440.518300] RDX: ffff9129fee5b780 RSI: 0000000000210000 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:20 2019][1689440.525606] RBP: ffff9156ee637bb8 R08: ffff9139ff79b780 R09: 0000000000000000 [Wed Feb 6 07:42:20 2019][1689440.532912] R10: 0000000000000096 R11: 000000005c5b0047 R12: 0000000000000000 [Wed Feb 6 07:42:20 2019][1689440.540218] R13: 0000000000000000 R14: 0000000000000002 R15: ffff91506da1da00 [Wed Feb 6 07:42:20 2019][1689440.547527] FS: 00007fc9e2d79880(0000) GS:ffff9139ff780000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:20 2019][1689440.555784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:20 2019][1689440.561704] CR2: 00007f63fb343000 CR3: 000000303a262000 CR4: 00000000003407e0 [Wed Feb 6 07:42:20 2019][1689440.569009] Call Trace: [Wed Feb 6 07:42:20 2019][1689440.571638] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:20 2019][1689440.578076] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:20 2019][1689440.583769] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689440.590501] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689440.597551] [] ? native_queued_spin_lock_slowpath+0x15e/0x200 [Wed Feb 6 07:42:20 2019][1689440.605147] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689440.611876] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689440.618958] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689440.625864] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689440.632949] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689440.640809] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689440.647804] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689440.654277] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:20 2019][1689440.661843] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:20 2019][1689440.666895] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:20 2019][1689440.673160] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:20 2019][1689440.679774] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:20 2019][1689440.686039] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:42:23 2019][1689443.446394] INFO: rcu_sched self-detected stall on CPU [Wed Feb 6 07:42:23 2019][1689443.446398] INFO: rcu_sched self-detected stall on CPU [Wed Feb 6 07:42:23 2019][1689443.446401] INFO: rcu_sched self-detected stall on CPU [Wed Feb 6 07:42:23 2019][1689443.446405] INFO: rcu_sched self-detected stall on CPU [Wed Feb 6 07:42:23 2019][1689443.446406] INFO: rcu_sched self-detected stall on CPU [Wed Feb 6 07:42:23 2019][1689443.446408] { [Wed Feb 6 07:42:23 2019][1689443.446410] { [Wed Feb 6 07:42:23 2019][1689443.446415] { [Wed Feb 6 07:42:23 2019][1689443.446417] { [Wed Feb 6 07:42:23 2019][1689443.446419] 5 [Wed Feb 6 07:42:23 2019][1689443.446421] 6 [Wed Feb 6 07:42:23 2019][1689443.446423] 16 [Wed Feb 6 07:42:23 2019][1689443.446425] 17 [Wed Feb 6 07:42:23 2019][1689443.446426] } [Wed Feb 6 07:42:23 2019][1689443.446428] } [Wed Feb 6 07:42:23 2019][1689443.446430] } [Wed Feb 6 07:42:23 2019][1689443.446437] (t=60000 jiffies g=134910431 c=134910430 q=65592) [Wed Feb 6 07:42:23 2019][1689443.446438] Task dump for CPU 5: [Wed Feb 6 07:42:23 2019][1689443.446442] (t=60000 jiffies g=134910431 c=134910430 q=65592) [Wed Feb 6 07:42:23 2019][1689443.446444] } [Wed Feb 6 07:42:23 2019][1689443.446447] (t=60000 jiffies g=134910431 c=134910430 q=65592) [Wed Feb 6 07:42:23 2019][1689443.446450] (t=60000 jiffies g=134910431 c=134910430 q=65592) [Wed Feb 6 07:42:23 2019][1689443.446453] ldlm_cn01_028 R running task 0 28329 2 0x00000088 [Wed Feb 6 07:42:23 2019][1689443.446454] Call Trace: [Wed Feb 6 07:42:23 2019][1689443.446464] [] sched_show_task+0xa8/0x110 [Wed Feb 6 07:42:23 2019][1689443.446467] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 07:42:23 2019][1689443.446470] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 07:42:23 2019][1689443.446473] [] rcu_check_callbacks+0x442/0x730 [Wed Feb 6 07:42:23 2019][1689443.446477] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 07:42:23 2019][1689443.446480] [] update_process_times+0x46/0x80 [Wed Feb 6 07:42:23 2019][1689443.446481] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 07:42:23 2019][1689443.446483] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 07:42:23 2019][1689443.446487] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 07:42:23 2019][1689443.446489] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 07:42:23 2019][1689443.446493] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 07:42:23 2019][1689443.446496] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 07:42:23 2019][1689443.446499] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 07:42:23 2019][1689443.446503] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:23 2019][1689443.446506] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:23 2019][1689443.446509] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:23 2019][1689443.446550] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:23 2019][1689443.446583] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:23 2019][1689443.446620] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:23 2019][1689443.446656] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:23 2019][1689443.446691] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:23 2019][1689443.446732] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:23 2019][1689443.446734] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:23 2019][1689443.446736] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:23 2019][1689443.446775] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:23 2019][1689443.446813] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:23 2019][1689443.446815] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:23 2019][1689443.446817] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:23 2019][1689443.446819] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:24 2019][1689443.446821] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:24 2019][1689443.446822] Task dump for CPU 6: [Wed Feb 6 07:42:24 2019][1689443.446824] ldlm_cn02_015 R running task 0 23318 2 0x00000088 [Wed Feb 6 07:42:24 2019][1689443.446825] Call Trace: [Wed Feb 6 07:42:24 2019][1689443.446837] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:24 2019][1689443.446839] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:24 2019][1689443.446840] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:24 2019][1689443.446842] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:24 2019][1689443.446874] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.446906] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.446940] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.446975] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447010] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447048] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447085] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447130] [] ? ptlrpc_nrs_req_add_nolock+0x6a/0x1d0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447167] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447204] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447206] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:24 2019][1689443.447207] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:24 2019][1689443.447209] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:24 2019][1689443.447211] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:24 2019][1689443.447212] Task dump for CPU 16: [Wed Feb 6 07:42:24 2019][1689443.447213] Task dump for CPU 5: [Wed Feb 6 07:42:24 2019][1689443.447219] ldlm_cn00_024 R [Wed Feb 6 07:42:24 2019][1689443.447220] ldlm_cn01_028 R [Wed Feb 6 07:42:24 2019][1689443.447222] running task [Wed Feb 6 07:42:24 2019][1689443.447224] 0 20043 2 0x00000088 [Wed Feb 6 07:42:24 2019][1689443.447226] running task 0 28329 2 0x00000088 [Wed Feb 6 07:42:24 2019][1689443.447228] Call Trace: [Wed Feb 6 07:42:24 2019][1689443.447230] Call Trace: [Wed Feb 6 07:42:24 2019][1689443.447239] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:24 2019][1689443.447292] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447302] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:24 2019][1689443.447338] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447342] [] ? native_queued_spin_lock_slowpath+0x156/0x200 [Wed Feb 6 07:42:24 2019][1689443.447385] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447387] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:24 2019][1689443.447395] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:24 2019][1689443.447399] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:24 2019][1689443.447404] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:24 2019][1689443.447438] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447478] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447514] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447553] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447589] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447594] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:24 2019][1689443.447597] INFO: rcu_sched self-detected stall on CPU { 34} (t=60001 jiffies g=134910431 c=134910430 q=65592) [Wed Feb 6 07:42:24 2019][1689443.447634] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447637] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:24 2019][1689443.447673] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447678] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:24 2019][1689443.447716] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447719] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:24 2019][1689443.447758] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447759] Task dump for CPU 6: [Wed Feb 6 07:42:24 2019][1689443.447798] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447838] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447842] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:24 2019][1689443.447844] ldlm_cn02_015 R running task 0 23318 2 0x00000088 [Wed Feb 6 07:42:24 2019][1689443.447847] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:24 2019][1689443.447848] Call Trace: [Wed Feb 6 07:42:24 2019][1689443.447851] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:24 2019][1689443.447856] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:24 2019][1689443.447859] [] sched_show_task+0xa8/0x110 [Wed Feb 6 07:42:24 2019][1689443.447861] Task dump for CPU 17: [Wed Feb 6 07:42:24 2019][1689443.447864] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 07:42:24 2019][1689443.447870] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 07:42:24 2019][1689443.447874] [] rcu_check_callbacks+0x442/0x730 [Wed Feb 6 07:42:24 2019][1689443.447876] ldlm_cn01_023 R running task 0 28088 2 0x00000088 [Wed Feb 6 07:42:24 2019][1689443.447881] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 07:42:24 2019][1689443.447882] Call Trace: [Wed Feb 6 07:42:24 2019][1689443.447886] [] update_process_times+0x46/0x80 [Wed Feb 6 07:42:24 2019][1689443.447893] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:24 2019][1689443.447896] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 07:42:24 2019][1689443.447900] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:24 2019][1689443.447902] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 07:42:24 2019][1689443.447905] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:24 2019][1689443.447909] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 07:42:24 2019][1689443.447911] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:24 2019][1689443.447915] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 07:42:24 2019][1689443.447947] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447952] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 07:42:24 2019][1689443.447986] [] ? ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.447989] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 07:42:24 2019][1689443.447993] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:24 2019][1689443.447996] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 07:42:24 2019][1689443.448029] [] ? ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448066] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448072] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:24 2019][1689443.448108] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448112] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:24 2019][1689443.448148] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448152] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:24 2019][1689443.448190] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448223] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448261] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448295] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448298] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:24 2019][1689443.448334] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448337] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:24 2019][1689443.448375] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448415] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448452] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448493] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448532] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448536] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:24 2019][1689443.448575] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448578] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:24 2019][1689443.448625] [] ? ptlrpc_nrs_req_add_nolock+0x6a/0x1d0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448627] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:24 2019][1689443.448667] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448669] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:24 2019][1689443.448708] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448710] Task dump for CPU 34: [Wed Feb 6 07:42:24 2019][1689443.448713] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:24 2019][1689443.448714] INFO: rcu_sched detected stalls on CPUs/tasks: { [Wed Feb 6 07:42:24 2019][1689443.448718] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:24 2019][1689443.448722] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:24 2019][1689443.448724] ldlm_cn02_023 R running task 0 28934 2 0x00000088 [Wed Feb 6 07:42:24 2019][1689443.448727] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:24 2019][1689443.448728] Call Trace: [Wed Feb 6 07:42:24 2019][1689443.448731] Task dump for CPU 16: [Wed Feb 6 07:42:24 2019][1689443.448733] Task dump for CPU 5: [Wed Feb 6 07:42:24 2019][1689443.448740] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:24 2019][1689443.448746] ldlm_cn00_024 R [Wed Feb 6 07:42:24 2019][1689443.448755] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:24 2019][1689443.448756] ldlm_cn01_028 R [Wed Feb 6 07:42:24 2019][1689443.448758] running task [Wed Feb 6 07:42:24 2019][1689443.448762] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:24 2019][1689443.448764] 0 20043 2 0x00000088 [Wed Feb 6 07:42:24 2019][1689443.448768] running task 0 28329 2 0x00000088 [Wed Feb 6 07:42:24 2019][1689443.448771] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:24 2019][1689443.448773] Call Trace: [Wed Feb 6 07:42:24 2019][1689443.448774] Call Trace: [Wed Feb 6 07:42:24 2019][1689443.448778] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:24 2019][1689443.448788] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:24 2019][1689443.448821] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448873] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448886] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:24 2019][1689443.448920] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448956] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.448959] [] ? native_queued_spin_lock_slowpath+0x156/0x200 [Wed Feb 6 07:42:24 2019][1689443.448995] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.449039] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.449042] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:24 2019][1689443.449079] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.449085] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:24 2019][1689443.449088] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:24 2019][1689443.449124] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.449129] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:24 2019][1689443.449162] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.449200] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.449241] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.449274] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.449313] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.449353] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.449390] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.449429] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:24 2019][1689443.449434] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:25 2019][1689443.449470] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.449509] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.449512] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.449548] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.449551] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:25 2019][1689443.449556] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:25 2019][1689443.449595] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.449598] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.449601] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.449640] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.449644] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:25 2019][1689443.449645] Task dump for CPU 6: [Wed Feb 6 07:42:25 2019][1689443.449684] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.449687] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.449728] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.449730] Task dump for CPU 44: [Wed Feb 6 07:42:25 2019][1689443.449735] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:25 2019][1689443.449737] ldlm_cn02_015 R running task [Wed Feb 6 07:42:25 2019][1689443.449740] 0 23318 2 0x00000088 [Wed Feb 6 07:42:25 2019][1689443.449743] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.449746] Call Trace: [Wed Feb 6 07:42:25 2019][1689443.449750] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:25 2019][1689443.449752] ldlm_cn00_028 R running task 0 20558 2 0x00000088 [Wed Feb 6 07:42:25 2019][1689443.449766] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:25 2019][1689443.449768] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.449770] Call Trace: [Wed Feb 6 07:42:25 2019][1689443.449776] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:25 2019][1689443.449777] Task dump for CPU 17: [Wed Feb 6 07:42:25 2019][1689443.449781] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:25 2019][1689443.449786] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:25 2019][1689443.449791] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:25 2019][1689443.449795] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:25 2019][1689443.449800] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:25 2019][1689443.449833] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.449836] ldlm_cn01_023 R running task 0 28088 2 0x00000088 [Wed Feb 6 07:42:25 2019][1689443.449869] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.449903] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.449904] Call Trace: [Wed Feb 6 07:42:25 2019][1689443.449938] [] ? ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.449975] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.449982] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:25 2019][1689443.449985] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:25 2019][1689443.450022] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450026] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:25 2019][1689443.450060] [] ? ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450096] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450099] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:25 2019][1689443.450136] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450175] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450178] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:25 2019][1689443.450215] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450254] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450286] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450323] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450371] [] ? ptlrpc_nrs_req_add_nolock+0x6a/0x1d0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450405] [] ? ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450444] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450483] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450487] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:25 2019][1689443.450525] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450565] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450599] [] ? ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450637] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450640] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:25 2019][1689443.450677] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450715] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450718] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.450755] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450757] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:25 2019][1689443.450761] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:25 2019][1689443.450797] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450800] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.450803] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.450842] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450845] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:25 2019][1689443.450847] Task dump for CPU 5: [Wed Feb 6 07:42:25 2019][1689443.450886] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450890] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.450895] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:25 2019][1689443.450900] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:25 2019][1689443.450903] ldlm_cn01_028 R running task 0 28329 2 0x00000088 [Wed Feb 6 07:42:25 2019][1689443.450942] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.450944] Call Trace: [Wed Feb 6 07:42:25 2019][1689443.450982] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451026] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451029] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:25 2019][1689443.451066] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451069] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.451109] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451112] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:25 2019][1689443.451117] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:25 2019][1689443.451120] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.451124] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:25 2019][1689443.451127] Task dump for CPU 34: [Wed Feb 6 07:42:25 2019][1689443.451129] Task dump for CPU 16: [Wed Feb 6 07:42:25 2019][1689443.451167] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451172] ldlm_cn02_023 R [Wed Feb 6 07:42:25 2019][1689443.451211] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451212] ldlm_cn00_024 R [Wed Feb 6 07:42:25 2019][1689443.451214] running task [Wed Feb 6 07:42:25 2019][1689443.451218] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:25 2019][1689443.451221] 0 28934 2 0x00000088 [Wed Feb 6 07:42:25 2019][1689443.451223] running task 0 20043 2 0x00000088 [Wed Feb 6 07:42:25 2019][1689443.451226] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.451228] Call Trace: [Wed Feb 6 07:42:25 2019][1689443.451229] Call Trace: [Wed Feb 6 07:42:25 2019][1689443.451234] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:25 2019][1689443.451241] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:25 2019][1689443.451246] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.451255] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:25 2019][1689443.451259] [] sched_show_task+0xa8/0x110 [Wed Feb 6 07:42:25 2019][1689443.451261] Task dump for CPU 6: [Wed Feb 6 07:42:25 2019][1689443.451264] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:25 2019][1689443.451268] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 07:42:25 2019][1689443.451273] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:25 2019][1689443.451278] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 07:42:25 2019][1689443.451282] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:25 2019][1689443.451286] [] rcu_check_callbacks+0x442/0x730 [Wed Feb 6 07:42:25 2019][1689443.451288] ldlm_cn02_015 R running task 0 23318 2 0x00000088 [Wed Feb 6 07:42:25 2019][1689443.451322] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451325] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 07:42:25 2019][1689443.451328] Call Trace: [Wed Feb 6 07:42:25 2019][1689443.451361] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451366] [] update_process_times+0x46/0x80 [Wed Feb 6 07:42:25 2019][1689443.451377] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:25 2019][1689443.451413] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451416] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 07:42:25 2019][1689443.451421] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:25 2019][1689443.451457] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451460] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 07:42:25 2019][1689443.451465] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:25 2019][1689443.451501] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451504] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 07:42:25 2019][1689443.451508] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:25 2019][1689443.451547] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451550] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 07:42:25 2019][1689443.451584] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451622] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451627] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 07:42:25 2019][1689443.451661] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451699] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451703] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 07:42:25 2019][1689443.451739] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451778] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451782] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 07:42:25 2019][1689443.451820] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451822] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:25 2019][1689443.451861] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451863] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.451868] [] ? native_queued_spin_lock_slowpath+0x156/0x200 [Wed Feb 6 07:42:25 2019][1689443.451906] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451909] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:25 2019][1689443.451912] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:25 2019][1689443.451950] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.451954] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:25 2019][1689443.451957] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:25 2019][1689443.452002] [] ? ptlrpc_nrs_req_add_nolock+0x6a/0x1d0 [ptlrpc] [Wed Feb 6 07:42:25 2019][1689443.452005] Task dump for CPU 44: [Wed Feb 6 07:42:25 2019][1689443.452037] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452076] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452112] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452151] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452188] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452191] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:26 2019][1689443.452194] ldlm_cn00_028 R running task 0 20558 2 0x00000088 [Wed Feb 6 07:42:26 2019][1689443.452231] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452233] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.452235] Call Trace: [Wed Feb 6 07:42:26 2019][1689443.452271] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452275] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:26 2019][1689443.452279] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:26 2019][1689443.452318] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452321] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.452323] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:26 2019][1689443.452363] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452364] Task dump for CPU 5: [Wed Feb 6 07:42:26 2019][1689443.452367] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:26 2019][1689443.452407] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452441] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452480] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452514] [] ? ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452517] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:26 2019][1689443.452520] ldlm_cn01_028 R running task 0 28329 2 0x00000088 [Wed Feb 6 07:42:26 2019][1689443.452524] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:26 2019][1689443.452526] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.452528] Call Trace: [Wed Feb 6 07:42:26 2019][1689443.452562] [] ? ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452566] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:26 2019][1689443.452616] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452652] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452655] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.452692] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452728] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452730] Task dump for CPU 17: [Wed Feb 6 07:42:26 2019][1689443.452772] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452808] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452817] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:26 2019][1689443.452854] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452859] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:26 2019][1689443.452897] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452900] ldlm_cn01_023 R running task 0 28088 2 0x00000088 [Wed Feb 6 07:42:26 2019][1689443.452940] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452977] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.452979] Call Trace: [Wed Feb 6 07:42:26 2019][1689443.453018] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453056] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453066] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:26 2019][1689443.453070] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:26 2019][1689443.453073] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:26 2019][1689443.453077] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:26 2019][1689443.453079] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.453082] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.453085] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:26 2019][1689443.453090] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:26 2019][1689443.453093] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:26 2019][1689443.453096] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:26 2019][1689443.453099] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.453102] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.453135] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453136] Task dump for CPU 6: [Wed Feb 6 07:42:26 2019][1689443.453170] [] ? ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453176] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:26 2019][1689443.453211] [] ? ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453213] ldlm_cn02_015 R running task 0 23318 2 0x00000088 [Wed Feb 6 07:42:26 2019][1689443.453249] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453251] Call Trace: [Wed Feb 6 07:42:26 2019][1689443.453287] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453300] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:26 2019][1689443.453336] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453342] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:26 2019][1689443.453380] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453384] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:26 2019][1689443.453423] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453427] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:26 2019][1689443.453430] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:26 2019][1689443.453463] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453466] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:26 2019][1689443.453499] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453538] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453574] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453612] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453648] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453651] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:26 2019][1689443.453686] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453689] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.453728] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453731] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:26 2019][1689443.453769] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453772] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.453818] [] ? ptlrpc_nrs_req_add_nolock+0x6a/0x1d0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453821] Task dump for CPU 34: [Wed Feb 6 07:42:26 2019][1689443.453823] Task dump for CPU 16: [Wed Feb 6 07:42:26 2019][1689443.453861] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453866] ldlm_cn02_023 R [Wed Feb 6 07:42:26 2019][1689443.453904] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.453906] ldlm_cn00_024 R [Wed Feb 6 07:42:26 2019][1689443.453908] running task [Wed Feb 6 07:42:26 2019][1689443.453911] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:26 2019][1689443.453913] 0 28934 2 0x00000088 [Wed Feb 6 07:42:26 2019][1689443.453916] running task 0 20043 2 0x00000088 [Wed Feb 6 07:42:26 2019][1689443.453919] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.453920] Call Trace: [Wed Feb 6 07:42:26 2019][1689443.453922] Call Trace: [Wed Feb 6 07:42:26 2019][1689443.453926] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:26 2019][1689443.453933] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:26 2019][1689443.453941] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:26 2019][1689443.453944] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.453954] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:26 2019][1689443.453963] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:26 2019][1689443.453969] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:26 2019][1689443.453973] [] ? native_queued_spin_lock_slowpath+0x156/0x200 [Wed Feb 6 07:42:26 2019][1689443.453977] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:26 2019][1689443.453981] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:26 2019][1689443.453984] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:26 2019][1689443.453987] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:26 2019][1689443.454020] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454053] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454086] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454120] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454156] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454192] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454229] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454265] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454301] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454337] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454376] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454415] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454453] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454491] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454530] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454568] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454607] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454645] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454648] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:26 2019][1689443.454650] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:26 2019][1689443.454653] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.454656] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.454659] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:26 2019][1689443.454662] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:26 2019][1689443.454665] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.454667] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:26 2019][1689443.454670] Task dump for CPU 44: [Wed Feb 6 07:42:26 2019][1689443.454671] Task dump for CPU 17: [Wed Feb 6 07:42:26 2019][1689443.454673] 5 6 [Wed Feb 6 07:42:26 2019][1689443.454676] ldlm_cn00_028 R [Wed Feb 6 07:42:26 2019][1689443.454677] ldlm_cn01_023 R [Wed Feb 6 07:42:26 2019][1689443.454679] running task [Wed Feb 6 07:42:26 2019][1689443.454681] 0 20558 2 0x00000088 [Wed Feb 6 07:42:26 2019][1689443.454684] running task 0 28088 2 0x00000088 [Wed Feb 6 07:42:26 2019][1689443.454685] Call Trace: [Wed Feb 6 07:42:26 2019][1689443.454687] Call Trace: [Wed Feb 6 07:42:26 2019][1689443.454690] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:26 2019][1689443.454695] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:26 2019][1689443.454699] [] sched_show_task+0xa8/0x110 [Wed Feb 6 07:42:26 2019][1689443.454702] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:26 2019][1689443.454705] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 07:42:26 2019][1689443.454737] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:26 2019][1689443.454741] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 07:42:26 2019][1689443.454774] [] ? ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.454777] [] rcu_check_callbacks+0x442/0x730 [Wed Feb 6 07:42:27 2019][1689443.454781] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:27 2019][1689443.454784] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 07:42:27 2019][1689443.454818] [] ? ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.454822] [] update_process_times+0x46/0x80 [Wed Feb 6 07:42:27 2019][1689443.454858] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.454861] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 07:42:27 2019][1689443.454897] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.454900] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 07:42:27 2019][1689443.454936] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.454940] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 07:42:27 2019][1689443.454978] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.454981] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 07:42:27 2019][1689443.455019] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455024] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 07:42:27 2019][1689443.455062] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455066] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 07:42:27 2019][1689443.455105] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455108] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 07:42:27 2019][1689443.455110] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:27 2019][1689443.455115] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:27 2019][1689443.455119] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:27 2019][1689443.455121] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:27 2019][1689443.455124] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:27 2019][1689443.455127] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:27 2019][1689443.455130] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:27 2019][1689443.455161] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455194] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455196] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:27 2019][1689443.455228] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455263] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455299] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455334] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455371] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455408] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455410] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:27 2019][1689443.455412] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:27 2019][1689443.455449] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455487] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455488] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:27 2019][1689443.455490] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:27 2019][1689443.455491] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:27 2019][1689443.455493] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:27 2019][1689443.455494] Task dump for CPU 34: [Wed Feb 6 07:42:27 2019][1689443.455495] Task dump for CPU 16: [Wed Feb 6 07:42:27 2019][1689443.455500] ldlm_cn02_023 R [Wed Feb 6 07:42:27 2019][1689443.455501] ldlm_cn00_024 R [Wed Feb 6 07:42:27 2019][1689443.455503] running task [Wed Feb 6 07:42:27 2019][1689443.455505] 0 28934 2 0x00000088 [Wed Feb 6 07:42:27 2019][1689443.455508] running task 0 20043 2 0x00000088 [Wed Feb 6 07:42:27 2019][1689443.455509] Call Trace: [Wed Feb 6 07:42:27 2019][1689443.455510] Call Trace: [Wed Feb 6 07:42:27 2019][1689443.455518] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:27 2019][1689443.455528] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:27 2019][1689443.455538] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:27 2019][1689443.455547] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:27 2019][1689443.455551] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:27 2019][1689443.455554] [] ? native_queued_spin_lock_slowpath+0x15e/0x200 [Wed Feb 6 07:42:27 2019][1689443.455556] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:27 2019][1689443.455559] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:27 2019][1689443.455562] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:27 2019][1689443.455565] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:27 2019][1689443.455597] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455630] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455663] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455696] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455732] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455768] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455804] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455841] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455876] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455912] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455950] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.455989] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456026] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456064] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456103] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456141] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456179] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456217] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456220] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:27 2019][1689443.456222] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:27 2019][1689443.456225] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:27 2019][1689443.456227] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:27 2019][1689443.456230] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:27 2019][1689443.456233] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:27 2019][1689443.456236] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:27 2019][1689443.456239] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:27 2019][1689443.456241] Task dump for CPU 44: [Wed Feb 6 07:42:27 2019][1689443.456243] Task dump for CPU 17: [Wed Feb 6 07:42:27 2019][1689443.456247] ldlm_cn00_028 R [Wed Feb 6 07:42:27 2019][1689443.456249] ldlm_cn01_023 R [Wed Feb 6 07:42:27 2019][1689443.456251] running task [Wed Feb 6 07:42:27 2019][1689443.456253] 0 20558 2 0x00000088 [Wed Feb 6 07:42:27 2019][1689443.456256] running task 0 28088 2 0x00000088 [Wed Feb 6 07:42:27 2019][1689443.456257] Call Trace: [Wed Feb 6 07:42:27 2019][1689443.456258] Call Trace: [Wed Feb 6 07:42:27 2019][1689443.456262] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:27 2019][1689443.456269] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:27 2019][1689443.456272] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:27 2019][1689443.456276] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:27 2019][1689443.456278] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:27 2019][1689443.456281] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:27 2019][1689443.456313] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456316] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:27 2019][1689443.456349] [] ? ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456382] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456385] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:27 2019][1689443.456418] [] ? ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456452] [] ? ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456455] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:27 2019][1689443.456491] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456525] [] ? ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456561] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456596] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456632] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456669] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456707] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456743] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456781] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456819] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456857] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456895] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456934] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456936] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:27 2019][1689443.456939] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:27 2019][1689443.456942] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:27 2019][1689443.456944] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:27 2019][1689443.456982] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.456985] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:27 2019][1689443.457023] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.457026] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:27 2019][1689443.457028] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:27 2019][1689443.457030] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:27 2019][1689443.457032] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:27 2019][1689443.457034] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:27 2019][1689443.457035] Task dump for CPU 34: [Wed Feb 6 07:42:27 2019][1689443.457036] 16 [Wed Feb 6 07:42:27 2019][1689443.457037] ldlm_cn02_023 R [Wed Feb 6 07:42:27 2019][1689443.457038] 17 [Wed Feb 6 07:42:27 2019][1689443.457039] running task 0 28934 2 0x00000088 [Wed Feb 6 07:42:27 2019][1689443.457040] Call Trace: [Wed Feb 6 07:42:27 2019][1689443.457043] [] sched_show_task+0xa8/0x110 [Wed Feb 6 07:42:27 2019][1689443.457045] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 07:42:27 2019][1689443.457049] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 07:42:27 2019][1689443.457051] [] rcu_check_callbacks+0x442/0x730 [Wed Feb 6 07:42:27 2019][1689443.457054] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 07:42:27 2019][1689443.457058] [] update_process_times+0x46/0x80 [Wed Feb 6 07:42:27 2019][1689443.457060] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 07:42:27 2019][1689443.457061] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 07:42:27 2019][1689443.457064] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 07:42:27 2019][1689443.457066] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 07:42:27 2019][1689443.457070] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 07:42:27 2019][1689443.457072] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 07:42:27 2019][1689443.457074] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 07:42:27 2019][1689443.457077] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:27 2019][1689443.457079] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:27 2019][1689443.457081] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:27 2019][1689443.457112] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.457146] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:27 2019][1689443.457181] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457217] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457251] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457288] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457325] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457362] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457400] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457401] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:28 2019][1689443.457403] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:28 2019][1689443.457405] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:28 2019][1689443.457406] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:28 2019][1689443.457407] Task dump for CPU 44: [Wed Feb 6 07:42:28 2019][1689443.457409] ldlm_cn00_028 R running task 0 20558 2 0x00000088 [Wed Feb 6 07:42:28 2019][1689443.457410] Call Trace: [Wed Feb 6 07:42:28 2019][1689443.457413] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:28 2019][1689443.457414] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:28 2019][1689443.457416] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:28 2019][1689443.457447] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457479] [] ? ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457482] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:28 2019][1689443.457514] [] ? ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457548] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457583] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457618] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457655] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457692] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457729] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457766] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457767] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:28 2019][1689443.457769] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:28 2019][1689443.457770] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:28 2019][1689443.457772] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:28 2019][1689443.457784] 34 44} (detected by 18, t=60011 jiffies, g=134910431, c=134910430, q=65592) [Wed Feb 6 07:42:28 2019][1689443.457785] Task dump for CPU 5: [Wed Feb 6 07:42:28 2019][1689443.457788] ldlm_cn01_028 R running task 0 28329 2 0x00000088 [Wed Feb 6 07:42:28 2019][1689443.457788] Call Trace: [Wed Feb 6 07:42:28 2019][1689443.457827] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457862] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457900] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457903] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:28 2019][1689443.457904] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:28 2019][1689443.457942] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457980] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.457981] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:28 2019][1689443.457983] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:28 2019][1689443.457985] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:28 2019][1689443.457986] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:28 2019][1689443.457987] Task dump for CPU 6: [Wed Feb 6 07:42:28 2019][1689443.457989] ldlm_cn02_015 R running task 0 23318 2 0x00000088 [Wed Feb 6 07:42:28 2019][1689443.457990] Call Trace: [Wed Feb 6 07:42:28 2019][1689443.457999] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:28 2019][1689443.458001] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:28 2019][1689443.458003] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:28 2019][1689443.458005] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:28 2019][1689443.458036] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458068] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458103] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458138] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458173] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458211] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458248] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458290] [] ? ptlrpc_nrs_req_add_nolock+0x6a/0x1d0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458327] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458364] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458366] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:28 2019][1689443.458368] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:28 2019][1689443.458370] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:28 2019][1689443.458371] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:28 2019][1689443.458372] Task dump for CPU 16: [Wed Feb 6 07:42:28 2019][1689443.458374] ldlm_cn00_024 R running task 0 20043 2 0x00000088 [Wed Feb 6 07:42:28 2019][1689443.458375] Call Trace: [Wed Feb 6 07:42:28 2019][1689443.458381] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:28 2019][1689443.458390] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:28 2019][1689443.458393] [] ? native_queued_spin_lock_slowpath+0x156/0x200 [Wed Feb 6 07:42:28 2019][1689443.458394] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:28 2019][1689443.458396] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:28 2019][1689443.458427] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458460] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458494] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458530] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458564] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458601] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458639] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458676] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458713] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458715] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:28 2019][1689443.458716] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:28 2019][1689443.458718] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:28 2019][1689443.458719] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:28 2019][1689443.458720] Task dump for CPU 17: [Wed Feb 6 07:42:28 2019][1689443.458722] ldlm_cn01_023 R running task 0 28088 2 0x00000088 [Wed Feb 6 07:42:28 2019][1689443.458723] Call Trace: [Wed Feb 6 07:42:28 2019][1689443.458728] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:28 2019][1689443.458731] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:28 2019][1689443.458732] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:28 2019][1689443.458734] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:28 2019][1689443.458765] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458798] [] ? ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458799] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:28 2019][1689443.458832] [] ? ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458867] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458902] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458936] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.458973] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459010] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459012] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:28 2019][1689443.459014] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:28 2019][1689443.459051] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459088] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459089] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:28 2019][1689443.459090] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:28 2019][1689443.459092] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:28 2019][1689443.459094] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:28 2019][1689443.459095] Task dump for CPU 34: [Wed Feb 6 07:42:28 2019][1689443.459096] ldlm_cn02_023 R running task 0 28934 2 0x00000088 [Wed Feb 6 07:42:28 2019][1689443.459097] Call Trace: [Wed Feb 6 07:42:28 2019][1689443.459103] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:28 2019][1689443.459111] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:28 2019][1689443.459113] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:28 2019][1689443.459115] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:28 2019][1689443.459116] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:28 2019][1689443.459148] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459180] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459215] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459250] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459285] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459322] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459359] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459396] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459433] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459435] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:28 2019][1689443.459436] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:28 2019][1689443.459438] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:28 2019][1689443.459439] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:28 2019][1689443.459440] Task dump for CPU 44: [Wed Feb 6 07:42:28 2019][1689443.459442] ldlm_cn00_028 R running task 0 20558 2 0x00000088 [Wed Feb 6 07:42:28 2019][1689443.459442] Call Trace: [Wed Feb 6 07:42:28 2019][1689443.459445] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:28 2019][1689443.459446] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:28 2019][1689443.459448] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:28 2019][1689443.459479] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459511] [] ? ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459513] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:28 2019][1689443.459545] [] ? ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459580] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459615] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459650] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459687] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459724] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459761] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459798] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:28 2019][1689443.459799] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:28 2019][1689443.459801] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:28 2019][1689443.459803] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:29 2019][1689443.459804] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:29 2019][1689447.625495] NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ldlm_cn03_000:6038] [Wed Feb 6 07:42:29 2019][1689447.625534] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:29 2019][1689447.625553] CPU: 23 PID: 6038 Comm: ldlm_cn03_000 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:29 2019][1689447.625553] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:29 2019][1689447.625554] task: ffff9129f3049040 ti: ffff9159a86e0000 task.ti: ffff9159a86e0000 [Wed Feb 6 07:42:29 2019][1689447.625560] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:29 2019][1689447.625561] RSP: 0018:ffff9159a86e3bb8 EFLAGS: 00000246 [Wed Feb 6 07:42:29 2019][1689447.625562] RAX: 0000000000000000 RBX: ffff9159a86e3bf0 RCX: 0000000000b90000 [Wed Feb 6 07:42:29 2019][1689447.625563] RDX: ffff9139ff69b780 RSI: 0000000000490000 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:29 2019][1689447.625563] RBP: ffff9159a86e3bb8 R08: ffff915a3f55b780 R09: 0000000000000000 [Wed Feb 6 07:42:29 2019][1689447.625564] R10: ffff911b3fc07600 R11: ffffdbb283493e00 R12: 0000000000000000 [Wed Feb 6 07:42:29 2019][1689447.625565] R13: 0000000000000000 R14: 0000000000000002 R15: ffff9137f52918c0 [Wed Feb 6 07:42:29 2019][1689447.625566] FS: 00007fe2d3435880(0000) GS:ffff915a3f540000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:29 2019][1689447.625567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:29 2019][1689447.625567] CR2: 00007fe2d0bac050 CR3: 000000403a6c6000 CR4: 00000000003407e0 [Wed Feb 6 07:42:29 2019][1689447.625568] Call Trace: [Wed Feb 6 07:42:29 2019][1689447.625572] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:29 2019][1689447.625575] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:29 2019][1689447.625621] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.625654] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.625657] [] ? native_queued_spin_lock_slowpath+0x158/0x200 [Wed Feb 6 07:42:29 2019][1689447.625689] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.625728] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.625764] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.625799] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.625841] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.625878] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.625916] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.625953] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.625958] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:29 2019][1689447.625959] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:29 2019][1689447.625962] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:29 2019][1689447.625964] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:29 2019][1689447.625983] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:29 2019][1689447.695496] NMI watchdog: BUG: soft lockup - CPU#31 stuck for 22s! [ldlm_cn03_022:26176] [Wed Feb 6 07:42:29 2019][1689447.695527] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:29 2019][1689447.695541] CPU: 31 PID: 26176 Comm: ldlm_cn03_022 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:29 2019][1689447.695542] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:29 2019][1689447.695542] task: ffff9129f375d140 ti: ffff911f093c0000 task.ti: ffff911f093c0000 [Wed Feb 6 07:42:29 2019][1689447.695546] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:29 2019][1689447.695547] RSP: 0018:ffff911f093c3bb8 EFLAGS: 00000246 [Wed Feb 6 07:42:29 2019][1689447.695548] RAX: 0000000000000000 RBX: ffff911f093c3bf0 RCX: 0000000000f90000 [Wed Feb 6 07:42:29 2019][1689447.695548] RDX: ffff9149ff75b780 RSI: 0000000000b10000 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:29 2019][1689447.695549] RBP: ffff911f093c3bb8 R08: ffff915a3f5db780 R09: 0000000000000000 [Wed Feb 6 07:42:29 2019][1689447.695550] R10: ffff911b3fc07600 R11: 0000000000000000 R12: 0000000000000000 [Wed Feb 6 07:42:29 2019][1689447.695550] R13: 0000000000000000 R14: 0000000000000002 R15: ffff911b9336c380 [Wed Feb 6 07:42:29 2019][1689447.695551] FS: 00007f0ff898a880(0000) GS:ffff915a3f5c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:29 2019][1689447.695552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:29 2019][1689447.695553] CR2: 00007f9200fd81b0 CR3: 000000202de1c000 CR4: 00000000003407e0 [Wed Feb 6 07:42:29 2019][1689447.695554] Call Trace: [Wed Feb 6 07:42:29 2019][1689447.695556] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:29 2019][1689447.695558] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:29 2019][1689447.695585] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.695613] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.695615] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:29 2019][1689447.695642] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.695671] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.695701] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.695730] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.695763] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.695796] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.695828] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.695860] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.695862] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:29 2019][1689447.695863] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:29 2019][1689447.695865] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:29 2019][1689447.695866] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:29 2019][1689447.695884] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:29 2019][1689447.765497] NMI watchdog: BUG: soft lockup - CPU#39 stuck for 22s! [ldlm_cn03_011:23312] [Wed Feb 6 07:42:29 2019][1689447.765524] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:29 2019][1689447.765537] CPU: 39 PID: 23312 Comm: ldlm_cn03_011 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:29 2019][1689447.765538] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:29 2019][1689447.765539] task: ffff9159a4beb0c0 ti: ffff915929b50000 task.ti: ffff915929b50000 [Wed Feb 6 07:42:29 2019][1689447.765543] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:29 2019][1689447.765544] RSP: 0018:ffff915929b53bb8 EFLAGS: 00000246 [Wed Feb 6 07:42:29 2019][1689447.765544] RAX: 0000000000000000 RBX: ffff915929b53bf0 RCX: 0000000001390000 [Wed Feb 6 07:42:29 2019][1689447.765545] RDX: ffff915a3f55b780 RSI: 0000000000b90000 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:29 2019][1689447.765546] RBP: ffff915929b53bb8 R08: ffff915a3f65b780 R09: 0000000000000000 [Wed Feb 6 07:42:29 2019][1689447.765547] R10: ffff911b3fc07600 R11: 0000000000000000 R12: 0000000000000000 [Wed Feb 6 07:42:29 2019][1689447.765547] R13: 0000000000000000 R14: 0000000000000002 R15: ffff913f9a6fd340 [Wed Feb 6 07:42:29 2019][1689447.765549] FS: 00007f72eaa7c700(0000) GS:ffff915a3f640000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:29 2019][1689447.765550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:29 2019][1689447.765550] CR2: 00007f2781328000 CR3: 000000403bd1e000 CR4: 00000000003407e0 [Wed Feb 6 07:42:29 2019][1689447.765551] Call Trace: [Wed Feb 6 07:42:29 2019][1689447.765554] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:29 2019][1689447.765556] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:29 2019][1689447.765588] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.765621] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.765623] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:29 2019][1689447.765656] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.765691] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.765727] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.765761] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.765800] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:29 2019][1689447.765839] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689447.765877] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689447.765915] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689447.765917] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:30 2019][1689447.765918] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:30 2019][1689447.765920] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:30 2019][1689447.765922] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:30 2019][1689447.765942] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:30 2019][1689449.769213] { 44} (t=66324 jiffies g=134910431 c=134910430 q=65605) [Wed Feb 6 07:42:30 2019][1689449.775912] Task dump for CPU 5: [Wed Feb 6 07:42:30 2019][1689449.779318] ldlm_cn01_028 R running task 0 28329 2 0x00000088 [Wed Feb 6 07:42:30 2019][1689449.786590] Call Trace: [Wed Feb 6 07:42:30 2019][1689449.789274] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.796357] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.803616] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.811623] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:30 2019][1689449.818059] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:30 2019][1689449.824100] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.830749] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.838315] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:30 2019][1689449.843539] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:30 2019][1689449.849808] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:30 2019][1689449.856591] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:30 2019][1689449.862857] Task dump for CPU 6: [Wed Feb 6 07:42:30 2019][1689449.866264] ldlm_cn02_015 R running task 0 23318 2 0x00000088 [Wed Feb 6 07:42:30 2019][1689449.873535] Call Trace: [Wed Feb 6 07:42:30 2019][1689449.876167] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:30 2019][1689449.883741] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:30 2019][1689449.890360] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:30 2019][1689449.896222] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.903129] [] ? ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.910354] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:30 2019][1689449.917950] [] ? ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.924861] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.932113] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.939193] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.946451] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.954486] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.961747] [] ? ptlrpc_nrs_req_add_nolock+0x6a/0x1d0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.969436] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.976082] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689449.983649] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:30 2019][1689449.988875] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:30 2019][1689449.995140] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:30 2019][1689450.001927] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:30 2019][1689450.008192] Task dump for CPU 16: [Wed Feb 6 07:42:30 2019][1689450.011685] ldlm_cn00_024 R running task 0 20043 2 0x00000088 [Wed Feb 6 07:42:30 2019][1689450.018957] Call Trace: [Wed Feb 6 07:42:30 2019][1689450.021592] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:30 2019][1689450.028292] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:30 2019][1689450.035859] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:30 2019][1689450.042471] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:30 2019][1689450.048332] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.055233] [] ? ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.062448] [] ? native_queued_spin_lock_slowpath+0x15e/0x200 [Wed Feb 6 07:42:30 2019][1689450.070045] [] ? ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.076954] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.084209] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.091290] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.098547] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.106581] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.113575] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.120222] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.127787] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:30 2019][1689450.133014] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:30 2019][1689450.139281] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:30 2019][1689450.146064] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:30 2019][1689450.152332] Task dump for CPU 17: [Wed Feb 6 07:42:30 2019][1689450.155825] ldlm_cn01_023 R running task 0 28088 2 0x00000088 [Wed Feb 6 07:42:30 2019][1689450.163094] Call Trace: [Wed Feb 6 07:42:30 2019][1689450.165727] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:30 2019][1689450.172442] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:30 2019][1689450.179572] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:30 2019][1689450.187138] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:30 2019][1689450.193752] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:30 2019][1689450.199614] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.206512] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.213414] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.220670] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.227749] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.235005] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.243039] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.249999] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:30 2019][1689450.256439] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:30 2019][1689450.262481] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.269129] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.276692] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:30 2019][1689450.281919] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:30 2019][1689450.288186] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:30 2019][1689450.294970] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:30 2019][1689450.301237] Task dump for CPU 34: [Wed Feb 6 07:42:30 2019][1689450.304731] ldlm_cn02_023 R running task 0 28934 2 0x00000088 [Wed Feb 6 07:42:30 2019][1689450.311999] Call Trace: [Wed Feb 6 07:42:30 2019][1689450.314633] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 07:42:30 2019][1689450.321343] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [Wed Feb 6 07:42:30 2019][1689450.328478] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:30 2019][1689450.336043] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:30 2019][1689450.342658] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:30 2019][1689450.348520] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.355428] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.362339] [] ? ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.369592] [] ? ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.376673] [] ? ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.383929] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.391964] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.399218] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.405866] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.413431] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:42:30 2019][1689450.418655] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:30 2019][1689450.424924] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:30 2019][1689450.431708] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:30 2019][1689450.437973] Task dump for CPU 44: [Wed Feb 6 07:42:30 2019][1689450.441465] ldlm_cn00_028 R running task 0 20558 2 0x00000088 [Wed Feb 6 07:42:30 2019][1689450.448738] Call Trace: [Wed Feb 6 07:42:30 2019][1689450.451367] [] sched_show_task+0xa8/0x110 [Wed Feb 6 07:42:30 2019][1689450.457831] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 07:42:30 2019][1689450.463406] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 07:42:30 2019][1689450.469497] [] rcu_check_callbacks+0x442/0x730 [Wed Feb 6 07:42:30 2019][1689450.475765] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 07:42:30 2019][1689450.482031] [] update_process_times+0x46/0x80 [Wed Feb 6 07:42:30 2019][1689450.488208] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 07:42:30 2019][1689450.494127] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 07:42:30 2019][1689450.499961] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 07:42:30 2019][1689450.506228] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 07:42:30 2019][1689450.512237] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 07:42:30 2019][1689450.518933] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 07:42:30 2019][1689450.525460] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 07:42:30 2019][1689450.531810] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:30 2019][1689450.540010] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:30 2019][1689450.546451] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:30 2019][1689450.552140] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.558865] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.565906] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:30 2019][1689450.573505] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.580240] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.587322] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.594229] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.601313] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.609172] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.616166] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.622640] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:30 2019][1689450.630207] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:30 2019][1689450.635260] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:30 2019][1689450.641526] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:30 2019][1689450.648139] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:31 2019][1689450.727354] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [Wed Feb 6 07:42:31 2019][1689450.736831] Lustre: 26053:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=49 reqQ=0 recA=24, svcEst=120, delay=6389 [Wed Feb 6 07:42:31 2019][1689450.747997] Lustre: 26053:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff9154c4eaa700 x1624732329148592/t0(0) o103->74576cb8-d6cd-cd3f-fbd2-32131b2925d8@10.9.107.19@o2ib4:162/0 lens 328/0 e 2 to 0 dl 1549467787 ref 2 fl New:H/0/ffffffff rc 0/-1 [Wed Feb 6 07:42:31 2019][1689450.779230] Lustre: 26053:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Feb 6 07:42:32 2019][1689451.835599] NMI watchdog: BUG: soft lockup - CPU#47 stuck for 22s! [ldlm_cn03_020:25319] [Wed Feb 6 07:42:32 2019][1689451.843854] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:32 2019][1689451.950625] CPU: 47 PID: 25319 Comm: ldlm_cn03_020 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:32 2019][1689451.963389] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:32 2019][1689451.971130] task: ffff9139f37630c0 ti: ffff91319b3ec000 task.ti: ffff91319b3ec000 [Wed Feb 6 07:42:32 2019][1689451.978782] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x128/0x200 [Wed Feb 6 07:42:32 2019][1689451.988888] RSP: 0018:ffff91319b3efc38 EFLAGS: 00000246 [Wed Feb 6 07:42:32 2019][1689451.994375] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001790000 [Wed Feb 6 07:42:32 2019][1689452.001679] RDX: ffff9149ff79b780 RSI: 0000000000d10001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:32 2019][1689452.008986] RBP: ffff91319b3efc38 R08: ffff915a3f6db780 R09: 0000000000000000 [Wed Feb 6 07:42:32 2019][1689452.016291] R10: ffff911b3fc07600 R11: ffffdbb29c73aa00 R12: ffff91319b3efbe0 [Wed Feb 6 07:42:32 2019][1689452.023597] R13: ffff91273b6849c0 R14: ffff91319b3efba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:42:32 2019][1689452.030905] FS: 00007f72eaa8d700(0000) GS:ffff915a3f6c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:32 2019][1689452.039164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:32 2019][1689452.045084] CR2: 00007effe6f18248 CR3: 000000403bd1e000 CR4: 00000000003407e0 [Wed Feb 6 07:42:32 2019][1689452.052389] Call Trace: [Wed Feb 6 07:42:32 2019][1689452.055019] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:32 2019][1689452.061458] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:32 2019][1689452.067157] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:32 2019][1689452.073890] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:32 2019][1689452.080629] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:32 2019][1689452.087707] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:32 2019][1689452.094615] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:32 2019][1689452.101700] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:32 2019][1689452.109557] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:32 2019][1689452.116552] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:32 2019][1689452.123774] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:42:32 2019][1689452.129553] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:32 2019][1689452.136026] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:32 2019][1689452.143593] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:32 2019][1689452.148644] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:32 2019][1689452.154912] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:32 2019][1689452.161523] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:32 2019][1689452.167790] Code: 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 85 c0 <74> f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 [Wed Feb 6 07:42:33 2019][1689452.771871] LustreError: 6509:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST000f-osc-MDT0002: cannot cleanup orphans: rc = -11 [Wed Feb 6 07:42:34 2019][1689454.669503] Lustre: 28330:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-11s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff913c24d48f00 x1624697938168560/t0(0) o103->62faecbe-6e06-a70f-dde9-8810f97cf707@10.9.103.17@o2ib4:156/0 lens 328/0 e 2 to 0 dl 1549467781 ref 2 fl New:H/0/ffffffff rc 0/-1 [Wed Feb 6 07:42:35 2019][1689454.700826] Lustre: 28330:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages [Wed Feb 6 07:42:35 2019][1689455.591699] NMI watchdog: BUG: soft lockup - CPU#19 stuck for 24s! [ldlm_cn03_026:28521] [Wed Feb 6 07:42:35 2019][1689455.599961] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel[1689455.651699] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 23s! [ldlm_cn02_021:23326] [Wed Feb 6 07:42:35 2019][1689455.651701] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:36 2019][1689455.651770] CPU: 26 PID: 23326 Comm: ldlm_cn02_021 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:36 2019][1689455.651771] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:36 2019][1689455.651773] task: ffff91599ff9d140 ti: ffff9159aa650000 task.ti: ffff9159aa650000 [Wed Feb 6 07:42:36 2019][1689455.651774] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:36 2019][1689455.651781] RSP: 0018:ffff9159aa653bb8 EFLAGS: 00000246 [Wed Feb 6 07:42:36 2019][1689455.651782] RAX: 0000000000000000 RBX: ffff9159aa653bf0 RCX: 0000000000d10000 [Wed Feb 6 07:42:36 2019][1689455.651783] RDX: ffff9129ff01b780 RSI: 0000000001010001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:36 2019][1689455.651784] RBP: ffff9159aa653bb8 R08: ffff9149ff79b780 R09: 0000000000000000 [Wed Feb 6 07:42:36 2019][1689455.651784] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: 0000000000000000 [Wed Feb 6 07:42:36 2019][1689455.651785] R13: 0000000000000000 R14: 0000000000000002 R15: ffff9136a9447980 [Wed Feb 6 07:42:36 2019][1689455.651786] FS: 00007f35005b3880(0000) GS:ffff9149ff780000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:36 2019][1689455.651787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:36 2019][1689455.651788] CR2: 00007fe833b97ff0 CR3: 0000001035820000 CR4: 00000000003407e0 [Wed Feb 6 07:42:36 2019][1689455.651790] Call Trace: [Wed Feb 6 07:42:36 2019][1689455.651795] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:36 2019][1689455.651799] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:36 2019][1689455.651841] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.651874] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.651877] [] ? native_queued_spin_lock_slowpath+0x156/0x200 [Wed Feb 6 07:42:36 2019][1689455.651909] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.651946] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.651982] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.652017] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.652058] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.652096] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.652134] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.652171] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.652175] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:36 2019][1689455.652177] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:36 2019][1689455.652180] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:36 2019][1689455.652181] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:36 2019][1689455.652182] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:42:36 2019][1689455.756699] NMI watchdog: BUG: soft lockup - CPU#38 stuck for 23s! [ldlm_cn02_008:9191] [Wed Feb 6 07:42:36 2019][1689455.756700] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:36 2019][1689455.756774] CPU: 38 PID: 9191 Comm: ldlm_cn02_008 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:36 2019][1689455.756775] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:36 2019][1689455.756776] task: ffff9159ad54b0c0 ti: ffff9159a6244000 task.ti: ffff9159a6244000 [Wed Feb 6 07:42:36 2019][1689455.756777] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x128/0x200 [Wed Feb 6 07:42:36 2019][1689455.756785] RSP: 0018:ffff9159a6247c38 EFLAGS: 00000246 [Wed Feb 6 07:42:36 2019][1689455.756786] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001310000 [Wed Feb 6 07:42:36 2019][1689455.756787] RDX: ffff915a3f41b780 RSI: 0000000000190001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:36 2019][1689455.756788] RBP: ffff9159a6247c38 R08: ffff9149ff85b780 R09: 0000000000000000 [Wed Feb 6 07:42:36 2019][1689455.756788] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: ffff9159a6247be0 [Wed Feb 6 07:42:36 2019][1689455.756789] R13: ffff91273b6849c0 R14: ffff9159a6247ba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:42:36 2019][1689455.756791] FS: 00007f5b92ec7740(0000) GS:ffff9149ff840000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:36 2019][1689455.756791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:36 2019][1689455.756792] CR2: 00007f5b929db1e0 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:42:36 2019][1689455.756793] Call Trace: [Wed Feb 6 07:42:36 2019][1689455.756800] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:36 2019][1689455.756804] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:36 2019][1689455.756847] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.756881] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.756921] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.756957] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.756992] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.757035] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.757074] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.757112] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.757151] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689455.757154] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:36 2019][1689455.757156] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:36 2019][1689455.757159] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:36 2019][1689455.757161] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:36 2019][1689455.757161] Code: 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 85 c0 <74> f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 [Wed Feb 6 07:42:36 2019] [Wed Feb 6 07:42:36 2019][1689456.346477] aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:36 2019][1689456.402526] CPU: 19 PID: 28521 Comm: ldlm_cn03_026 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:36 2019][1689456.415291] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:36 2019][1689456.423031] task: ffff911a29b7b0c0 ti: ffff9150e6bbc000 task.ti: ffff9150e6bbc000 [Wed Feb 6 07:42:36 2019][1689456.430684] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:36 2019][1689456.440798] RSP: 0018:ffff9150e6bbfbb8 EFLAGS: 00000246 [Wed Feb 6 07:42:36 2019][1689456.446284] RAX: 0000000000000000 RBX: ffff9150e6bbfbf0 RCX: 0000000000990000 [Wed Feb 6 07:42:36 2019][1689456.453588] RDX: ffff9139ff6db780 RSI: 0000000000690000 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:36 2019][1689456.460895] RBP: ffff9150e6bbfbb8 R08: ffff915a3f51b780 R09: 0000000000000000 [Wed Feb 6 07:42:36 2019][1689456.468202] R10: ffff911b3fc07600 R11: ffffdbb2d6a69000 R12: 0000000000000000 [Wed Feb 6 07:42:36 2019][1689456.475508] R13: 0000000000000000 R14: 0000000000000002 R15: ffff912ee9414ec0 [Wed Feb 6 07:42:36 2019][1689456.482815] FS: 00007fe848361880(0000) GS:ffff915a3f500000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:36 2019][1689456.491074] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:36 2019][1689456.496994] CR2: 000055fdd41372a8 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:42:36 2019][1689456.504300] Call Trace: [Wed Feb 6 07:42:36 2019][1689456.506930] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:36 2019][1689456.513368] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:36 2019][1689456.519067] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689456.525795] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689456.532840] [] ? native_queued_spin_lock_slowpath+0x156/0x200 [Wed Feb 6 07:42:36 2019][1689456.540432] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689456.547165] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689456.554242] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689456.561139] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689456.568220] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689456.576075] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689456.583062] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689456.589534] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:36 2019][1689456.597100] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:36 2019][1689456.602148] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:36 2019][1689456.608417] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:36 2019][1689456.615028] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:36 2019][1689456.621294] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:42:37 2019][1689456.789008] LustreError: 6481:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0015-osc-MDT0002: cannot cleanup orphans: rc = -11 [Wed Feb 6 07:42:37 2019][1689457.519752] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [Wed Feb 6 07:42:37 2019][1689457.529235] Lustre: Skipped 3 previous similar messages [Wed Feb 6 07:42:37 2019][1689457.534643] Lustre: 14886:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=31, svcEst=145, delay=0 [Wed Feb 6 07:42:37 2019][1689457.545076] Lustre: 14886:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 3 previous similar messages [Wed Feb 6 07:42:40 2019][1689459.707934] LustreError: 6521:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0025-osc-MDT0002: cannot cleanup orphans: rc = -11 [Wed Feb 6 07:42:41 2019][1689461.020208] LustreError: 6249:0:(osp_precreate.c:656:osp_precreate_send()) fir-OST000c-osc-MDT0000: can't precreate: rc = -11 [Wed Feb 6 07:42:41 2019][1689461.031694] LustreError: 6249:0:(osp_precreate.c:1312:osp_precreate_thread()) fir-OST000c-osc-MDT0000: cannot precreate objects: rc = -11 [Wed Feb 6 07:42:43 2019][1689463.522895] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 21s! [ldlm_elt:6043] [Wed Feb 6 07:42:43 2019][1689463.530640] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:43 2019][1689463.637410] CPU: 11 PID: 6043 Comm: ldlm_elt Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:43 2019][1689463.649659] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:43 2019][1689463.657399] task: ffff9139f615b0c0 ti: ffff9159f1704000 task.ti: ffff9159f1704000 [Wed Feb 6 07:42:43 2019][1689463.665054] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:43 2019][1689463.675160] RSP: 0018:ffff9159f1707de0 EFLAGS: 00000246 [Wed Feb 6 07:42:43 2019][1689463.680645] RAX: 0000000000000000 RBX: ffff9139f8e6c4c0 RCX: 0000000000590000 [Wed Feb 6 07:42:43 2019][1689463.687950] RDX: ffff9149ff8db780 RSI: 0000000001710001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:43 2019][1689463.695256] RBP: ffff9159f1707de0 R08: ffff915a3f49b780 R09: 0000000000000000 [Wed Feb 6 07:42:43 2019][1689463.702563] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9159f1707dd0 [Wed Feb 6 07:42:43 2019][1689463.709870] R13: ffffffffc0e6d2f7 R14: ffff9159f1707d50 R15: ffffffffc0f69020 [Wed Feb 6 07:42:43 2019][1689463.717175] FS: 00007fe848361880(0000) GS:ffff915a3f480000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:44 2019][1689463.725434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:44 2019][1689463.731355] CR2: 00007fe837a53ff0 CR3: 000000202e3fe000 CR4: 00000000003407e0 [Wed Feb 6 07:42:44 2019][1689463.738661] Call Trace: [Wed Feb 6 07:42:44 2019][1689463.741290] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:44 2019][1689463.747729] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:44 2019][1689463.753435] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:44 2019][1689463.760175] [] expired_lock_main+0x8d1/0xc00 [ptlrpc] [Wed Feb 6 07:42:44 2019][1689463.767048] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:42:44 2019][1689463.772824] [] ? ldlm_server_glimpse_ast+0x3f0/0x3f0 [ptlrpc] [Wed Feb 6 07:42:44 2019][1689463.780395] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:44 2019][1689463.785444] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:44 2019][1689463.791714] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:44 2019][1689463.798325] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:44 2019][1689463.804588] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:42:44 2019][1689464.337019] Lustre: fir-MDT0002: Received new LWP connection from 0@lo, removing former export from same NID [Wed Feb 6 07:42:44 2019][1689464.347018] Lustre: Skipped 8 previous similar messages [Wed Feb 6 07:42:46 2019][1689466.494324] Lustre: 27821:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-2s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff913872933300 x1624711630445184/t0(0) o103->bd6b0907-bbf0-754e-ba62-411999a5fe50@10.8.15.1@o2ib6:177/0 lens 328/192 e 3 to 0 dl 1549467802 ref 1 fl Complete:/0/0 rc 0/0 [Wed Feb 6 07:42:46 2019][1689466.525217] Lustre: 27821:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 55 previous similar messages [Wed Feb 6 07:42:47 2019][1689467.036147] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [Wed Feb 6 07:42:47 2019][1689467.045629] Lustre: 21075:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=3 reqQ=0 recA=30, svcEst=146, delay=73 [Wed Feb 6 07:42:47 2019][1689467.056239] LustreError: 21075:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.8.23@o2ib6: deadline 120:3s ago [Wed Feb 6 07:42:47 2019][1689467.056239] req@ffff91338c5fef00 x1624712606996944/t0(0) o103->9633991e-ce4f-d92c-b6aa-ec983a0f2b80@10.8.8.23@o2ib6:177/0 lens 328/0 e 3 to 0 dl 1549467802 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Feb 6 07:42:47 2019][1689467.088260] Lustre: 21075:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (120:3s); client may timeout. req@ffff91338c5fef00 x1624712606996944/t0(0) o103->9633991e-ce4f-d92c-b6aa-ec983a0f2b80@10.8.8.23@o2ib6:177/0 lens 328/0 e 3 to 0 dl 1549467802 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Feb 6 07:42:47 2019][1689467.182050] LustreError: 27457:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(131072) req@ffff9159a0614050 x1624699308056592/t0(0) o4->e0d0d81e-4164-f236-7d8a-b466fb3eea50@10.9.104.65@o2ib4:205/0 lens 488/448 e 3 to 0 dl 1549467830 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 07:42:47 2019][1689467.207165] Lustre: fir-MDT0002: Bulk IO write error with e0d0d81e-4164-f236-7d8a-b466fb3eea50 (at 10.9.104.65@o2ib4), client will retry: rc = -110 [Wed Feb 6 07:42:47 2019][1689467.565996] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 23s! [ldlm_cn00_024:20043] [Wed Feb 6 07:42:47 2019][1689467.574258] Modules linked in:[1689467.574999] NMI watchdog: BUG: soft lockup - CPU#17 stuck for 23s! [ldlm_cn01_023:28088] [Wed Feb 6 07:42:47 2019][1689467.575001] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:47 2019][1689467.575077] CPU: 17 PID: 28088 Comm: ldlm_cn01_023 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:47 2019][1689467.575078] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:47 2019][1689467.575079] task: ffff915929b4c100 ti: ffff91578b654000 task.ti: ffff91578b654000 [Wed Feb 6 07:42:47 2019][1689467.575080] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:48 2019][1689467.575088] RSP: 0018:ffff91578b657c38 EFLAGS: 00000246 [Wed Feb 6 07:42:48 2019][1689467.575089] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000890000 [Wed Feb 6 07:42:48 2019][1689467.575090] RDX: ffff915a3f61b780 RSI: 0000000001190001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:48 2019][1689467.575090] RBP: ffff91578b657c38 R08: ffff9139ff71b780 R09: 0000000000000000 [Wed Feb 6 07:42:48 2019][1689467.575091] R10: ffff911b3fc07600 R11: ffff913957f6a660 R12: ffff91578b657be0 [Wed Feb 6 07:42:48 2019][1689467.575092] R13: ffff91273b6849c0 R14: ffff91578b657ba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:42:48 2019][1689467.575093] FS: 00007f790e18c700(0000) GS:ffff9139ff700000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:48 2019][1689467.575094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:48 2019][1689467.575095] CR2: 00007f2781326000 CR3: 0000001028b2a000 CR4: 00000000003407e0 [Wed Feb 6 07:42:48 2019][1689467.575096] Call Trace: [Wed Feb 6 07:42:48 2019][1689467.575101] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:48 2019][1689467.575105] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:48 2019][1689467.575150] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.575184] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.575221] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.575257] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.575293] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.575332] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.575370] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.575374] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:42:48 2019][1689467.575376] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:42:48 2019][1689467.575414] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.575451] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.575455] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:48 2019][1689467.575457] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:48 2019][1689467.575459] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:48 2019][1689467.575461] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:48 2019][1689467.575461] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:48 2019][1689467.722002] NMI watchdog: BUG: soft lockup - CPU#34 stuck for 22s! [ldlm_cn02_023:28934] [Wed Feb 6 07:42:48 2019][1689467.722003] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:48 2019][1689467.722078] CPU: 34 PID: 28934 Comm: ldlm_cn02_023 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:48 2019][1689467.722079] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:48 2019][1689467.722081] task: ffff91488911a080 ti: ffff9156a4e78000 task.ti: ffff9156a4e78000 [Wed Feb 6 07:42:48 2019][1689467.722082] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:48 2019][1689467.722090] RSP: 0000:ffff9156a4e7bbb8 EFLAGS: 00000246 [Wed Feb 6 07:42:48 2019][1689467.722091] RAX: 0000000000000000 RBX: ffff9156a4e7bbf0 RCX: 0000000001110000 [Wed Feb 6 07:42:48 2019][1689467.722091] RDX: ffff9129fee1b780 RSI: 0000000000010000 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:48 2019][1689467.722092] RBP: ffff9156a4e7bbb8 R08: ffff9149ff81b780 R09: 0000000000000000 [Wed Feb 6 07:42:48 2019][1689467.722093] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: 0000000000000000 [Wed Feb 6 07:42:48 2019][1689467.722093] R13: 0000000000000000 R14: 0000000000000002 R15: ffff911e5981dc40 [Wed Feb 6 07:42:48 2019][1689467.722095] FS: 00007f7c8de93740(0000) GS:ffff9149ff800000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:48 2019][1689467.722096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:48 2019][1689467.722097] CR2: 00007f7c8d92f210 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:42:48 2019][1689467.722098] Call Trace: [Wed Feb 6 07:42:48 2019][1689467.722103] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:48 2019][1689467.722107] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:48 2019][1689467.722151] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.722185] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.722187] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:48 2019][1689467.722220] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.722257] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.722293] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.722329] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.722370] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.722407] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.722445] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.722482] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689467.722486] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:48 2019][1689467.722488] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:48 2019][1689467.722491] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:48 2019][1689467.722492] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:48 2019][1689467.722493] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:48 2019] [Wed Feb 6 07:42:48 2019][1689468.283784] osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:48 2019][1689468.389167] CPU: 16 PID: 20043 Comm: ldlm_cn00_024 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:48 2019][1689468.401930] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:48 2019][1689468.409669] task: ffff9139f700a080 ti: ffff9158ffab0000 task.ti: ffff9158ffab0000 [Wed Feb 6 07:42:48 2019][1689468.417321] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:48 2019][1689468.427437] RSP: 0018:ffff9158ffab3c38 EFLAGS: 00000246 [Wed Feb 6 07:42:48 2019][1689468.432923] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000810000 [Wed Feb 6 07:42:48 2019][1689468.440229] RDX: ffff915a3f6db780 RSI: 0000000001790001 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:48 2019][1689468.447535] RBP: ffff9158ffab3c38 R08: ffff9129fef1b780 R09: 0000000000000000 [Wed Feb 6 07:42:48 2019][1689468.454841] R10: ffff911b3fc07600 R11: ffffdbb268b18400 R12: ffff9158ffab3be0 [Wed Feb 6 07:42:48 2019][1689468.462146] R13: ffff91273b6849c0 R14: ffff9158ffab3ba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:42:48 2019][1689468.469454] FS: 00007fe2d3435880(0000) GS:ffff9129fef00000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:48 2019][1689468.477713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:48 2019][1689468.483633] CR2: 00007fe2d0ba8050 CR3: 000000403a6c6000 CR4: 00000000003407e0 [Wed Feb 6 07:42:48 2019][1689468.490938] Call Trace: [Wed Feb 6 07:42:48 2019][1689468.493569] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:48 2019][1689468.500009] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:48 2019][1689468.505719] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689468.512448] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689468.519186] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689468.526265] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689468.533172] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689468.540260] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689468.548126] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689468.555120] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689468.561595] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:48 2019][1689468.569161] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:48 2019][1689468.574212] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:48 2019][1689468.580478] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:48 2019][1689468.587089] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:48 2019][1689468.593354] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:42:50 2019][1689470.146846] LustreError: 19974:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.7.15@o2ib6: deadline 120:6s ago [Wed Feb 6 07:42:50 2019][1689470.146846] req@ffff91338c5ff200 x1624711751362544/t0(0) o103->6d1355d0-7b33-677d-c8cf-a270e3061917@10.8.7.15@o2ib6:177/0 lens 328/0 e 3 to 0 dl 1549467802 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Feb 6 07:42:50 2019][1689470.512287] LustreError: 15458:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(131072) req@ffff912643022450 x1624699308055440/t0(0) o4->e0d0d81e-4164-f236-7d8a-b466fb3eea50@10.9.104.65@o2ib4:199/0 lens 488/448 e 3 to 0 dl 1549467824 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 07:42:50 2019][1689470.537404] Lustre: fir-MDT0002: Bulk IO write error with e0d0d81e-4164-f236-7d8a-b466fb3eea50 (at 10.9.104.65@o2ib4), client will retry: rc = -110 [Wed Feb 6 07:42:52 2019][1689471.750601] LustreError: 6291:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0015-osc-MDT0000: cannot cleanup orphans: rc = -11 [Wed Feb 6 07:42:52 2019][1689472.333957] LustreError: 27821:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.7.13@o2ib6: deadline 120:8s ago [Wed Feb 6 07:42:52 2019][1689472.333957] req@ffff91338c5ff500 x1624711742968752/t0(0) o103->a8b0e693-902f-1f36-293b-a7b32f871782@10.8.7.13@o2ib6:177/0 lens 328/0 e 3 to 0 dl 1549467802 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Feb 6 07:42:55 2019][1689475.253322] LNetError: 16288:0:(o2iblnd_cb.c:3281:kiblnd_cm_callback()) 10.0.10.212@o2ib7 DISCONNECTED [Wed Feb 6 07:42:55 2019][1689475.696200] NMI watchdog: BUG: soft lockup - CPU#31 stuck for 22s! [ldlm_cn03_022:26176] [Wed Feb 6 07:42:55 2019][1689475.704465] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs][1689475.810205] NMI watchdog: BUG: soft lockup - CPU#44 stuck for 23s! [ldlm_cn00_028:20558] [Wed Feb 6 07:42:56 2019][1689475.810207] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:42:56 2019][1689475.810295] CPU: 44 PID: 20558 Comm: ldlm_cn00_028 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:56 2019][1689475.810296] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:56 2019][1689475.810297] task: ffff9146d2b8a080 ti: ffff9159f5ffc000 task.ti: ffff9159f5ffc000 [Wed Feb 6 07:42:56 2019][1689475.810299] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:42:56 2019][1689475.810308] RSP: 0018:ffff9159f5fffbb8 EFLAGS: 00000246 [Wed Feb 6 07:42:56 2019][1689475.810309] RAX: 0000000000000000 RBX: ffff9159f5fffbf0 RCX: 0000000001610000 [Wed Feb 6 07:42:56 2019][1689475.810310] RDX: ffff9129fee9b780 RSI: 0000000000410000 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:56 2019][1689475.810311] RBP: ffff9159f5fffbb8 R08: ffff9129ff0db780 R09: 0000000000000000 [Wed Feb 6 07:42:56 2019][1689475.810312] R10: ffff911b3fc07600 R11: ffffdbb24dcd6c00 R12: 0000000000000000 [Wed Feb 6 07:42:56 2019][1689475.810313] R13: 0000000000000000 R14: 0000000000000002 R15: ffff91263fde1f80 [Wed Feb 6 07:42:56 2019][1689475.810314] FS: 00007f63fb26e700(0000) GS:ffff9129ff0c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:56 2019][1689475.810316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:56 2019][1689475.810317] CR2: 00007f01048dc1a0 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:42:56 2019][1689475.810318] Call Trace: [Wed Feb 6 07:42:56 2019][1689475.810325] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:56 2019][1689475.810329] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:56 2019][1689475.810386] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689475.810423] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689475.810427] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:56 2019][1689475.810462] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689475.810503] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689475.810542] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689475.810581] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689475.810626] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689475.810666] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689475.810708] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689475.810750] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689475.810754] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:56 2019][1689475.810756] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:56 2019][1689475.810761] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:56 2019][1689475.810764] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:56 2019][1689475.810764] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:42:56 2019] [Wed Feb 6 07:42:56 2019][1689476.165212] [Wed Feb 6 07:42:56 2019][1689476.167086] CPU: 31 PID: 26176 Comm: ldlm_cn03_022 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:42:56 2019][1689476.179851] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:42:56 2019][1689476.187592] task: ffff9129f375d140 ti: ffff911f093c0000 task.ti: ffff911f093c0000 [Wed Feb 6 07:42:56 2019][1689476.195243] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:56 2019][1689476.205349] RSP: 0018:ffff911f093c3bb8 EFLAGS: 00000246 [Wed Feb 6 07:42:56 2019][1689476.210836] RAX: 0000000000000000 RBX: ffff911f093c3bf0 RCX: 0000000000f90000 [Wed Feb 6 07:42:56 2019][1689476.218143] RDX: ffff9129feedb780 RSI: 0000000000610000 RDI: ffff911e813fa6dc [Wed Feb 6 07:42:56 2019][1689476.225448] RBP: ffff911f093c3bb8 R08: ffff915a3f5db780 R09: 0000000000000000 [Wed Feb 6 07:42:56 2019][1689476.232752] R10: ffff911b3fc07600 R11: 0000000000000000 R12: 0000000000000000 [Wed Feb 6 07:42:56 2019][1689476.240059] R13: 0000000000000000 R14: 0000000000000002 R15: ffff913873f3de80 [Wed Feb 6 07:42:56 2019][1689476.247365] FS: 00007f0ff898a880(0000) GS:ffff915a3f5c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:42:56 2019][1689476.255626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:42:56 2019][1689476.261545] CR2: 00007f9200fd81b0 CR3: 000000202de1c000 CR4: 00000000003407e0 [Wed Feb 6 07:42:56 2019][1689476.268852] Call Trace: [Wed Feb 6 07:42:56 2019][1689476.271480] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:42:56 2019][1689476.277919] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:42:56 2019][1689476.283616] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689476.290337] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689476.297385] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:42:56 2019][1689476.304976] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689476.311705] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689476.318777] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689476.325675] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689476.332752] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689476.340610] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689476.347605] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689476.354080] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:42:56 2019][1689476.361648] [] kthread+0xd1/0xe0 [Wed Feb 6 07:42:56 2019][1689476.366700] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:56 2019][1689476.372968] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:42:56 2019][1689476.379581] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:42:56 2019][1689476.385846] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:42:56 2019][1689476.492264] LustreError: 8616:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.7.17@o2ib6: deadline 120:12s ago [Wed Feb 6 07:42:56 2019][1689476.492264] req@ffff91382fce6600 x1624711748232032/t0(0) o103->0fd04801-8b1d-0b4d-67be-94bc61a5dfe9@10.8.7.17@o2ib6:177/0 lens 328/0 e 3 to 0 dl 1549467802 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Feb 6 07:42:56 2019][1689476.524280] LustreError: 8616:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 1 previous similar message [Wed Feb 6 07:42:57 2019][1689477.513059] LNetError: 5890:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) [Wed Feb 6 07:43:00 2019][1689479.836305] NMI watchdog: BUG: soft lockup - CPU#47 stuck for 22s! [ldlm_cn03_020:25319] [Wed Feb 6 07:43:00 2019][1689479.844565] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE)[1689479.920634] Lustre: 28512:0:(service.c:1322:peply()) @@@ Already past deadline (-92s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff913f3af7d100 x1624697938130960/t0(0) o103->62faecbe-6e06-a70f-dde9-8810f97cf707@10.9.103.17@o2ib4:101/0 lens 328/0 e 0 to 0 dl 1549467726 ref 1 fl Interpret:H/0/ffffffff rc 0/-1 [Wed Feb 6 07:43:00 2019][1689479.920637] Lustre: 28512:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 25 previous similar messages [Wed Feb 6 07:43:00 2019][1689479.921723] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [Wed Feb 6 07:43:00 2019][1689479.921727] Lustre: 28512:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=0 reqQ=0 recA=26, svcEst=146, delay=1 [Wed Feb 6 07:43:00 2019] ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:43:00 2019][1689480.013232] CPU: 47 PID: 25319 Comm: ldlm_cn03_020 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:43:00 2019][1689480.025997] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:43:00 2019][1689480.033738] task: ffff9139f37630c0 ti: ffff91319b3ec000 task.ti: ffff91319b3ec000 [Wed Feb 6 07:43:00 2019][1689480.041390] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:43:00 2019][1689480.051496] RSP: 0018:ffff91319b3efbb8 EFLAGS: 00000246 [Wed Feb 6 07:43:00 2019][1689480.056983] RAX: 0000000000000000 RBX: ffff91319b3efbf0 RCX: 0000000001790000 [Wed Feb 6 07:43:00 2019][1689480.064288] RDX: ffff9149ff7db780 RSI: 0000000000f10000 RDI: ffff911e813fa6dc [Wed Feb 6 07:43:00 2019][1689480.071594] RBP: ffff91319b3efbb8 R08: ffff915a3f6db780 R09: 0000000000000000 [Wed Feb 6 07:43:00 2019][1689480.078901] R10: ffff911b3fc07600 R11: ffffdbb29c73aa00 R12: 0000000000000000 [Wed Feb 6 07:43:00 2019][1689480.086207] R13: 0000000000000000 R14: 0000000000000002 R15: ffff91263d32ee40 [Wed Feb 6 07:43:00 2019][1689480.093512] FS: 00007f72eaa8d700(0000) GS:ffff915a3f6c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:43:00 2019][1689480.101771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:43:00 2019][1689480.107693] CR2: 00007effe6f18248 CR3: 000000403bd1e000 CR4: 00000000003407e0 [Wed Feb 6 07:43:00 2019][1689480.114997] Call Trace: [Wed Feb 6 07:43:00 2019][1689480.117626] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:43:00 2019][1689480.124067] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:43:00 2019][1689480.129768] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:43:00 2019][1689480.136498] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:43:00 2019][1689480.143550] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:43:00 2019][1689480.151151] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:43:00 2019][1689480.157890] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:43:00 2019][1689480.164975] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:43:00 2019][1689480.171882] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:43:00 2019][1689480.178969] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:43:00 2019][1689480.186837] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:43:00 2019][1689480.193837] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:43:00 2019][1689480.201055] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:43:00 2019][1689480.206839] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:43:00 2019][1689480.213322] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:43:00 2019][1689480.220891] [] kthread+0xd1/0xe0 [Wed Feb 6 07:43:00 2019][1689480.225944] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:00 2019][1689480.232212] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:43:00 2019][1689480.238821] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:00 2019][1689480.245087] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:43:02 2019][1689482.697923] LNetError: 7282:0:(o2iblnd_cb.c:3281:kiblnd_cm_callback()) 10.0.10.106@o2ib7 DISCONNECTED [Wed Feb 6 07:43:03 2019][1689483.652401] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 23s! [ldlm_cn02_021:23326] [Wed Feb 6 07:43:03 2019][1689483.660665] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:43:04 2019][1689483.767433] CPU: 26 PID: 23326 Comm: ldlm_cn02_021 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:43:04 2019][1689483.780197] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:43:04 2019][1689483.787936] task: ffff91599ff9d140 ti: ffff9159aa650000 task.ti: ffff9159aa650000 [Wed Feb 6 07:43:04 2019][1689483.795590] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:43:04 2019][1689483.805703] RSP: 0018:ffff9159aa653bb8 EFLAGS: 00000246 [Wed Feb 6 07:43:04 2019][1689483.811189] RAX: 0000000000000000 RBX: ffff9159aa653bf0 RCX: 0000000000d10000 [Wed Feb 6 07:43:04 2019][1689483.818497] RDX: ffff9139ff6db780 RSI: 0000000000690000 RDI: ffff911e813fa6dc [Wed Feb 6 07:43:04 2019][1689483.825802] RBP: ffff9159aa653bb8 R08: ffff9149ff79b780 R09: 0000000000000000 [Wed Feb 6 07:43:04 2019][1689483.833109] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: 0000000000000000 [Wed Feb 6 07:43:04 2019][1689483.840413] R13: 0000000000000000 R14: 0000000000000002 R15: ffff9135178c45c0 [Wed Feb 6 07:43:04 2019][1689483.847722] FS: 00007f35005b3880(0000) GS:ffff9149ff780000(0000) knlGS:0000000000000000 [Wed Feb 6 07:43:04 2019][1689483.855981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:43:04 2019][1689483.861901] CR2: 00007fe833b97ff0 CR3: 0000001035820000 CR4: 00000000003407e0 [Wed Feb 6 07:43:04 2019][1689483.869206] Call Trace: [Wed Feb 6 07:43:04 2019][1689483.871838] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:43:04 2019][1689483.878283] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:43:04 2019][1689483.883980] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:43:04 2019][1689483.890716] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:43:04 2019][1689483.897765] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:43:04 2019][1689483.905361] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:43:04 2019][1689483.912090] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:43:04 2019][1689483.919171] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:43:04 2019][1689483.926077] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:43:04 2019][1689483.933166] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:43:04 2019][1689483.941033] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:43:04 2019][1689483.948287] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:43:04 2019][1689483.954760] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:43:04 2019][1689483.962327] [] kthread+0xd1/0xe0 [Wed Feb 6 07:43:04 2019][1689483.967376] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:04 2019][1689483.973644] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:43:04 2019][1689483.980256] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:04 2019][1689483.986519] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:43:06 2019][1689485.981975] LustreError: 6331:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0025-osc-MDT0000: cannot cleanup orphans: rc = -11 [Wed Feb 6 07:43:09 2019][1689488.901158] Lustre: fir-OST0013-osc-MDT0002: Connection to fir-OST0013 (at 10.0.10.104@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 07:43:09 2019][1689488.917402] Lustre: Skipped 34 previous similar messages [Wed Feb 6 07:43:13 2019][1689493.496048] LustreError: 21239:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.7.21@o2ib6: deadline 95:44s ago [Wed Feb 6 07:43:13 2019][1689493.496048] req@ffff911ed8928000 x1624712145642768/t0(0) o103->d6636cab-7aff-905c-7654-df6ea400308a@10.8.7.21@o2ib6:162/0 lens 328/0 e 2 to 0 dl 1549467787 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Feb 6 07:43:13 2019][1689493.528065] LustreError: 21239:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 2 previous similar messages [Wed Feb 6 07:43:15 2019][1689495.566701] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [ldlm_cn00_024:20043] [Wed Feb 6 07:43:15 2019][1689495.574960] Modules linked in:[1689495.575703] NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [ldlm_cn01_023:28088] [Wed Feb 6 07:43:15 2019][1689495.575704] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:43:15 2019][1689495.575795] CPU: 17 PID: 28088 Comm: ldlm_cn01_023 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:43:15 2019][1689495.575796] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:43:15 2019][1689495.575798] task: ffff915929b4c100 ti: ffff91578b654000 task.ti: ffff91578b654000 [Wed Feb 6 07:43:16 2019][1689495.575799] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:43:16 2019][1689495.575807] RSP: 0018:ffff91578b657c38 EFLAGS: 00000246 [Wed Feb 6 07:43:16 2019][1689495.575808] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000890000 [Wed Feb 6 07:43:16 2019][1689495.575810] RDX: ffff9139ff8db780 RSI: 0000000001690001 RDI: ffff911e813fa6dc [Wed Feb 6 07:43:16 2019][1689495.575811] RBP: ffff91578b657c38 R08: ffff9139ff71b780 R09: 0000000000000000 [Wed Feb 6 07:43:16 2019][1689495.575811] R10: ffff911b3fc07600 R11: ffff913957f6a660 R12: ffff91578b657be0 [Wed Feb 6 07:43:16 2019][1689495.575812] R13: ffff91273b6849c0 R14: ffff91578b657ba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:43:16 2019][1689495.575814] FS: 00007f790e18c700(0000) GS:ffff9139ff700000(0000) knlGS:0000000000000000 [Wed Feb 6 07:43:16 2019][1689495.575815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:43:16 2019][1689495.575816] CR2: 00007f2781326000 CR3: 0000001028b2a000 CR4: 00000000003407e0 [Wed Feb 6 07:43:16 2019][1689495.575818] Call Trace: [Wed Feb 6 07:43:16 2019][1689495.575823] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:43:16 2019][1689495.575828] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:43:16 2019][1689495.575870] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.575907] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.575948] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.575987] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.576026] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.576069] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.576111] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.576116] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 07:43:16 2019][1689495.576119] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:43:16 2019][1689495.576161] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.576203] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.576207] [] kthread+0xd1/0xe0 [Wed Feb 6 07:43:16 2019][1689495.576209] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:16 2019][1689495.576213] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:43:16 2019][1689495.576215] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:16 2019][1689495.576216] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:43:16 2019][1689495.722704] NMI watchdog: BUG: soft lockup - CPU#34 stuck for 22s! [ldlm_cn02_023:28934] [Wed Feb 6 07:43:16 2019][1689495.722705] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:43:16 2019][1689495.722776] CPU: 34 PID: 28934 Comm: ldlm_cn02_023 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:43:16 2019][1689495.722776] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:43:16 2019][1689495.722778] task: ffff91488911a080 ti: ffff9156a4e78000 task.ti: ffff9156a4e78000 [Wed Feb 6 07:43:16 2019][1689495.722779] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 07:43:16 2019][1689495.722788] RSP: 0000:ffff9156a4e7bbb8 EFLAGS: 00000246 [Wed Feb 6 07:43:16 2019][1689495.722789] RAX: 0000000000000000 RBX: ffff9156a4e7bbf0 RCX: 0000000001110000 [Wed Feb 6 07:43:16 2019][1689495.722790] RDX: ffff9129ff0db780 RSI: 0000000001610000 RDI: ffff911e813fa6dc [Wed Feb 6 07:43:16 2019][1689495.722791] RBP: ffff9156a4e7bbb8 R08: ffff9149ff81b780 R09: 0000000000000000 [Wed Feb 6 07:43:16 2019][1689495.722791] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: 0000000000000000 [Wed Feb 6 07:43:16 2019][1689495.722792] R13: 0000000000000000 R14: 0000000000000002 R15: ffff91373a1a21c0 [Wed Feb 6 07:43:16 2019][1689495.722793] FS: 00007f7c8de93740(0000) GS:ffff9149ff800000(0000) knlGS:0000000000000000 [Wed Feb 6 07:43:16 2019][1689495.722794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:43:16 2019][1689495.722795] CR2: 00007f7c8d92f210 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:43:16 2019][1689495.722796] Call Trace: [Wed Feb 6 07:43:16 2019][1689495.722801] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:43:16 2019][1689495.722805] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:43:16 2019][1689495.722847] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.722881] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.722883] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:43:16 2019][1689495.722916] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.722954] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.722990] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.723026] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.723066] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.723104] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.723142] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.723180] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689495.723183] [] kthread+0xd1/0xe0 [Wed Feb 6 07:43:16 2019][1689495.723185] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:16 2019][1689495.723188] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:43:16 2019][1689495.723189] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:16 2019][1689495.723190] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 07:43:16 2019] [Wed Feb 6 07:43:16 2019][1689496.284474] osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:43:16 2019][1689496.389857] CPU: 16 PID: 20043 Comm: ldlm_cn00_024 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:43:16 2019][1689496.402619] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:43:16 2019][1689496.410360] task: ffff9139f700a080 ti: ffff9158ffab0000 task.ti: ffff9158ffab0000 [Wed Feb 6 07:43:16 2019][1689496.418013] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:43:16 2019][1689496.428126] RSP: 0018:ffff9158ffab3c38 EFLAGS: 00000246 [Wed Feb 6 07:43:16 2019][1689496.433613] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000810000 [Wed Feb 6 07:43:16 2019][1689496.440918] RDX: ffff9139ff75b780 RSI: 0000000000a90001 RDI: ffff911e813fa6dc [Wed Feb 6 07:43:16 2019][1689496.448226] RBP: ffff9158ffab3c38 R08: ffff9129fef1b780 R09: 0000000000000000 [Wed Feb 6 07:43:16 2019][1689496.455532] R10: ffff911b3fc07600 R11: ffffdbb268b18400 R12: ffff9158ffab3be0 [Wed Feb 6 07:43:16 2019][1689496.462838] R13: ffff91273b6849c0 R14: ffff9158ffab3ba0 R15: ffffffffc0ac8378 [Wed Feb 6 07:43:16 2019][1689496.470144] FS: 00007fe2d3435880(0000) GS:ffff9129fef00000(0000) knlGS:0000000000000000 [Wed Feb 6 07:43:16 2019][1689496.478403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:43:16 2019][1689496.484322] CR2: 00007fe2d0ba8050 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:43:16 2019][1689496.491629] Call Trace: [Wed Feb 6 07:43:16 2019][1689496.494260] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:43:16 2019][1689496.500699] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:43:16 2019][1689496.506414] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689496.513148] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689496.519886] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689496.526964] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689496.533870] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689496.540961] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689496.548825] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689496.555819] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689496.562295] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:43:16 2019][1689496.569862] [] kthread+0xd1/0xe0 [Wed Feb 6 07:43:16 2019][1689496.574910] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:16 2019][1689496.581178] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:43:16 2019][1689496.587790] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:16 2019][1689496.594055] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:43:20 2019][1689500.117026] Lustre: 14889:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (95:51s); client may timeout. req@ffff911ed8928900 x1624712533518272/t0(0) o103->2513721a-1a7a-beeb-ce96-babfef130551@10.8.18.32@o2ib6:162/0 lens 328/0 e 2 to 0 dl 1549467787 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Feb 6 07:43:20 2019][1689500.146183] Lustre: 14889:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 13 previous similar messages [Wed Feb 6 07:43:21 2019][1689501.138736] Lustre: MGS: Received new LWP connection from 10.0.10.106@o2ib7, removing former export from same NID [Wed Feb 6 07:43:21 2019][1689501.149173] Lustre: Skipped 3 previous similar messages [Wed Feb 6 07:43:26 2019][1689506.390391] Lustre: 6195:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff9129a5287850 x1623682073876000/t0(0) o400->fir-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 [Wed Feb 6 07:43:26 2019][1689506.412184] Lustre: 6195:0:(service.c:2011:ptlrpc_server_handle_req_in()) Skipped 25 previous similar messages [Wed Feb 6 07:43:30 2019][1689510.400048] LustreError: 9373:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.102.47@o2ib4: deadline 95:61s ago [Wed Feb 6 07:43:30 2019][1689510.400048] req@ffff913a8e0e8900 x1624673039126384/t0(0) o103->fd145a37-f0f0-19f7-9cb5-107d7b8bb223@10.9.102.47@o2ib4:162/0 lens 328/0 e 2 to 0 dl 1549467787 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Feb 6 07:43:30 2019][1689510.432326] LustreError: 9373:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 5 previous similar messages [Wed Feb 6 07:43:31 2019][1689510.919675] LNetError: 122079:0:(o2iblnd_cb.c:3281:kiblnd_cm_callback()) 10.0.10.202@o2ib7 DISCONNECTED [Wed Feb 6 07:43:31 2019][1689510.952846] LustreError: 6465:0:(osp_precreate.c:656:osp_precreate_send()) fir-OST0013-osc-MDT0002: can't precreate: rc = -11 [Wed Feb 6 07:43:31 2019][1689510.964349] LustreError: 6465:0:(osp_precreate.c:1312:osp_precreate_thread()) fir-OST0013-osc-MDT0002: cannot precreate objects: rc = -11 [Wed Feb 6 07:43:31 2019][1689511.640826] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 114 seconds [Wed Feb 6 07:43:31 2019][1689511.651173] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 63 previous similar messages [Wed Feb 6 07:43:31 2019][1689511.653102] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 23s! [ldlm_cn02_021:23326] [Wed Feb 6 07:43:31 2019][1689511.653141] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:43:32 2019][1689511.653159] CPU: 26 PID: 23326 Comm: ldlm_cn02_021 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:43:32 2019][1689511.653159] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:43:32 2019][1689511.653160] task: ffff91599ff9d140 ti: ffff9159aa650000 task.ti: ffff9159aa650000 [Wed Feb 6 07:43:32 2019][1689511.653168] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 07:43:32 2019][1689511.653169] RSP: 0018:ffff9159aa653c10 EFLAGS: 00000246 [Wed Feb 6 07:43:32 2019][1689511.653169] RAX: 0000000000000000 RBX: ffff9159aa653ba0 RCX: 0000000000d10000 [Wed Feb 6 07:43:32 2019][1689511.653170] RDX: ffff9139ff69b780 RSI: 0000000000490001 RDI: ffff911e813fa6dc [Wed Feb 6 07:43:32 2019][1689511.653171] RBP: ffff9159aa653c10 R08: ffff9149ff79b780 R09: 0000000000000000 [Wed Feb 6 07:43:32 2019][1689511.653171] R10: ffff911b3fc07600 R11: 00000000fffffff5 R12: ffff9159aa653bb0 [Wed Feb 6 07:43:32 2019][1689511.653172] R13: ffff911e813fa6dc R14: 0000000000690001 R15: ffff9139ff6db780 [Wed Feb 6 07:43:32 2019][1689511.653173] FS: 00007f35005b3880(0000) GS:ffff9149ff780000(0000) knlGS:0000000000000000 [Wed Feb 6 07:43:32 2019][1689511.653174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:43:32 2019][1689511.653174] CR2: 00007fe833b97ff0 CR3: 0000001035820000 CR4: 00000000003407e0 [Wed Feb 6 07:43:32 2019][1689511.653175] Call Trace: [Wed Feb 6 07:43:32 2019][1689511.653180] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 07:43:32 2019][1689511.653185] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 07:43:32 2019][1689511.653225] [] __ldlm_reprocess_all+0xfa/0x380 [ptlrpc] [Wed Feb 6 07:43:32 2019][1689511.653252] [] ldlm_reprocess_all+0x10/0x20 [ptlrpc] [Wed Feb 6 07:43:32 2019][1689511.653282] [] ldlm_request_cancel+0x3ed/0x740 [ptlrpc] [Wed Feb 6 07:43:32 2019][1689511.653312] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 07:43:32 2019][1689511.653341] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 07:43:32 2019][1689511.653375] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 07:43:32 2019][1689511.653406] [] ? ptlrpc_start_thread+0x27d/0xbf0 [ptlrpc] [Wed Feb 6 07:43:32 2019][1689511.653438] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 07:43:32 2019][1689511.653469] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 07:43:32 2019][1689511.653473] [] kthread+0xd1/0xe0 [Wed Feb 6 07:43:32 2019][1689511.653474] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:32 2019][1689511.653477] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:43:32 2019][1689511.653478] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:32 2019][1689511.653496] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 07:43:32 2019][1689511.998773] LustreError: 6471:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0011-osc-MDT0002: cannot cleanup orphans: rc = -11 [Wed Feb 6 07:43:32 2019][1689512.000869] LustreError: 5873:0:(events.c:450:server_bulk_callback()) event type 3, status -103, desc ffff91236d276a00 [Wed Feb 6 07:43:32 2019][1689512.000890] LustreError: 27688:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE req@ffff911e3e7be850 x1624732282737312/t0(0) o4->2d486816-0fee-3e02-cb25-89549e06c293@10.9.104.52@o2ib4:239/0 lens 488/448 e 4 to 0 dl 1549467864 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 07:43:32 2019][1689512.000918] Lustre: fir-MDT0002: Bulk IO write error with 2d486816-0fee-3e02-cb25-89549e06c293 (at 10.9.104.52@o2ib4), client will retry: rc = -110 [Wed Feb 6 07:43:32 2019][1689512.014441] LNetError: 5873:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Wed Feb 6 07:43:32 2019][1689512.060982] LNet: 5873:0:(o2iblnd_cb.c:1484:kiblnd_reconnect_peer()) Abort reconnection of 10.0.10.212@o2ib7: connected [Wed Feb 6 07:43:32 2019][1689512.449567] LustreError: 15989:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff913a97dc6f00 x1623682074145312/t0(0) o105->fir-MDT0002@10.8.8.33@o2ib6:15/16 lens 304/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 07:43:34 2019][1689514.078178] LNet: 5873:0:(o2iblnd_cb.c:1484:kiblnd_reconnect_peer()) Abort reconnection of 10.0.10.105@o2ib7: accepting [Wed Feb 6 07:43:34 2019][1689514.467132] LustreError: 15163:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff912bd0a16f00 x1623682074174512/t0(0) o104->fir-MDT0002@10.9.102.33@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 07:43:34 2019][1689514.488399] LustreError: 15163:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 35 previous similar messages [Wed Feb 6 07:43:38 2019][1689518.499265] LustreError: 15163:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9132b2e3da00 x1623682074177040/t0(0) o104->fir-MDT0002@10.9.102.33@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 07:43:38 2019][1689518.520528] LustreError: 15163:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 103 previous similar messages [Wed Feb 6 07:43:45 2019][1689525.369490] INFO: task kworker/33:1:321 blocked for more than 120 seconds. [Wed Feb 6 07:43:45 2019][1689525.376542] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Feb 6 07:43:45 2019][1689525.384544] kworker/33:1 D ffff912aa9ad5140 0 321 2 0x00000000 [Wed Feb 6 07:43:45 2019][1689525.391827] Workqueue: events key_garbage_collector [Wed Feb 6 07:43:45 2019][1689525.396906] Call Trace: [Wed Feb 6 07:43:45 2019][1689525.399542] [] ? ttwu_do_activate+0x6f/0x80 [Wed Feb 6 07:43:45 2019][1689525.405555] [] schedule+0x29/0x70 [Wed Feb 6 07:43:45 2019][1689525.410699] [] schedule_timeout+0x221/0x2d0 [Wed Feb 6 07:43:45 2019][1689525.416708] [] ? free_pcppages_bulk+0x17e/0x3a0 [Wed Feb 6 07:43:45 2019][1689525.423068] [] wait_for_completion+0xfd/0x140 [Wed Feb 6 07:43:45 2019][1689525.429247] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:43:45 2019][1689525.434996] [] ? __call_rcu+0x2c0/0x2c0 [Wed Feb 6 07:43:45 2019][1689525.440662] [] wait_rcu_gp+0x5e/0x80 [Wed Feb 6 07:43:45 2019][1689525.446068] [] ? ftrace_raw_output_rcu_utilization+0x70/0x70 [Wed Feb 6 07:43:45 2019][1689525.453549] [] synchronize_sched+0x3b/0x50 [Wed Feb 6 07:43:45 2019][1689525.459468] [] key_garbage_collector+0x1c5/0x390 [Wed Feb 6 07:43:45 2019][1689525.465907] [] process_one_work+0x17f/0x440 [Wed Feb 6 07:43:45 2019][1689525.471914] [] worker_thread+0x126/0x3c0 [Wed Feb 6 07:43:45 2019][1689525.477660] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 07:43:45 2019][1689525.484361] [] kthread+0xd1/0xe0 [Wed Feb 6 07:43:45 2019][1689525.489422] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:45 2019][1689525.495699] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:43:45 2019][1689525.502318] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:46 2019][1689526.532096] LustreError: 15184:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff913aca30e000 x1623682074182784/t0(0) o104->fir-MDT0002@10.9.103.17@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 07:43:46 2019][1689526.553363] LustreError: 15184:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 206 previous similar messages [Wed Feb 6 07:43:50 2019][1689529.798777] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 07:43:50 2019][1689529.815200] LustreError: Skipped 3 previous similar messages [Wed Feb 6 07:43:53 2019][1689533.147551] Lustre: MGS: Received new LWP connection from 10.8.26.3@o2ib6, removing former export from same NID [Wed Feb 6 07:43:53 2019][1689533.157809] Lustre: Skipped 406 previous similar messages [Wed Feb 6 07:43:55 2019][1689535.663710] NMI watchdog: BUG: soft lockup - CPU#27 stuck for 23s! [ldlm_bl_03:7089] [Wed Feb 6 07:43:55 2019][1689535.671627] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:43:56 2019][1689535.778394] CPU: 27 PID: 7089 Comm: ldlm_bl_03 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:43:56 2019][1689535.790813] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:43:56 2019][1689535.798553] task: ffff9149f3e35140 ti: ffff9159a92c4000 task.ti: ffff9159a92c4000 [Wed Feb 6 07:43:56 2019][1689535.806204] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 07:43:56 2019][1689535.816700] RSP: 0018:ffff9159a92c7bb0 EFLAGS: 00000282 [Wed Feb 6 07:43:56 2019][1689535.822188] RAX: 0000000000000101 RBX: ffff911fbf44bf60 RCX: ffff911fbf44bf60 [Wed Feb 6 07:43:56 2019][1689535.829493] RDX: ffff9159a92c7ca8 RSI: ffff913ad0374380 RDI: ffff9131c475bf00 [Wed Feb 6 07:43:56 2019][1689535.836798] RBP: ffff9159a92c7c08 R08: ffff9159a92c7ca8 R09: 00000000c0001753 [Wed Feb 6 07:43:56 2019][1689535.844106] R10: 0000000000000053 R11: ffff913ad0374380 R12: ffff9159a92c7ca8 [Wed Feb 6 07:43:56 2019][1689535.851412] R13: 00000000c0001753 R14: 0000000000000053 R15: ffff913ad0374380 [Wed Feb 6 07:43:56 2019][1689535.858719] FS: 00007fccefbfb840(0000) GS:ffff915a3f580000(0000) knlGS:0000000000000000 [Wed Feb 6 07:43:56 2019][1689535.866977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:43:56 2019][1689535.872898] CR2: 000055fdd41372a8 CR3: 000000102dee6000 CR4: 00000000003407e0 [Wed Feb 6 07:43:56 2019][1689535.880202] Call Trace: [Wed Feb 6 07:43:56 2019][1689535.882862] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 07:43:56 2019][1689535.890545] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 07:43:56 2019][1689535.898480] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 07:43:56 2019][1689535.905640] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 07:43:56 2019][1689535.912800] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 07:43:56 2019][1689535.921259] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 07:43:56 2019][1689535.929462] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 07:43:56 2019][1689535.936514] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:43:56 2019][1689535.942286] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 07:43:56 2019][1689535.949850] [] kthread+0xd1/0xe0 [Wed Feb 6 07:43:56 2019][1689535.954903] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:56 2019][1689535.961171] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:43:56 2019][1689535.967784] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:43:56 2019][1689535.974047] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 07:43:59 2019][1689539.558805] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [ldlm_bl_25:15988] [Wed Feb 6 07:43:59 2019][1689539.566808] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:43:59 2019][1689539.673577] CPU: 15 PID: 15988 Comm: ldlm_bl_25 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:43:59 2019][1689539.686083] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:43:59 2019][1689539.693823] task: ffff9141bfb80000 ti: ffff91461b220000 task.ti: ffff91461b220000 [Wed Feb 6 07:43:59 2019][1689539.701476] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 07:43:59 2019][1689539.711972] RSP: 0018:ffff91461b223bb0 EFLAGS: 00000282 [Wed Feb 6 07:44:00 2019][1689539.717456] RAX: 0000000000000101 RBX: ffff913962646a20 RCX: ffff913962646a20 [Wed Feb 6 07:44:00 2019][1689539.724762] RDX: ffff91461b223ca8 RSI: ffff912e93780240 RDI: ffff913ca8dcb3c0 [Wed Feb 6 07:44:00 2019][1689539.732069] RBP: ffff91461b223c08 R08: ffff91461b223ca8 R09: 00000000c000171d [Wed Feb 6 07:44:00 2019][1689539.739375] R10: 000000000000001d R11: ffff912e93780240 R12: ffff91461b223ca8 [Wed Feb 6 07:44:00 2019][1689539.746681] R13: 00000000c000171d R14: 000000000000001d R15: ffff912e93780240 [Wed Feb 6 07:44:00 2019][1689539.753989] FS: 00007f64178d5700(0000) GS:ffff915a3f4c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:44:00 2019][1689539.762247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:44:00 2019][1689539.768168] CR2: 00007fdc13e932e0 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:44:00 2019][1689539.775474] Call Trace: [Wed Feb 6 07:44:00 2019][1689539.778133] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 07:44:00 2019][1689539.785819] [] ? ldlm_process_inodebits_lock+0xd/0x3d0 [ptlrpc] [Wed Feb 6 07:44:00 2019][1689539.793590] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 07:44:00 2019][1689539.801525] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 07:44:00 2019][1689539.808686] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 07:44:00 2019][1689539.815843] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 07:44:00 2019][1689539.824301] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 07:44:00 2019][1689539.832504] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 07:44:00 2019][1689539.839551] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:44:00 2019][1689539.845322] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 07:44:00 2019][1689539.852888] [] kthread+0xd1/0xe0 [Wed Feb 6 07:44:00 2019][1689539.857939] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:44:00 2019][1689539.864205] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:44:00 2019][1689539.870819] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:44:00 2019][1689539.877084] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 07:44:02 2019][1689542.568828] LustreError: 15184:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff913a975d2100 x1623682074198288/t0(0) o104->fir-MDT0002@10.9.103.17@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 07:44:02 2019][1689542.590093] LustreError: 15184:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 416 previous similar messages [Wed Feb 6 07:44:27 2019][1689567.559506] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [ldlm_bl_25:15988] [Wed Feb 6 07:44:27 2019][1689567.567504] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:44:27 2019][1689567.674291] CPU: 15 PID: 15988 Comm: ldlm_bl_25 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:44:27 2019][1689567.686794] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:44:27 2019][1689567.694533] task: ffff9141bfb80000 ti: ffff91461b220000 task.ti: ffff91461b220000 [Wed Feb 6 07:44:27 2019][1689567.702187] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 07:44:27 2019][1689567.712683] RSP: 0018:ffff91461b223bb0 EFLAGS: 00000282 [Wed Feb 6 07:44:28 2019][1689567.718168] RAX: 0000000000000101 RBX: ffff913962646a20 RCX: ffff913962646a20 [Wed Feb 6 07:44:28 2019][1689567.725474] RDX: ffff91461b223ca8 RSI: ffff912e93780240 RDI: ffff91493733dc40 [Wed Feb 6 07:44:28 2019][1689567.732780] RBP: ffff91461b223c08 R08: ffff91461b223ca8 R09: 00000000c000171d [Wed Feb 6 07:44:28 2019][1689567.740089] R10: 000000000000001d R11: ffff912e93780240 R12: ffff91461b223ca8 [Wed Feb 6 07:44:28 2019][1689567.747394] R13: 00000000c000171d R14: 000000000000001d R15: ffff912e93780240 [Wed Feb 6 07:44:28 2019][1689567.754700] FS: 00007f64178d5700(0000) GS:ffff915a3f4c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:44:28 2019][1689567.762960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:44:28 2019][1689567.768878] CR2: 00007fdc13e932e0 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:44:28 2019][1689567.776184] Call Trace: [Wed Feb 6 07:44:28 2019][1689567.778848] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 07:44:28 2019][1689567.786531] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 07:44:28 2019][1689567.794473] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 07:44:28 2019][1689567.801632] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 07:44:28 2019][1689567.808791] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 07:44:28 2019][1689567.817248] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 07:44:28 2019][1689567.825451] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 07:44:28 2019][1689567.832498] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:44:28 2019][1689567.838276] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 07:44:28 2019][1689567.845842] [] kthread+0xd1/0xe0 [Wed Feb 6 07:44:28 2019][1689567.850896] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:44:28 2019][1689567.857162] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:44:28 2019][1689567.863773] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:44:28 2019][1689567.870038] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 07:44:30 2019][1689570.347710] Lustre: MGS: haven't heard from client 7690a4eb-01d5-36eb-a4b1-affbc86df541 (at 10.9.103.25@o2ib4) in 186 seconds. I think it's dead, and I am evicting it. exp ffff9135fcb34800, cur 1549467908 expire 1549467758 last 1549467722 [Wed Feb 6 07:44:30 2019][1689570.368978] Lustre: Skipped 2 previous similar messages [Wed Feb 6 07:44:33 2019][1689573.322650] INFO: rcu_sched self-detected stall on CPU[1689573.324651] INFO: rcu_sched detected stalls on CPUs/tasks: { 15} (detected by 19, t=60002 jiffies, g=134910432, c=134910431, q=611400) [Wed Feb 6 07:44:33 2019][1689573.324666] Task dump for CPU 15: [Wed Feb 6 07:44:33 2019][1689573.324667] ldlm_bl_25 R running task 0 15988 2 0x00000088 [Wed Feb 6 07:44:33 2019][1689573.324670] Call Trace: [Wed Feb 6 07:44:33 2019][1689573.324716] [] ? ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 07:44:33 2019][1689573.324747] [] ? ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 07:44:33 2019][1689573.324751] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:44:33 2019][1689573.324784] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 07:44:33 2019][1689573.324787] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:44:33 2019][1689573.324789] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:44:33 2019][1689573.324792] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:44:33 2019][1689573.324794] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:44:33 2019] [Wed Feb 6 07:44:33 2019][1689573.406955] { 15} (t=60086 jiffies g=134910432 c=134910431 q=611640) [Wed Feb 6 07:44:33 2019][1689573.413940] Task dump for CPU 15: [Wed Feb 6 07:44:33 2019][1689573.417431] ldlm_bl_25 R running task 0 15988 2 0x00000088 [Wed Feb 6 07:44:33 2019][1689573.424704] Call Trace: [Wed Feb 6 07:44:33 2019][1689573.427331] [] sched_show_task+0xa8/0x110 [Wed Feb 6 07:44:33 2019][1689573.433797] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 07:44:33 2019][1689573.439371] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 07:44:33 2019][1689573.445461] [] rcu_check_callbacks+0x442/0x730 [Wed Feb 6 07:44:33 2019][1689573.451731] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 07:44:33 2019][1689573.457994] [] update_process_times+0x46/0x80 [Wed Feb 6 07:44:33 2019][1689573.464174] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 07:44:33 2019][1689573.470094] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 07:44:33 2019][1689573.475928] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 07:44:33 2019][1689573.482191] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 07:44:33 2019][1689573.488198] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 07:44:33 2019][1689573.494900] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 07:44:33 2019][1689573.501425] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 07:44:33 2019][1689573.507775] [] ? ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 07:44:33 2019][1689573.516391] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 07:44:33 2019][1689573.524071] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 07:44:33 2019][1689573.532007] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 07:44:33 2019][1689573.539169] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 07:44:33 2019][1689573.546325] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 07:44:33 2019][1689573.554786] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 07:44:33 2019][1689573.562985] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 07:44:33 2019][1689573.570031] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:44:33 2019][1689573.575804] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 07:44:33 2019][1689573.583370] [] kthread+0xd1/0xe0 [Wed Feb 6 07:44:33 2019][1689573.588422] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:44:33 2019][1689573.594688] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:44:33 2019][1689573.601301] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:44:34 2019][1689574.588351] LustreError: 15163:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff91476964aa00 x1623682074279776/t0(0) o104->fir-MDT0002@10.9.102.33@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 07:44:34 2019][1689574.609618] LustreError: 15163:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 874 previous similar messages [Wed Feb 6 07:44:53 2019][1689593.032158] LNet: Service thread pid 15184 was inactive for 362.56s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 07:44:53 2019][1689593.049264] Pid: 15184, comm: mdt02_032 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 07:44:53 2019][1689593.059190] Call Trace: [Wed Feb 6 07:44:53 2019][1689593.061821] [] 0xffffffffffffffff [Wed Feb 6 07:44:53 2019][1689593.067019] LustreError: dumping log to /tmp/lustre-log.1549467931.15184 [Wed Feb 6 07:44:59 2019][1689599.560310] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [ldlm_bl_25:15988] [Wed Feb 6 07:44:59 2019][1689599.568312] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:44:59 2019][1689599.675108] CPU: 15 PID: 15988 Comm: ldlm_bl_25 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:44:59 2019][1689599.687610] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:44:59 2019][1689599.695351] task: ffff9141bfb80000 ti: ffff91461b220000 task.ti: ffff91461b220000 [Wed Feb 6 07:44:59 2019][1689599.703003] RIP: 0010:[] [] ldlm_add_ast_work_item+0x33/0x3d0 [ptlrpc] [Wed Feb 6 07:44:59 2019][1689599.712970] RSP: 0018:ffff91461b223b80 EFLAGS: 00000246 [Wed Feb 6 07:45:00 2019][1689599.718457] RAX: ffff911f85a51a40 RBX: 000000000001ab80 RCX: ffff913962646a20 [Wed Feb 6 07:45:00 2019][1689599.725761] RDX: ffff91461b223ca8 RSI: ffff912e93780240 RDI: ffff9137b5941b00 [Wed Feb 6 07:45:00 2019][1689599.733070] RBP: ffff91461b223ba0 R08: ffff91461b223ca8 R09: 00000000c000171d [Wed Feb 6 07:45:00 2019][1689599.740374] R10: 000000000000001d R11: ffff912e93780240 R12: 0000000000000027 [Wed Feb 6 07:45:00 2019][1689599.747680] R13: ffff914aa95d8400 R14: 0000000000000000 R15: ffffffffadce3ce4 [Wed Feb 6 07:45:00 2019][1689599.754988] FS: 00007f64178d5700(0000) GS:ffff915a3f4c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:45:00 2019][1689599.763246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:45:00 2019][1689599.769165] CR2: 00007fdc13e932e0 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:45:00 2019][1689599.776472] Call Trace: [Wed Feb 6 07:45:00 2019][1689599.779138] [] ldlm_inodebits_compat_queue+0x198/0x440 [ptlrpc] [Wed Feb 6 07:45:00 2019][1689599.786912] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 07:45:00 2019][1689599.794590] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 07:45:00 2019][1689599.802526] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 07:45:00 2019][1689599.809685] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 07:45:00 2019][1689599.816843] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 07:45:00 2019][1689599.825302] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 07:45:00 2019][1689599.833503] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 07:45:00 2019][1689599.840549] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:45:00 2019][1689599.846324] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 07:45:00 2019][1689599.853894] [] kthread+0xd1/0xe0 [Wed Feb 6 07:45:00 2019][1689599.858949] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:45:00 2019][1689599.865214] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:45:00 2019][1689599.871826] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:45:00 2019][1689599.878092] Code: 5b ad cc ff 01 48 89 e5 41 56 49 89 d6 41 55 41 54 49 89 f4 53 48 89 fb 74 0d f6 05 45 ad cc ff 01 0f 85 c1 00 00 00 48 8b 43 48 <8b> 40 1c 85 c0 0f 84 47 02 00 00 4d 85 e4 0f 84 91 01 00 00 4c [Wed Feb 6 07:45:27 2019][1689627.561016] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [ldlm_bl_25:15988] [Wed Feb 6 07:45:27 2019][1689627.569016] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:45:27 2019][1689627.675785] CPU: 15 PID: 15988 Comm: ldlm_bl_25 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:45:27 2019][1689627.688289] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:45:27 2019][1689627.696028] task: ffff9141bfb80000 ti: ffff91461b220000 task.ti: ffff91461b220000 [Wed Feb 6 07:45:27 2019][1689627.703682] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 07:45:27 2019][1689627.714178] RSP: 0018:ffff91461b223bb0 EFLAGS: 00000282 [Wed Feb 6 07:45:28 2019][1689627.719664] RAX: 0000000000000101 RBX: ffff913962646a20 RCX: ffff913962646a20 [Wed Feb 6 07:45:28 2019][1689627.726970] RDX: ffff91461b223ca8 RSI: ffff912e93780240 RDI: ffff9122cd8745c0 [Wed Feb 6 07:45:28 2019][1689627.734275] RBP: ffff91461b223c08 R08: ffff91461b223ca8 R09: 00000000c000171d [Wed Feb 6 07:45:28 2019][1689627.741582] R10: 000000000000001d R11: ffff912e93780240 R12: ffff91461b223ca8 [Wed Feb 6 07:45:28 2019][1689627.748889] R13: 00000000c000171d R14: 000000000000001d R15: ffff912e93780240 [Wed Feb 6 07:45:28 2019][1689627.756194] FS: 00007f64178d5700(0000) GS:ffff915a3f4c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:45:28 2019][1689627.764454] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:45:28 2019][1689627.770374] CR2: 00007fdc13e932e0 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:45:28 2019][1689627.777679] Call Trace: [Wed Feb 6 07:45:28 2019][1689627.780341] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 07:45:28 2019][1689627.788026] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 07:45:28 2019][1689627.795970] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 07:45:28 2019][1689627.803126] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 07:45:28 2019][1689627.810284] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 07:45:28 2019][1689627.818744] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 07:45:28 2019][1689627.826944] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 07:45:28 2019][1689627.833990] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:45:28 2019][1689627.839763] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 07:45:28 2019][1689627.847330] [] kthread+0xd1/0xe0 [Wed Feb 6 07:45:28 2019][1689627.852382] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:45:28 2019][1689627.858649] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:45:28 2019][1689627.865259] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:45:28 2019][1689627.871523] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 07:45:38 2019][1689638.591728] LustreError: 15163:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff913298823600 x1623682074448592/t0(0) o104->fir-MDT0002@10.9.102.33@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 07:45:38 2019][1689638.612995] LustreError: 15163:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1757 previous similar messages [Wed Feb 6 07:45:45 2019][1689645.511506] INFO: task kworker/33:1:321 blocked for more than 120 seconds. [Wed Feb 6 07:45:45 2019][1689645.518559] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Feb 6 07:45:45 2019][1689645.526571] kworker/33:1 D ffff912aa9ad5140 0 321 2 0x00000000 [Wed Feb 6 07:45:45 2019][1689645.533853] Workqueue: events key_garbage_collector [Wed Feb 6 07:45:45 2019][1689645.538957] Call Trace: [Wed Feb 6 07:45:45 2019][1689645.541594] [] ? ttwu_do_activate+0x6f/0x80 [Wed Feb 6 07:45:45 2019][1689645.547607] [] schedule+0x29/0x70 [Wed Feb 6 07:45:45 2019][1689645.552771] [] schedule_timeout+0x221/0x2d0 [Wed Feb 6 07:45:45 2019][1689645.558788] [] ? free_pcppages_bulk+0x17e/0x3a0 [Wed Feb 6 07:45:45 2019][1689645.565153] [] wait_for_completion+0xfd/0x140 [Wed Feb 6 07:45:45 2019][1689645.571449] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:45:45 2019][1689645.577317] [] ? __call_rcu+0x2c0/0x2c0 [Wed Feb 6 07:45:45 2019][1689645.583064] [] wait_rcu_gp+0x5e/0x80 [Wed Feb 6 07:45:45 2019][1689645.588488] [] ? ftrace_raw_output_rcu_utilization+0x70/0x70 [Wed Feb 6 07:45:45 2019][1689645.595976] [] synchronize_sched+0x3b/0x50 [Wed Feb 6 07:45:45 2019][1689645.601902] [] key_garbage_collector+0x1c5/0x390 [Wed Feb 6 07:45:45 2019][1689645.608367] [] process_one_work+0x17f/0x440 [Wed Feb 6 07:45:45 2019][1689645.614383] [] worker_thread+0x126/0x3c0 [Wed Feb 6 07:45:45 2019][1689645.620156] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 07:45:45 2019][1689645.626896] [] kthread+0xd1/0xe0 [Wed Feb 6 07:45:45 2019][1689645.631961] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:45:45 2019][1689645.638245] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:45:45 2019][1689645.644888] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:45:45 2019][1689645.651200] INFO: task systemd-logind:17603 blocked for more than 120 seconds. [Wed Feb 6 07:45:45 2019][1689645.658600] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Feb 6 07:45:45 2019][1689645.666633] systemd-logind D ffff9149f3e32080 0 17603 1 0x00000080 [Wed Feb 6 07:45:45 2019][1689645.673911] Call Trace: [Wed Feb 6 07:45:45 2019][1689645.676544] [] ? ttwu_do_activate+0x6f/0x80 [Wed Feb 6 07:45:45 2019][1689645.682567] [] schedule+0x29/0x70 [Wed Feb 6 07:45:45 2019][1689645.687712] [] schedule_timeout+0x221/0x2d0 [Wed Feb 6 07:45:46 2019][1689645.693732] [] ? __slab_free+0x81/0x2f0 [Wed Feb 6 07:45:46 2019][1689645.699423] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:45:46 2019][1689645.705446] [] wait_for_completion+0xfd/0x140 [Wed Feb 6 07:45:46 2019][1689645.711648] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:45:46 2019][1689645.717412] [] ? __call_rcu+0x2c0/0x2c0 [Wed Feb 6 07:45:46 2019][1689645.723076] [] wait_rcu_gp+0x5e/0x80 [Wed Feb 6 07:45:46 2019][1689645.728481] [] ? ftrace_raw_output_rcu_utilization+0x70/0x70 [Wed Feb 6 07:45:46 2019][1689645.735987] [] ? umount_tree+0x184/0x2a0 [Wed Feb 6 07:45:46 2019][1689645.741741] [] synchronize_sched+0x3b/0x50 [Wed Feb 6 07:45:46 2019][1689645.747670] [] namespace_unlock+0x68/0x80 [Wed Feb 6 07:45:46 2019][1689645.753528] [] SyS_umount+0x239/0x410 [Wed Feb 6 07:45:46 2019][1689645.759023] [] ? ____fput+0xe/0x10 [Wed Feb 6 07:45:46 2019][1689645.764257] [] ? task_work_run+0xc0/0xe0 [Wed Feb 6 07:45:46 2019][1689645.770029] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 07:45:55 2019][1689655.561718] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [ldlm_bl_25:15988] [Wed Feb 6 07:45:55 2019][1689655.569722] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:45:55 2019][1689655.676507] CPU: 15 PID: 15988 Comm: ldlm_bl_25 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:45:55 2019][1689655.689013] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:45:55 2019][1689655.696751] task: ffff9141bfb80000 ti: ffff91461b220000 task.ti: ffff91461b220000 [Wed Feb 6 07:45:55 2019][1689655.704404] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 07:45:55 2019][1689655.714902] RSP: 0018:ffff91461b223bb0 EFLAGS: 00000282 [Wed Feb 6 07:45:56 2019][1689655.720386] RAX: 0000000000000101 RBX: ffff913962646a20 RCX: ffff913962646a20 [Wed Feb 6 07:45:56 2019][1689655.727694] RDX: ffff91461b223ca8 RSI: ffff912e93780240 RDI: ffff9159e5bdb180 [Wed Feb 6 07:45:56 2019][1689655.735000] RBP: ffff91461b223c08 R08: ffff91461b223ca8 R09: 00000000c000171d [Wed Feb 6 07:45:56 2019][1689655.742305] R10: 000000000000001d R11: ffff912e93780240 R12: ffff91461b223ca8 [Wed Feb 6 07:45:56 2019][1689655.749621] R13: 00000000c000171d R14: 000000000000001d R15: ffff912e93780240 [Wed Feb 6 07:45:56 2019][1689655.756928] FS: 00007f64178d5700(0000) GS:ffff915a3f4c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:45:56 2019][1689655.765186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:45:56 2019][1689655.771106] CR2: 00007fdc13e932e0 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:45:56 2019][1689655.778412] Call Trace: [Wed Feb 6 07:45:56 2019][1689655.781083] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 07:45:56 2019][1689655.788746] [] ? _raw_write_lock+0x10/0x20 [Wed Feb 6 07:45:56 2019][1689655.794694] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 07:45:56 2019][1689655.802645] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 07:45:56 2019][1689655.809804] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 07:45:56 2019][1689655.816965] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 07:45:56 2019][1689655.825422] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 07:45:56 2019][1689655.833622] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 07:45:56 2019][1689655.840668] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:45:56 2019][1689655.846440] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 07:45:56 2019][1689655.854004] [] kthread+0xd1/0xe0 [Wed Feb 6 07:45:56 2019][1689655.859058] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:45:56 2019][1689655.865325] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:45:56 2019][1689655.871938] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:45:56 2019][1689655.878202] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 07:46:04 2019][1689664.242952] LustreError: 17203:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.112.12@tcp) failed to reply to blocking AST (req@ffff915893f36000 x1623682074174176 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff9121aa835340/0x21ed9351afd7118b lrc: 4/0,0 mode: PR/PR res: [0x2c0001746:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.10.112.12@tcp remote: 0xefba423e132a5554 expref: 20 pid: 15228 timeout: 1689770 lvb_type: 0 [Wed Feb 6 07:46:04 2019][1689664.285630] LustreError: 17203:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 8 previous similar messages [Wed Feb 6 07:46:04 2019][1689664.296272] LustreError: 17203:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff913563990800 ns: mdt-fir-MDT0002_UUID lock: ffff911d037445c0/0x21ed9351afd711a0 lrc: 1/0,0 mode: EX/EX res: [0x2c0001746:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.112.12@tcp remote: 0xefba423e132a5562 expref: 4 pid: 17203 timeout: 0 lvb_type: 3 [Wed Feb 6 07:46:04 2019][1689664.330630] LustreError: 17203:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 23 previous similar messages [Wed Feb 6 07:46:23 2019][1689683.562421] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [ldlm_bl_25:15988] [Wed Feb 6 07:46:23 2019][1689683.570420] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:46:23 2019][1689683.677189] CPU: 15 PID: 15988 Comm: ldlm_bl_25 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:46:23 2019][1689683.689693] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:46:23 2019][1689683.697434] task: ffff9141bfb80000 ti: ffff91461b220000 task.ti: ffff91461b220000 [Wed Feb 6 07:46:23 2019][1689683.705086] RIP: 0010:[] [] ldlm_add_ast_work_item+0x33/0x3d0 [ptlrpc] [Wed Feb 6 07:46:23 2019][1689683.715054] RSP: 0018:ffff91461b223b80 EFLAGS: 00000246 [Wed Feb 6 07:46:24 2019][1689683.720538] RAX: ffff911f85a51a40 RBX: 000000000001ab80 RCX: ffff913962646a20 [Wed Feb 6 07:46:24 2019][1689683.727846] RDX: ffff91461b223ca8 RSI: ffff912e93780240 RDI: ffff9123ea4a5c40 [Wed Feb 6 07:46:24 2019][1689683.735152] RBP: ffff91461b223ba0 R08: ffff91461b223ca8 R09: 00000000c000171d [Wed Feb 6 07:46:24 2019][1689683.742458] R10: 000000000000001d R11: ffff912e93780240 R12: 0000000000000027 [Wed Feb 6 07:46:24 2019][1689683.749764] R13: ffff914aa95d8400 R14: 0000000000000000 R15: ffffffffadce3ce4 [Wed Feb 6 07:46:24 2019][1689683.757070] FS: 00007f64178d5700(0000) GS:ffff915a3f4c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:46:24 2019][1689683.765329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:46:24 2019][1689683.771250] CR2: 00007fdc13e932e0 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:46:24 2019][1689683.778555] Call Trace: [Wed Feb 6 07:46:24 2019][1689683.781218] [] ldlm_inodebits_compat_queue+0x198/0x440 [ptlrpc] [Wed Feb 6 07:46:24 2019][1689683.788987] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 07:46:24 2019][1689683.796671] [] ? ldlm_export_lock_object+0x10/0x10 [ptlrpc] [Wed Feb 6 07:46:24 2019][1689683.804089] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 07:46:24 2019][1689683.812025] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 07:46:24 2019][1689683.819184] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 07:46:24 2019][1689683.826345] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 07:46:24 2019][1689683.834804] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 07:46:24 2019][1689683.843004] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 07:46:24 2019][1689683.850052] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:46:24 2019][1689683.855820] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 07:46:24 2019][1689683.863387] [] kthread+0xd1/0xe0 [Wed Feb 6 07:46:24 2019][1689683.868440] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:46:24 2019][1689683.874707] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:46:24 2019][1689683.881319] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:46:24 2019][1689683.887583] Code: 5b ad cc ff 01 48 89 e5 41 56 49 89 d6 41 55 41 54 49 89 f4 53 48 89 fb 74 0d f6 05 45 ad cc ff 01 0f 85 c1 00 00 00 48 8b 43 48 <8b> 40 1c 85 c0 0f 84 47 02 00 00 4d 85 e4 0f 84 91 01 00 00 4c [Wed Feb 6 07:46:51 2019][1689711.563122] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [ldlm_bl_25:15988] [Wed Feb 6 07:46:51 2019][1689711.571120] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:46:51 2019][1689711.677889] CPU: 15 PID: 15988 Comm: ldlm_bl_25 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:46:51 2019][1689711.690394] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:46:51 2019][1689711.698134] task: ffff9141bfb80000 ti: ffff91461b220000 task.ti: ffff91461b220000 [Wed Feb 6 07:46:51 2019][1689711.705787] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 07:46:51 2019][1689711.716281] RSP: 0018:ffff91461b223bb0 EFLAGS: 00000282 [Wed Feb 6 07:46:52 2019][1689711.721767] RAX: 0000000000000101 RBX: ffff913962646a20 RCX: ffff913962646a20 [Wed Feb 6 07:46:52 2019][1689711.729074] RDX: ffff91461b223ca8 RSI: ffff912e93780240 RDI: ffff9150a13857c0 [Wed Feb 6 07:46:52 2019][1689711.736381] RBP: ffff91461b223c08 R08: ffff91461b223ca8 R09: 00000000c000171d [Wed Feb 6 07:46:52 2019][1689711.743686] R10: 000000000000001d R11: ffff912e93780240 R12: ffff91461b223ca8 [Wed Feb 6 07:46:52 2019][1689711.750993] R13: 00000000c000171d R14: 000000000000001d R15: ffff912e93780240 [Wed Feb 6 07:46:52 2019][1689711.758299] FS: 00007f64178d5700(0000) GS:ffff915a3f4c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:46:52 2019][1689711.766558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:46:52 2019][1689711.772477] CR2: 00007fdc13e932e0 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:46:52 2019][1689711.779784] Call Trace: [Wed Feb 6 07:46:52 2019][1689711.782445] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 07:46:52 2019][1689711.790127] [] ? ldlm_export_lock_object+0x10/0x10 [ptlrpc] [Wed Feb 6 07:46:52 2019][1689711.797546] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 07:46:52 2019][1689711.805481] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 07:46:52 2019][1689711.812641] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 07:46:52 2019][1689711.819798] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 07:46:52 2019][1689711.828260] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 07:46:52 2019][1689711.836460] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 07:46:52 2019][1689711.843505] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:46:52 2019][1689711.849276] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 07:46:52 2019][1689711.856843] [] kthread+0xd1/0xe0 [Wed Feb 6 07:46:52 2019][1689711.861896] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:46:52 2019][1689711.868161] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:46:52 2019][1689711.874775] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:46:52 2019][1689711.881040] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 07:46:59 2019][1689719.623323] NMI watchdog: BUG: soft lockup - CPU#22 stuck for 23s! [migration/22:122] [Wed Feb 6 07:46:59 2019][1689719.631322] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:47:00 2019][1689719.738092] CPU: 22 PID: 122 Comm: migration/22 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:47:00 2019][1689719.750597] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:47:00 2019][1689719.758336] task: ffff913aa98cd140 ti: ffff913aa990c000 task.ti: ffff913aa990c000 [Wed Feb 6 07:47:00 2019][1689719.765988] RIP: 0010:[] [] multi_cpu_stop+0x4f/0x110 [Wed Feb 6 07:47:00 2019][1689719.774448] RSP: 0000:ffff913aa990fd98 EFLAGS: 00000246 [Wed Feb 6 07:47:00 2019][1689719.779934] RAX: 0000000000000001 RBX: ffff9139eef23af8 RCX: dead000000000200 [Wed Feb 6 07:47:00 2019][1689719.787241] RDX: ffff9149ff755ff0 RSI: 0000000000000282 RDI: ffff9139eef23ab0 [Wed Feb 6 07:47:00 2019][1689719.794546] RBP: ffff913aa990fdc0 R08: ffff9139eef23a80 R09: 0000000000000001 [Wed Feb 6 07:47:00 2019][1689719.801850] R10: 000000000000be9c R11: 0000000000000001 R12: 0000000000000086 [Wed Feb 6 07:47:00 2019][1689719.809158] R13: 000000000001ab80 R14: ffff913a00000006 R15: ffffffffadcd64f0 [Wed Feb 6 07:47:00 2019][1689719.816465] FS: 00007f78f341b880(0000) GS:ffff9149ff740000(0000) knlGS:0000000000000000 [Wed Feb 6 07:47:00 2019][1689719.824725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:47:00 2019][1689719.830642] CR2: 00007f78e39cff50 CR3: 0000004033a9a000 CR4: 00000000003407e0 [Wed Feb 6 07:47:00 2019][1689719.837951] Call Trace: [Wed Feb 6 07:47:00 2019][1689719.840578] [] ? cpu_stop_should_run+0x50/0x50 [Wed Feb 6 07:47:00 2019][1689719.846843] [] cpu_stopper_thread+0x99/0x150 [Wed Feb 6 07:47:00 2019][1689719.852940] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 07:47:00 2019][1689719.858605] [] ? cpu_stop_queue_work+0x70/0x70 [Wed Feb 6 07:47:00 2019][1689719.864872] [] smpboot_thread_fn+0x144/0x1a0 [Wed Feb 6 07:47:00 2019][1689719.870962] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 07:47:00 2019][1689719.876971] [] kthread+0xd1/0xe0 [Wed Feb 6 07:47:00 2019][1689719.882020] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:47:00 2019][1689719.888289] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:47:00 2019][1689719.894899] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:47:00 2019][1689719.901167] Code: 89 c5 48 8b 47 18 48 85 c0 0f 84 b3 00 00 00 0f a3 18 19 db 85 db 41 0f 95 c6 45 31 ff 31 c0 0f 1f 44 00 00 f3 90 41 8b 5c 24 20 <39> c3 74 5d 83 fb 02 74 68 83 fb 03 75 05 45 84 f6 75 6e f0 41 [Wed Feb 6 07:47:03 2019][1689723.548438] LustreError: 17053:0:(upcall_cache.c:233:upcall_cache_get_entry()) acquire for key 297421: error -110 [Wed Feb 6 07:47:12 2019][1689732.485654] LustreError: 15183:0:(upcall_cache.c:233:upcall_cache_get_entry()) acquire for key 298495: error -110 [Wed Feb 6 07:47:19 2019][1689738.827815] LustreError: 7081:0:(upcall_cache.c:233:upcall_cache_get_entry()) acquire for key 318438: error -110 [Wed Feb 6 07:47:19 2019][1689739.563824] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [ldlm_bl_25:15988] [Wed Feb 6 07:47:19 2019][1689739.571821] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:47:19 2019][1689739.678591] CPU: 15 PID: 15988 Comm: ldlm_bl_25 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:47:19 2019][1689739.691096] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:47:19 2019][1689739.698835] task: ffff9141bfb80000 ti: ffff91461b220000 task.ti: ffff91461b220000 [Wed Feb 6 07:47:19 2019][1689739.706489] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 07:47:19 2019][1689739.716985] RSP: 0018:ffff91461b223bb0 EFLAGS: 00000282 [Wed Feb 6 07:47:19 2019][1689739.722469] RAX: 0000000000000101 RBX: ffff913962646a20 RCX: ffff913962646a20 [Wed Feb 6 07:47:20 2019][1689739.729775] RDX: ffff91461b223ca8 RSI: ffff912e93780240 RDI: ffff913f53e25a00 [Wed Feb 6 07:47:20 2019][1689739.737083] RBP: ffff91461b223c08 R08: ffff91461b223ca8 R09: 00000000c000171d [Wed Feb 6 07:47:20 2019][1689739.744387] R10: 000000000000001d R11: ffff912e93780240 R12: ffff91461b223ca8 [Wed Feb 6 07:47:20 2019][1689739.751695] R13: 00000000c000171d R14: 000000000000001d R15: ffff912e93780240 [Wed Feb 6 07:47:20 2019][1689739.759002] FS: 00007f64178d5700(0000) GS:ffff915a3f4c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:47:20 2019][1689739.767261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:47:20 2019][1689739.773180] CR2: 00007fdc13e932e0 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 07:47:20 2019][1689739.780485] Call Trace: [Wed Feb 6 07:47:20 2019][1689739.783145] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 07:47:20 2019][1689739.790831] [] ? ldlm_export_lock_object+0x10/0x10 [ptlrpc] [Wed Feb 6 07:47:20 2019][1689739.798255] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 07:47:20 2019][1689739.806193] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 07:47:20 2019][1689739.813351] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 07:47:20 2019][1689739.820508] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 07:47:20 2019][1689739.828970] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 07:47:20 2019][1689739.837170] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 07:47:20 2019][1689739.844215] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:47:20 2019][1689739.849987] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 07:47:20 2019][1689739.857555] [] kthread+0xd1/0xe0 [Wed Feb 6 07:47:20 2019][1689739.862606] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:47:20 2019][1689739.868872] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:47:20 2019][1689739.875483] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:47:20 2019][1689739.881749] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 07:47:27 2019][1689747.624025] NMI watchdog: BUG: soft lockup - CPU#22 stuck for 23s! [migration/22:122] [Wed Feb 6 07:47:27 2019][1689747.632025] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:47:28 2019][1689747.738795] CPU: 22 PID: 122 Comm: migration/22 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:47:28 2019][1689747.751299] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:47:28 2019][1689747.759039] task: ffff913aa98cd140 ti: ffff913aa990c000 task.ti: ffff913aa990c000 [Wed Feb 6 07:47:28 2019][1689747.766692] RIP: 0010:[] [] multi_cpu_stop+0x4f/0x110 [Wed Feb 6 07:47:28 2019][1689747.775149] RSP: 0000:ffff913aa990fd98 EFLAGS: 00000246 [Wed Feb 6 07:47:28 2019][1689747.780635] RAX: 0000000000000001 RBX: ffff9139eef23af8 RCX: dead000000000200 [Wed Feb 6 07:47:28 2019][1689747.787942] RDX: ffff9149ff755ff0 RSI: 0000000000000282 RDI: ffff9139eef23ab0 [Wed Feb 6 07:47:28 2019][1689747.795249] RBP: ffff913aa990fdc0 R08: ffff9139eef23a80 R09: 0000000000000001 [Wed Feb 6 07:47:28 2019][1689747.802554] R10: 000000000000be9c R11: 0000000000000001 R12: 0000000000000086 [Wed Feb 6 07:47:28 2019][1689747.809862] R13: 000000000001ab80 R14: ffff913a00000006 R15: ffffffffadcd64f0 [Wed Feb 6 07:47:28 2019][1689747.817168] FS: 00007f78f341b880(0000) GS:ffff9149ff740000(0000) knlGS:0000000000000000 [Wed Feb 6 07:47:28 2019][1689747.825426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:47:28 2019][1689747.831346] CR2: 00007f78e39cff50 CR3: 0000004033a9a000 CR4: 00000000003407e0 [Wed Feb 6 07:47:28 2019][1689747.838652] Call Trace: [Wed Feb 6 07:47:28 2019][1689747.841280] [] ? cpu_stop_should_run+0x50/0x50 [Wed Feb 6 07:47:28 2019][1689747.847546] [] cpu_stopper_thread+0x99/0x150 [Wed Feb 6 07:47:28 2019][1689747.853640] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 07:47:28 2019][1689747.859299] [] ? cpu_stop_queue_work+0x70/0x70 [Wed Feb 6 07:47:28 2019][1689747.865567] [] smpboot_thread_fn+0x144/0x1a0 [Wed Feb 6 07:47:28 2019][1689747.871656] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 07:47:28 2019][1689747.877662] [] kthread+0xd1/0xe0 [Wed Feb 6 07:47:28 2019][1689747.882715] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:47:28 2019][1689747.888981] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:47:28 2019][1689747.895593] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:47:28 2019][1689747.901858] Code: 89 c5 48 8b 47 18 48 85 c0 0f 84 b3 00 00 00 0f a3 18 19 db 85 db 41 0f 95 c6 45 31 ff 31 c0 0f 1f 44 00 00 f3 90 41 8b 5c 24 20 <39> c3 74 5d 83 fb 02 74 68 83 fb 03 75 05 45 84 f6 75 6e f0 41 [Wed Feb 6 07:47:31 2019][1689751.556124] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [migration/14:82] [Wed Feb 6 07:47:31 2019][1689751.564044] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 07:47:31 2019][1689751.670812] CPU: 14 PID: 82 Comm: migration/14 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 07:47:31 2019][1689751.683231] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 07:47:31 2019][1689751.690970] task: ffff913aa987d140 ti: ffff913aa9898000 task.ti: ffff913aa9898000 [Wed Feb 6 07:47:31 2019][1689751.698623] RIP: 0010:[] [] multi_cpu_stop+0x4a/0x110 [Wed Feb 6 07:47:31 2019][1689751.707080] RSP: 0000:ffff913aa989bd98 EFLAGS: 00000246 [Wed Feb 6 07:47:31 2019][1689751.712566] RAX: 0000000000000001 RBX: ffff913aa9a59000 RCX: dead000000000200 [Wed Feb 6 07:47:32 2019][1689751.719874] RDX: ffff9149ff6d5ff0 RSI: 0000000000000282 RDI: ffff912d6f94fab0 [Wed Feb 6 07:47:32 2019][1689751.727180] RBP: ffff913aa989bdc0 R08: ffff912d6f94fa50 R09: 0000000000000001 [Wed Feb 6 07:47:32 2019][1689751.734485] R10: 000000000000bacb R11: 0000000000000001 R12: ffff913aa989bd20 [Wed Feb 6 07:47:32 2019][1689751.741792] R13: ffff9149f8e91040 R14: ffffffffadcd2ea0 R15: ffff913aa989bd00 [Wed Feb 6 07:47:32 2019][1689751.749100] FS: 00007f2b3eadb740(0000) GS:ffff9149ff6c0000(0000) knlGS:0000000000000000 [Wed Feb 6 07:47:32 2019][1689751.757360] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 07:47:32 2019][1689751.763277] CR2: 00000000027bf400 CR3: 0000004031fb0000 CR4: 00000000003407e0 [Wed Feb 6 07:47:32 2019][1689751.770584] Call Trace: [Wed Feb 6 07:47:32 2019][1689751.773214] [] ? cpu_stop_should_run+0x50/0x50 [Wed Feb 6 07:47:32 2019][1689751.779477] [] cpu_stopper_thread+0x99/0x150 [Wed Feb 6 07:47:32 2019][1689751.785573] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 07:47:32 2019][1689751.791231] [] smpboot_thread_fn+0x144/0x1a0 [Wed Feb 6 07:47:32 2019][1689751.797322] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 07:47:32 2019][1689751.803330] [] kthread+0xd1/0xe0 [Wed Feb 6 07:47:32 2019][1689751.808379] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:47:32 2019][1689751.814648] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:47:32 2019][1689751.821258] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:47:32 2019][1689751.827524] Code: 66 90 66 90 49 89 c5 48 8b 47 18 48 85 c0 0f 84 b3 00 00 00 0f a3 18 19 db 85 db 41 0f 95 c6 45 31 ff 31 c0 0f 1f 44 00 00 f3 90 <41> 8b 5c 24 20 39 c3 74 5d 83 fb 02 74 68 83 fb 03 75 05 45 84 [Wed Feb 6 07:47:33 2019][1689753.332166] INFO: rcu_sched self-detected stall on CPU[1689753.334169] INFO: rcu_sched detected stalls on CPUs/tasks: { 15} (detected by 26, t=240007 jiffies, g=134910432, c=134910431, q=1434043) [Wed Feb 6 07:47:33 2019][1689753.334184] Task dump for CPU 15: [Wed Feb 6 07:47:33 2019][1689753.334185] ldlm_bl_25 R running task 0 15988 2 0x00000088 [Wed Feb 6 07:47:33 2019][1689753.334188] Call Trace: [Wed Feb 6 07:47:33 2019][1689753.334234] [] ? ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 07:47:33 2019][1689753.334264] [] ? ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 07:47:33 2019][1689753.334270] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:47:33 2019][1689753.334298] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 07:47:33 2019][1689753.334301] [] ? kthread+0xd1/0xe0 [Wed Feb 6 07:47:33 2019][1689753.334302] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:47:33 2019][1689753.334306] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:47:33 2019][1689753.334308] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:47:33 2019] [Wed Feb 6 07:47:33 2019][1689753.416642] { 15} (t=240091 jiffies g=134910432 c=134910431 q=1434271) [Wed Feb 6 07:47:33 2019][1689753.423800] Task dump for CPU 15: [Wed Feb 6 07:47:33 2019][1689753.427294] ldlm_bl_25 R running task 0 15988 2 0x00000088 [Wed Feb 6 07:47:33 2019][1689753.434566] Call Trace: [Wed Feb 6 07:47:33 2019][1689753.437191] [] sched_show_task+0xa8/0x110 [Wed Feb 6 07:47:33 2019][1689753.443657] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 07:47:33 2019][1689753.449232] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 07:47:33 2019][1689753.455323] [] rcu_check_callbacks+0x442/0x730 [Wed Feb 6 07:47:33 2019][1689753.461590] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 07:47:33 2019][1689753.467856] [] update_process_times+0x46/0x80 [Wed Feb 6 07:47:33 2019][1689753.474034] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 07:47:33 2019][1689753.479954] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 07:47:33 2019][1689753.485788] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 07:47:33 2019][1689753.492053] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 07:47:33 2019][1689753.498059] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 07:47:33 2019][1689753.504759] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 07:47:33 2019][1689753.511287] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 07:47:33 2019][1689753.517637] [] ? ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 07:47:33 2019][1689753.526261] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 07:47:33 2019][1689753.533942] [] ? ldlm_export_lock_object+0x10/0x10 [ptlrpc] [Wed Feb 6 07:47:33 2019][1689753.541361] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 07:47:33 2019][1689753.549298] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 07:47:33 2019][1689753.556456] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 07:47:33 2019][1689753.563183] LustreError: 9184:0:(upcall_cache.c:233:upcall_cache_get_entry()) acquire for key 297421: error -110 [Wed Feb 6 07:47:33 2019][1689753.563187] LustreError: 9184:0:(upcall_cache.c:233:upcall_cache_get_entry()) Skipped 1 previous similar message [Wed Feb 6 07:47:33 2019][1689753.584293] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 07:47:33 2019][1689753.592753] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 07:47:33 2019][1689753.600952] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 07:47:33 2019][1689753.607999] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:47:33 2019][1689753.613772] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 07:47:33 2019][1689753.621337] [] kthread+0xd1/0xe0 [Wed Feb 6 07:47:33 2019][1689753.626389] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:47:33 2019][1689753.632655] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:47:33 2019][1689753.639269] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:47:46 2019][1689765.778527] INFO: task kworker/33:1:321 blocked for more than 120 seconds. [Wed Feb 6 07:47:46 2019][1689765.785582] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Feb 6 07:47:46 2019][1689765.793593] kworker/33:1 D ffff912aa9ad5140 0 321 2 0x00000000 [Wed Feb 6 07:47:46 2019][1689765.800873] Workqueue: events key_garbage_collector [Wed Feb 6 07:47:46 2019][1689765.805953] Call Trace: [Wed Feb 6 07:47:46 2019][1689765.808590] [] ? ttwu_do_activate+0x6f/0x80 [Wed Feb 6 07:47:46 2019][1689765.814602] [] schedule+0x29/0x70 [Wed Feb 6 07:47:46 2019][1689765.819749] [] schedule_timeout+0x221/0x2d0 [Wed Feb 6 07:47:46 2019][1689765.825768] [] ? free_pcppages_bulk+0x17e/0x3a0 [Wed Feb 6 07:47:46 2019][1689765.832126] [] wait_for_completion+0xfd/0x140 [Wed Feb 6 07:47:46 2019][1689765.838313] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:47:46 2019][1689765.844069] [] ? __call_rcu+0x2c0/0x2c0 [Wed Feb 6 07:47:46 2019][1689765.849737] [] wait_rcu_gp+0x5e/0x80 [Wed Feb 6 07:47:46 2019][1689765.855145] [] ? ftrace_raw_output_rcu_utilization+0x70/0x70 [Wed Feb 6 07:47:46 2019][1689765.862632] [] synchronize_sched+0x3b/0x50 [Wed Feb 6 07:47:46 2019][1689765.868561] [] key_garbage_collector+0x1c5/0x390 [Wed Feb 6 07:47:46 2019][1689765.875007] [] process_one_work+0x17f/0x440 [Wed Feb 6 07:47:46 2019][1689765.881024] [] worker_thread+0x126/0x3c0 [Wed Feb 6 07:47:46 2019][1689765.886778] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 07:47:46 2019][1689765.893488] [] kthread+0xd1/0xe0 [Wed Feb 6 07:47:46 2019][1689765.898546] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:47:46 2019][1689765.904827] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 07:47:46 2019][1689765.911446] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 07:47:46 2019][1689765.917746] INFO: task systemd-logind:17603 blocked for more than 120 seconds. [Wed Feb 6 07:47:46 2019][1689765.925144] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Feb 6 07:47:46 2019][1689765.933146] systemd-logind D ffff9149f3e32080 0 17603 1 0x00000084 [Wed Feb 6 07:47:46 2019][1689765.940430] Call Trace: [Wed Feb 6 07:47:46 2019][1689765.943064] [] ? ttwu_do_activate+0x6f/0x80 [Wed Feb 6 07:47:46 2019][1689765.949087] [] schedule+0x29/0x70 [Wed Feb 6 07:47:46 2019][1689765.954233] [] schedule_timeout+0x221/0x2d0 [Wed Feb 6 07:47:46 2019][1689765.960260] [] ? __slab_free+0x81/0x2f0 [Wed Feb 6 07:47:46 2019][1689765.965925] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 07:47:46 2019][1689765.971941] [] wait_for_completion+0xfd/0x140 [Wed Feb 6 07:47:46 2019][1689765.978126] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 07:47:46 2019][1689765.983884] [] ? __call_rcu+0x2c0/0x2c0 [Wed Feb 6 07:47:46 2019][1689765.989550] [] wait_rcu_gp+0x5e/0x80 [Wed Feb 6 07:47:46 2019][1689765.994959] [] ? ftrace_raw_output_rcu_utilization+0x70/0x70 [Wed Feb 6 07:47:46 2019][1689766.002446] [] ? umount_tree+0x184/0x2a0 [Wed Feb 6 07:47:46 2019][1689766.008200] [] synchronize_sched+0x3b/0x50 [Wed Feb 6 07:47:46 2019][1689766.014129] [] namespace_unlock+0x68/0x80 [Wed Feb 6 07:47:46 2019][1689766.019974] [] SyS_umount+0x239/0x410 [Wed Feb 6 07:47:46 2019][1689766.025467] [] ? ____fput+0xe/0x10 [Wed Feb 6 07:47:46 2019][1689766.030702] [] ? task_work_run+0xc0/0xe0 [Wed Feb 6 07:47:46 2019][1689766.036456] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 07:47:46 2019][1689766.627417] LustreError: 15184:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff913a93630300 x1623682074759840/t0(0) o104->fir-MDT0002@10.9.103.17@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 07:47:46 2019][1689766.648690] LustreError: 15184:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 3500 previous similar messages [Wed Feb 6 07:48:13 2019][1689793.406185] LNetError: 9030:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.15@tcp from [Wed Feb 6 07:48:13 2019][1689793.416880] LNetError: 9030:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 93717913 previous similar messages [Wed Feb 6 07:48:21 2019][1689801.428380] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549468139/real 1549468139] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549468150 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 07:48:21 2019][1689801.456252] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 94073277 previous similar messages [Wed Feb 6 07:48:35 2019][1689815.246755] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.15@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 07:48:35 2019][1689815.246785] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 151s: evicting client at 10.10.112.12@tcp ns: mdt-fir-MDT0002_UUID lock: ffff9152a53ee9c0/0x21ed9351b0229148 lrc: 3/0,0 mode: PR/PR res: [0x2c0001746:0x1:0x0].0x0 bits 0x40/0x0 rrc: 9 type: IBT flags: 0x60000400000020 nid: 10.10.112.12@tcp remote: 0xefba423e132a5792 expref: 17 pid: 6178 timeout: 0 lvb_type: 0 [Wed Feb 6 07:48:35 2019][1689815.246788] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 24638 previous similar messages [Wed Feb 6 07:48:35 2019][1689815.306660] LustreError: Skipped 11 previous similar messages [Wed Feb 6 07:51:06 2019][1689966.251545] Lustre: fir-MDT0002: Connection restored to e6807a0b-87d6-292a-3e89-235f77ec28d8 (at 10.10.113.15@tcp) [Wed Feb 6 07:51:06 2019][1689966.262075] Lustre: Skipped 2881 previous similar messages [Wed Feb 6 07:52:02 2019][1690022.645146] LustreError: 15163:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9135bb9b9800 x1623682075450480/t0(0) o104->fir-MDT0002@10.9.102.33@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 07:52:02 2019][1690022.666413] LustreError: 15163:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 10528 previous similar messages [Wed Feb 6 07:56:08 2019][1690268.258105] LustreError: 6178:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff91592ca52700 x1623682075709024 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff913e03d9c380/0x21ed9351b0f77b54 lrc: 4/0,0 mode: PR/PR res: [0x2c0001730:0x2:0x0].0x0 bits 0x5b/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e18e78 expref: 20 pid: 8982 timeout: 1690374 lvb_type: 0 [Wed Feb 6 07:56:08 2019][1690268.258397] LustreError: 15385:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff912b10710000 ns: mdt-fir-MDT0002_UUID lock: ffff913993b098c0/0x21ed9351b0f76d8c lrc: 1/0,0 mode: EX/EX res: [0x2c000172e:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.10.112.12@tcp remote: 0xefba423e132a59c9 expref: 4 pid: 15385 timeout: 0 lvb_type: 3 [Wed Feb 6 07:56:08 2019][1690268.258399] LustreError: 15385:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 5 previous similar messages [Wed Feb 6 07:56:08 2019][1690268.345200] LustreError: 6178:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 11 previous similar messages [Wed Feb 6 07:56:11 2019][1690270.685714] Lustre: fir-MDT0002: Client 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) reconnecting [Wed Feb 6 07:56:11 2019][1690270.695886] Lustre: Skipped 1793 previous similar messages [Wed Feb 6 07:58:13 2019][1690393.421237] LNetError: 6804:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.15@tcp from [Wed Feb 6 07:58:13 2019][1690393.431939] LNetError: 6804:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 145389965 previous similar messages [Wed Feb 6 07:58:21 2019][1690401.443442] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549468739/real 1549468739] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549468750 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 07:58:21 2019][1690401.471301] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 145704744 previous similar messages [Wed Feb 6 07:58:35 2019][1690415.261811] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 246s: evicting client at 10.9.103.17@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff9129557f0240/0x21ed9351afabb73c lrc: 3/0,0 mode: PR/PR res: [0x2c0001702:0x51:0x0].0x0 bits 0x40/0x0 rrc: 449957 type: IBT flags: 0x60000400010020 nid: 10.9.103.17@o2ib4 remote: 0x7117099f95343406 expref: 450038 pid: 7500 timeout: 1690372 lvb_type: 0 [Wed Feb 6 07:58:35 2019][1690415.300419] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 13947 previous similar messages [Wed Feb 6 07:58:39 2019][1690419.261899] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.15@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 07:58:39 2019][1690419.274582] LustreError: Skipped 7 previous similar messages [Wed Feb 6 08:00:34 2019][1690534.686107] LustreError: 15163:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9139cbbd2a00 x1623682077819664/t0(0) o104->fir-MDT0002@10.9.102.33@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 08:00:34 2019][1690534.707370] LustreError: 15163:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 31310 previous similar messages [Wed Feb 6 08:01:10 2019][1690570.266547] Lustre: fir-MDT0002: Connection restored to e6807a0b-87d6-292a-3e89-235f77ec28d8 (at 10.10.113.15@tcp) [Wed Feb 6 08:01:10 2019][1690570.277099] Lustre: Skipped 10 previous similar messages [Wed Feb 6 08:04:15 2019][1690755.694325] NMI watchdog: BUG: soft lockup - CPU#27 stuck for 22s! [ldlm_bl_25:15988] [Wed Feb 6 08:04:15 2019][1690755.702325] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:04:16 2019][1690755.809131] CPU: 27 PID: 15988 Comm: ldlm_bl_25 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:04:16 2019][1690755.821644] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:04:16 2019][1690755.829383] task: ffff9141bfb80000 ti: ffff91461b220000 task.ti: ffff91461b220000 [Wed Feb 6 08:04:16 2019][1690755.837036] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 08:04:16 2019][1690755.847541] RSP: 0018:ffff91461b223bb0 EFLAGS: 00000282 [Wed Feb 6 08:04:16 2019][1690755.853025] RAX: 0000000000000101 RBX: ffff913962646a20 RCX: ffff913962646a20 [Wed Feb 6 08:04:16 2019][1690755.860342] RDX: ffff91461b223ca8 RSI: ffff912e93780240 RDI: ffff911adc33ca40 [Wed Feb 6 08:04:16 2019][1690755.867656] RBP: ffff91461b223c08 R08: ffff91461b223ca8 R09: 00000000c000171d [Wed Feb 6 08:04:16 2019][1690755.874962] R10: 000000000000001d R11: ffff912e93780240 R12: ffff91461b223ca8 [Wed Feb 6 08:04:16 2019][1690755.882266] R13: 00000000c000171d R14: 000000000000001d R15: ffff912e93780240 [Wed Feb 6 08:04:16 2019][1690755.889574] FS: 00007f0efc85b880(0000) GS:ffff915a3f580000(0000) knlGS:0000000000000000 [Wed Feb 6 08:04:16 2019][1690755.897834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:04:16 2019][1690755.903754] CR2: 00007f0efc872000 CR3: 000000202df12000 CR4: 00000000003407e0 [Wed Feb 6 08:04:16 2019][1690755.911061] Call Trace: [Wed Feb 6 08:04:16 2019][1690755.913734] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 08:04:16 2019][1690755.921444] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 08:04:16 2019][1690755.929405] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 08:04:16 2019][1690755.936583] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 08:04:16 2019][1690755.943768] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 08:04:16 2019][1690755.952253] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:04:16 2019][1690755.960472] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:04:16 2019][1690755.967519] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:04:16 2019][1690755.973315] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:04:16 2019][1690755.980881] [] kthread+0xd1/0xe0 [Wed Feb 6 08:04:16 2019][1690755.985932] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:04:16 2019][1690755.992204] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:04:16 2019][1690755.998826] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:04:16 2019][1690756.005094] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 08:04:43 2019][1690783.695027] NMI watchdog: BUG: soft lockup - CPU#27 stuck for 23s! [ldlm_bl_25:15988] [Wed Feb 6 08:04:43 2019][1690783.703030] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:04:44 2019][1690783.809798] CPU: 27 PID: 15988 Comm: ldlm_bl_25 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:04:44 2019][1690783.822303] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:04:44 2019][1690783.830042] task: ffff9141bfb80000 ti: ffff91461b220000 task.ti: ffff91461b220000 [Wed Feb 6 08:04:44 2019][1690783.837694] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 08:04:44 2019][1690783.848190] RSP: 0018:ffff91461b223bb0 EFLAGS: 00000282 [Wed Feb 6 08:04:44 2019][1690783.853676] RAX: 0000000000000101 RBX: 000000000000001d RCX: ffff913962646a20 [Wed Feb 6 08:04:44 2019][1690783.860983] RDX: ffff91461b223ca8 RSI: ffff912e93780240 RDI: ffff913b78cdc380 [Wed Feb 6 08:04:44 2019][1690783.868289] RBP: ffff91461b223c08 R08: ffff91461b223ca8 R09: 00000000c000171d [Wed Feb 6 08:04:44 2019][1690783.875594] R10: 000000000000001d R11: ffff912e93780240 R12: ffff91461b223ca8 [Wed Feb 6 08:04:44 2019][1690783.882901] R13: ffff91461b223c08 R14: 000000000000001d R15: ffff912e93780240 [Wed Feb 6 08:04:44 2019][1690783.890207] FS: 00007f0efc85b880(0000) GS:ffff915a3f580000(0000) knlGS:0000000000000000 [Wed Feb 6 08:04:44 2019][1690783.898465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:04:44 2019][1690783.904387] CR2: 00007f0efc872000 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:04:44 2019][1690783.911692] Call Trace: [Wed Feb 6 08:04:44 2019][1690783.914360] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 08:04:44 2019][1690783.922055] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 08:04:44 2019][1690783.930004] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 08:04:44 2019][1690783.937172] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 08:04:44 2019][1690783.944337] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 08:04:44 2019][1690783.952806] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:04:44 2019][1690783.961016] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:04:44 2019][1690783.968064] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:04:44 2019][1690783.973844] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:04:44 2019][1690783.981412] [] kthread+0xd1/0xe0 [Wed Feb 6 08:04:44 2019][1690783.986463] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:04:44 2019][1690783.992729] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:04:44 2019][1690783.999342] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:04:44 2019][1690784.005606] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 08:04:48 2019][1690788.019132] INFO: rcu_sched self-detected stall on CPU[1690788.021132] INFO: rcu_sched detected stalls on CPUs/tasks: { 27} (detected by 15, t=60002 jiffies, g=134910663, c=134910662, q=1348764) [Wed Feb 6 08:04:48 2019][1690788.021147] Task dump for CPU 27: [Wed Feb 6 08:04:48 2019][1690788.021148] ldlm_bl_25 R running task 0 15988 2 0x00000088 [Wed Feb 6 08:04:48 2019][1690788.021152] Call Trace: [Wed Feb 6 08:04:48 2019][1690788.021195] [] ? ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:04:48 2019][1690788.021226] [] ? ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:04:48 2019][1690788.021230] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:04:48 2019][1690788.021258] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:04:48 2019][1690788.021260] [] ? kthread+0xd1/0xe0 [Wed Feb 6 08:04:48 2019][1690788.021262] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:04:48 2019][1690788.021265] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:04:48 2019][1690788.021266] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:04:48 2019] [Wed Feb 6 08:04:48 2019][1690788.103524] { 27} (t=60086 jiffies g=134910663 c=134910662 q=1350220) [Wed Feb 6 08:04:48 2019][1690788.110595] Task dump for CPU 27: [Wed Feb 6 08:04:48 2019][1690788.114088] ldlm_bl_25 R running task 0 15988 2 0x00000088 [Wed Feb 6 08:04:48 2019][1690788.121360] Call Trace: [Wed Feb 6 08:04:48 2019][1690788.123987] [] sched_show_task+0xa8/0x110 [Wed Feb 6 08:04:48 2019][1690788.130453] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 08:04:48 2019][1690788.136026] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 08:04:48 2019][1690788.142119] [] rcu_check_callbacks+0x442/0x730 [Wed Feb 6 08:04:48 2019][1690788.148385] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 08:04:48 2019][1690788.154651] [] update_process_times+0x46/0x80 [Wed Feb 6 08:04:48 2019][1690788.160831] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 08:04:48 2019][1690788.166750] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 08:04:48 2019][1690788.172583] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 08:04:48 2019][1690788.178847] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 08:04:48 2019][1690788.184855] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 08:04:48 2019][1690788.191555] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 08:04:48 2019][1690788.198081] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 08:04:48 2019][1690788.204431] [] ? ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 08:04:48 2019][1690788.213065] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 08:04:48 2019][1690788.220750] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 08:04:48 2019][1690788.228696] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 08:04:48 2019][1690788.235862] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 08:04:48 2019][1690788.243024] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 08:04:48 2019][1690788.251489] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:04:48 2019][1690788.259690] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:04:48 2019][1690788.266740] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:04:48 2019][1690788.272519] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:04:48 2019][1690788.280087] [] kthread+0xd1/0xe0 [Wed Feb 6 08:04:48 2019][1690788.285139] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:04:48 2019][1690788.291406] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:04:48 2019][1690788.298016] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:06:12 2019][1690872.273272] LustreError: 6690:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff912096b90900 x1623682079600720 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff913c6d83f740/0x21ed9351b2fc1d85 lrc: 4/0,0 mode: PR/PR res: [0x2c0001744:0x1:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e19348 expref: 26 pid: 8983 timeout: 1690978 lvb_type: 0 [Wed Feb 6 08:06:12 2019][1690872.315778] LustreError: 6690:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages [Wed Feb 6 08:07:12 2019][1690931.768740] NMI watchdog: BUG: soft lockup - CPU#35 stuck for 22s! [ldlm_bl_25:15988] [Wed Feb 6 08:07:12 2019][1690931.776745] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:07:12 2019][1690931.883516] CPU: 35 PID: 15988 Comm: ldlm_bl_25 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:07:12 2019][1690931.896019] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:07:12 2019][1690931.903759] task: ffff9141bfb80000 ti: ffff91461b220000 task.ti: ffff91461b220000 [Wed Feb 6 08:07:12 2019][1690931.911412] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 08:07:12 2019][1690931.921908] RSP: 0018:ffff91461b223bb0 EFLAGS: 00000282 [Wed Feb 6 08:07:12 2019][1690931.927393] RAX: 0000000000000101 RBX: 000000000000001d RCX: ffff913962646a20 [Wed Feb 6 08:07:12 2019][1690931.934701] RDX: ffff91461b223ca8 RSI: ffff912e93780240 RDI: ffff9122e81b2ac0 [Wed Feb 6 08:07:12 2019][1690931.942007] RBP: ffff91461b223c08 R08: ffff91461b223ca8 R09: 00000000c000171d [Wed Feb 6 08:07:12 2019][1690931.949314] R10: 000000000000001d R11: ffff912e93780240 R12: ffff91461b223ca8 [Wed Feb 6 08:07:12 2019][1690931.956619] R13: ffff91461b223c08 R14: 000000000000001d R15: ffff912e93780240 [Wed Feb 6 08:07:12 2019][1690931.963926] FS: 00007f6417cd6700(0000) GS:ffff915a3f600000(0000) knlGS:0000000000000000 [Wed Feb 6 08:07:12 2019][1690931.972185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:07:12 2019][1690931.978103] CR2: 00007fab2eafe248 CR3: 000000403b6ca000 CR4: 00000000003407e0 [Wed Feb 6 08:07:12 2019][1690931.985411] Call Trace: [Wed Feb 6 08:07:12 2019][1690931.988083] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 08:07:12 2019][1690931.995771] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 08:07:12 2019][1690932.003712] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 08:07:12 2019][1690932.010879] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 08:07:12 2019][1690932.018048] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 08:07:12 2019][1690932.026515] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:07:12 2019][1690932.034727] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:07:12 2019][1690932.041774] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:07:12 2019][1690932.047554] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:07:12 2019][1690932.055120] [] kthread+0xd1/0xe0 [Wed Feb 6 08:07:12 2019][1690932.060171] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:07:12 2019][1690932.066440] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:07:12 2019][1690932.073052] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:07:12 2019][1690932.079316] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 08:08:13 2019][1690993.436295] LNetError: 6690:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.113.15@tcp from [Wed Feb 6 08:08:13 2019][1690993.446991] LNetError: 6690:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 120073794 previous similar messages [Wed Feb 6 08:08:21 2019][1691001.458498] Lustre: 6690:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549469339/real 1549469339] req@ffff91236d275a00 x1623682082253584/t0(0) o104->fir-MDT0002@10.10.113.15@tcp:15/16 lens 296/224 e 0 to 1 dl 1549469350 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 [Wed Feb 6 08:08:21 2019][1691001.486180] Lustre: 6690:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 120100483 previous similar messages [Wed Feb 6 08:08:35 2019][1691015.276870] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 246s: evicting client at 10.9.103.17@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff9137bf7ad7c0/0x21ed9351af9acded lrc: 3/0,0 mode: PR/PR res: [0x2c0001702:0x51:0x0].0x0 bits 0x40/0x0 rrc: 447255 type: IBT flags: 0x60000400010020 nid: 10.9.103.17@o2ib4 remote: 0x7117099f9530787f expref: 447336 pid: 15275 timeout: 1690972 lvb_type: 0 [Wed Feb 6 08:08:35 2019][1691015.315567] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 33691 previous similar messages [Wed Feb 6 08:08:43 2019][1691023.277062] LustreError: 138-a: fir-MDT0002: A client on nid 10.10.113.15@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 08:08:43 2019][1691023.289753] LustreError: Skipped 3 previous similar messages [Wed Feb 6 08:08:46 2019][1691026.319836] Lustre: fir-MDT0002: Client 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) reconnecting [Wed Feb 6 08:10:32 2019][1691132.054820] Lustre: fir-OST001f-osc-MDT0002: Connection to fir-OST001f (at 10.0.10.106@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 08:10:32 2019][1691132.071056] Lustre: Skipped 8 previous similar messages [Wed Feb 6 08:10:32 2019][1691132.073948] Lustre: MGS: Received new LWP connection from 10.9.101.61@o2ib4, removing former export from same NID [Wed Feb 6 08:10:32 2019][1691132.073951] Lustre: Skipped 453 previous similar messages [Wed Feb 6 08:10:37 2019][1691137.204712] LustreError: 15163:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9138ee652d00 x1623682090843520/t0(0) o104->fir-MDT0002@10.9.102.33@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 08:10:37 2019][1691137.225985] LustreError: 15163:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 460673 previous similar messages [Wed Feb 6 08:10:41 2019][1691140.909709] Lustre: mgs: This server is not able to keep up with request traffic (cpu-bound). [Wed Feb 6 08:10:41 2019][1691140.918412] Lustre: Skipped 1 previous similar message [Wed Feb 6 08:10:41 2019][1691140.923733] Lustre: 6048:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=0, svcEst=32, delay=0 [Wed Feb 6 08:10:41 2019][1691140.933907] Lustre: 6048:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 1 previous similar message [Wed Feb 6 08:10:41 2019][1691140.943397] Lustre: 6048:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-2s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff9145edbb7c50 x1624712198523680/t0(0) o400->84f47100-42cc-ffab-f145-c5a65283c902@10.8.20.9@o2ib6:342/0 lens 224/0 e 0 to 0 dl 1549469477 ref 2 fl New:/0/ffffffff rc 0/-1 [Wed Feb 6 08:10:41 2019][1691140.974283] Lustre: 6048:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 146 previous similar messages [Wed Feb 6 08:10:41 2019][1691140.984461] LustreError: 6048:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.20.9@o2ib6: deadline 6:2s ago [Wed Feb 6 08:10:41 2019][1691140.984461] req@ffff9145edbb7c50 x1624712198523680/t0(0) o400->84f47100-42cc-ffab-f145-c5a65283c902@10.8.20.9@o2ib6:342/0 lens 224/0 e 0 to 0 dl 1549469477 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Feb 6 08:10:41 2019][1691141.016218] LustreError: 6048:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 162 previous similar messages [Wed Feb 6 08:10:41 2019][1691141.027002] Lustre: 6048:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:2s); client may timeout. req@ffff9145edbb7c50 x1624712198523680/t0(0) o400->84f47100-42cc-ffab-f145-c5a65283c902@10.8.20.9@o2ib6:342/0 lens 224/0 e 0 to 0 dl 1549469477 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Feb 6 08:10:41 2019][1691141.055805] Lustre: 6048:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 172 previous similar messages [Wed Feb 6 08:10:44 2019][1691144.611436] Lustre: MGS: Received new LWP connection from 10.9.0.1@o2ib4, removing former export from same NID [Wed Feb 6 08:10:49 2019][1691149.669628] LNetError: 5879:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) [Wed Feb 6 08:10:49 2019][1691149.682233] LNetError: 5879:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages [Wed Feb 6 08:10:56 2019][1691156.240591] Lustre: fir-OST002b-osc-MDT0000: Connection to fir-OST002b (at 10.0.10.108@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 08:10:56 2019][1691156.256838] Lustre: Skipped 20 previous similar messages [Wed Feb 6 08:11:02 2019][1691162.361185] Lustre: mdt_readpage: This server is not able to keep up with request traffic (cpu-bound). [Wed Feb 6 08:11:02 2019][1691162.370666] Lustre: 12704:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=0 reqQ=0 recA=7, svcEst=35, delay=7385 [Wed Feb 6 08:11:03 2019][1691163.716684] Lustre: 9147:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 10s req@ffff9149f9a25050 x1624705147242864/t0(0) o35->d35217f0-5ade-6000-33a4-4fb011efe6c5@10.8.4.8@o2ib6:0/0 lens 392/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 [Wed Feb 6 08:11:08 2019][1691168.698622] LNetError: 5881:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) [Wed Feb 6 08:11:10 2019][1691170.746723] Lustre: mdt_readpage: This server is not able to keep up with request traffic (cpu-bound). [Wed Feb 6 08:11:11 2019][1691170.756203] Lustre: 9151:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=13, svcEst=35, delay=30170 [Wed Feb 6 08:11:11 2019][1691170.766820] Lustre: 9151:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-10s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff912f9f865400 x1624699201436800/t0(0) o35->dd23da3a-484d-0eb4-af89-89a069ff0621@10.9.104.49@o2ib4:363/0 lens 392/456 e 0 to 0 dl 1549469498 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:11:12 2019][1691172.404153] Lustre: MGS: Received new LWP connection from 10.8.12.23@o2ib6, removing former export from same NID [Wed Feb 6 08:11:12 2019][1691172.409115] Lustre: fir-OST000c-osc-MDT0000: Connection restored to 10.0.10.103@o2ib7 (at 10.0.10.103@o2ib7) [Wed Feb 6 08:11:12 2019][1691172.409118] Lustre: Skipped 543 previous similar messages [Wed Feb 6 08:11:12 2019][1691172.430070] Lustre: Skipped 108 previous similar messages [Wed Feb 6 08:11:12 2019][1691172.511388] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.201@o2ib7: 24 seconds [Wed Feb 6 08:11:12 2019][1691172.521730] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3730 previous similar messages [Wed Feb 6 08:11:27 2019][1691187.710179] LustreError: 27730:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff911d96ccb850 x1624699216438768/t0(0) o4->64d3667d-1c77-7867-b1f3-ec9c4a6035ad@10.9.104.56@o2ib4:422/0 lens 488/448 e 1 to 0 dl 1549469557 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:11:27 2019][1691187.734478] LustreError: 27730:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 1 previous similar message [Wed Feb 6 08:11:27 2019][1691187.743988] Lustre: fir-MDT0002: Bulk IO write error with 64d3667d-1c77-7867-b1f3-ec9c4a6035ad (at 10.9.104.56@o2ib4), client will retry: rc = -110 [Wed Feb 6 08:11:30 2019][1691189.928251] Lustre: fir-OST0014-osc-MDT0000: Connection to fir-OST0014 (at 10.0.10.103@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 08:11:30 2019][1691189.944504] Lustre: Skipped 19 previous similar messages [Wed Feb 6 08:11:31 2019][1691191.614166] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.201@o2ib7: 17 seconds [Wed Feb 6 08:11:31 2019][1691191.624520] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 45 previous similar messages [Wed Feb 6 08:11:31 2019][1691191.634123] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 3 seconds [Wed Feb 6 08:11:31 2019][1691191.644207] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 4 previous similar messages [Wed Feb 6 08:11:31 2019][1691191.654473] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.103@o2ib7 (8): c: 1, oc: 0, rc: 8 [Wed Feb 6 08:11:31 2019][1691191.666552] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 4 previous similar messages [Wed Feb 6 08:11:31 2019][1691191.676659] Lustre: 7470:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (25:31s); client may timeout. req@ffff912f9f865400 x1624699201436800/t13606332798(0) o35->dd23da3a-484d-0eb4-af89-89a069ff0621@10.9.104.49@o2ib4:363/0 lens 392/424 e 0 to 0 dl 1549469498 ref 1 fl Complete:/0/0 rc 0/0 [Wed Feb 6 08:11:38 2019][1691198.120448] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.103@o2ib7: 1 seconds [Wed Feb 6 08:11:38 2019][1691198.130701] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 395 previous similar messages [Wed Feb 6 08:11:40 2019][1691200.583927] LNetError: 7282:0:(o2iblnd_cb.c:3281:kiblnd_cm_callback()) 10.0.10.204@o2ib7 DISCONNECTED [Wed Feb 6 08:11:40 2019][1691200.593322] LNetError: 7282:0:(o2iblnd_cb.c:3281:kiblnd_cm_callback()) Skipped 3 previous similar messages [Wed Feb 6 08:11:44 2019][1691204.000717] LustreError: 9000:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(131072) req@ffff9159fc4adc50 x1624699398921792/t0(0) o4->2daf93cd-09cc-690c-e30a-05a92ce8115a@10.9.104.51@o2ib4:428/0 lens 488/448 e 1 to 0 dl 1549469563 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:11:44 2019][1691204.025741] Lustre: fir-MDT0002: Bulk IO write error with 2daf93cd-09cc-690c-e30a-05a92ce8115a (at 10.9.104.51@o2ib4), client will retry: rc = -110 [Wed Feb 6 08:11:44 2019][1691204.450665] Lustre: MGS: Received new LWP connection from 10.8.13.13@o2ib6, removing former export from same NID [Wed Feb 6 08:11:44 2019][1691204.461014] Lustre: Skipped 144 previous similar messages [Wed Feb 6 08:11:48 2019][1691208.120692] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.204@o2ib7: 1 seconds [Wed Feb 6 08:11:48 2019][1691208.130948] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 19 previous similar messages [Wed Feb 6 08:11:54 2019][1691213.875159] LustreError: 27749:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff913be5f47c50 x1624711747876624/t0(0) o4->8b9d39cd-5973-2d32-c0c6-445ad6e2af9d@10.8.6.15@o2ib6:522/0 lens 488/448 e 0 to 0 dl 1549469657 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:11:54 2019][1691213.899326] Lustre: fir-MDT0002: Bulk IO write error with 8b9d39cd-5973-2d32-c0c6-445ad6e2af9d (at 10.8.6.15@o2ib6), client will retry: rc = -110 [Wed Feb 6 08:12:09 2019][1691229.509954] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Wed Feb 6 08:12:09 2019][1691229.520210] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 1 previous similar message [Wed Feb 6 08:12:09 2019][1691229.530378] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.210@o2ib7 (1): c: 0, oc: 2, rc: 8 [Wed Feb 6 08:12:09 2019][1691229.542448] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 1 previous similar message [Wed Feb 6 08:12:09 2019][1691229.692463] LustreError: 27688:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff911d96ccf450 x1624732283432880/t0(0) o4->2d486816-0fee-3e02-cb25-89549e06c293@10.9.104.52@o2ib4:460/0 lens 488/448 e 0 to 0 dl 1549469595 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Feb 6 08:12:09 2019][1691229.716801] Lustre: fir-MDT0002: Bulk IO write error with 2d486816-0fee-3e02-cb25-89549e06c293 (at 10.9.104.52@o2ib4), client will retry: rc = -110 [Wed Feb 6 08:12:11 2019][1691231.610265] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 23s! [ldlm_cn00_023:20040] [Wed Feb 6 08:12:11 2019][1691231.618528] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:11 2019][1691231.725296] CPU: 16 PID: 20040 Comm: ldlm_cn00_023 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:11 2019][1691231.738061] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:11 2019][1691231.745801] task: ffff9149ca785140 ti: ffff915909ff0000 task.ti: ffff915909ff0000 [Wed Feb 6 08:12:11 2019][1691231.753454] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:11 2019][1691231.763567] RSP: 0018:ffff915909ff3c38 EFLAGS: 00000246 [Wed Feb 6 08:12:12 2019][1691231.769052] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000810000 [Wed Feb 6 08:12:12 2019][1691231.776361] RDX: ffff9139ff81b780 RSI: 0000000001090001 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:12 2019][1691231.783666] RBP: ffff915909ff3c38 R08: ffff9129fef1b780 R09: 0000000000000000 [Wed Feb 6 08:12:12 2019][1691231.790972] R10: ffff911b3fc07600 R11: ffff9127df78b660 R12: ffff915909ff3be0 [Wed Feb 6 08:12:12 2019][1691231.798277] R13: ffff914fcb37fc80 R14: ffff915909ff3ba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:12 2019][1691231.805585] FS: 00007f78f341b880(0000) GS:ffff9129fef00000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:12 2019][1691231.813843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:12 2019][1691231.819764] CR2: 00007f63fb343000 CR3: 0000004033a9a000 CR4: 00000000003407e0 [Wed Feb 6 08:12:12 2019][1691231.827070] Call Trace: [Wed Feb 6 08:12:12 2019][1691231.829702] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:12 2019][1691231.836149] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:12 2019][1691231.841859] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:12 2019][1691231.848587] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:12 2019][1691231.855327] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:12 2019][1691231.862407] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:12 2019][1691231.869312] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:12 2019][1691231.876402] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:12 2019][1691231.884269] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:12 2019][1691231.891234] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:12 2019][1691231.897672] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:12 2019][1691231.903716] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:12 2019][1691231.910199] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:12 2019][1691231.917762] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:12 2019][1691231.922812] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:12 2019][1691231.929081] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:12 2019][1691231.935692] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:12 2019][1691231.941957] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:12:12 2019][1691232.722866] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.210@o2ib7: 3 seconds [Wed Feb 6 08:12:12 2019][1691232.733123] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 31 previous similar messages [Wed Feb 6 08:12:13 2019][1691232.937200] LustreError: 15460:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff91365012cc50 x1624712309495344/t0(0) o4->3e106753-f830-b044-4cf1-87da4380b4a5@10.8.6.12@o2ib6:524/0 lens 488/448 e 0 to 0 dl 1549469659 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:12:19 2019][1691239.559464] NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [ldlm_cn02_016:23319] [Wed Feb 6 08:12:19 2019][1691239.567724] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class[1691239.670467] NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ldlm_cn03_024:28512] [Wed Feb 6 08:12:19 2019][1691239.670469] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:20 2019][1691239.670568] CPU: 23 PID: 28512 Comm: ldlm_cn03_024 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:20 2019][1691239.670569] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:20 2019][1691239.670571] task: ffff9156a426b0c0 ti: ffff915338b40000 task.ti: ffff915338b40000 [Wed Feb 6 08:12:20 2019][1691239.670572] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:20 2019][1691239.670579] RSP: 0018:ffff915338b43bb8 EFLAGS: 00000246 [Wed Feb 6 08:12:20 2019][1691239.670580] RAX: 0000000000000000 RBX: ffff915338b43bf0 RCX: 0000000000b90000 [Wed Feb 6 08:12:20 2019][1691239.670582] RDX: ffff9129fee5b780 RSI: 0000000000210000 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:20 2019][1691239.670583] RBP: ffff915338b43bb8 R08: ffff915a3f55b780 R09: 0000000000000000 [Wed Feb 6 08:12:20 2019][1691239.670584] R10: ffff911b3fc07600 R11: 0000000000000400 R12: 0000000000000000 [Wed Feb 6 08:12:20 2019][1691239.670584] R13: 0000000000000000 R14: 0000000000000002 R15: ffff912ec14bd580 [Wed Feb 6 08:12:20 2019][1691239.670586] FS: 00007fe2d3435880(0000) GS:ffff915a3f540000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:20 2019][1691239.670587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:20 2019][1691239.670589] CR2: 00007f2781328000 CR3: 000000403a6c6000 CR4: 00000000003407e0 [Wed Feb 6 08:12:20 2019][1691239.670590] Call Trace: [Wed Feb 6 08:12:20 2019][1691239.670595] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:20 2019][1691239.670599] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:20 2019][1691239.670658] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.670695] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.670699] [] ? native_queued_spin_lock_slowpath+0x156/0x200 [Wed Feb 6 08:12:20 2019][1691239.670735] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.670776] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.670817] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.670854] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.670901] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.670946] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.670949] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:20 2019][1691239.670952] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:20 2019][1691239.670995] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.671039] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.671042] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:20 2019][1691239.671045] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:20 2019][1691239.671048] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:20 2019][1691239.671051] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:20 2019][1691239.671052] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:12:20 2019][1691239.687799] LustreError: 17102:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(131072) req@ffff9159a0616850 x1624699201436816/t0(0) o4->dd23da3a-484d-0eb4-af89-89a069ff0621@10.9.104.49@o2ib4:451/0 lens 488/448 e 2 to 0 dl 1549469586 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Feb 6 08:12:20 2019][1691239.687832] Lustre: fir-MDT0002: Bulk IO write error with dd23da3a-484d-0eb4-af89-89a069ff0621 (at 10.9.104.49@o2ib4), client will retry: rc = -110 [Wed Feb 6 08:12:20 2019][1691239.687834] Lustre: Skipped 1 previous similar message [Wed Feb 6 08:12:20 2019][1691239.696466] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [ldlm_cn02_018:23322] [Wed Feb 6 08:12:20 2019][1691239.696466] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:20 2019][1691239.696522] CPU: 26 PID: 23322 Comm: ldlm_cn02_018 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:20 2019][1691239.696523] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:20 2019][1691239.696524] task: ffff9159e661e180 ti: ffff9158796f0000 task.ti: ffff9158796f0000 [Wed Feb 6 08:12:20 2019][1691239.696526] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:20 2019][1691239.696534] RSP: 0018:ffff9158796f3c38 EFLAGS: 00000246 [Wed Feb 6 08:12:20 2019][1691239.696535] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000d10000 [Wed Feb 6 08:12:20 2019][1691239.696536] RDX: ffff9139ff85b780 RSI: 0000000001290001 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:20 2019][1691239.696537] RBP: ffff9158796f3c38 R08: ffff9149ff79b780 R09: 0000000000000000 [Wed Feb 6 08:12:20 2019][1691239.696538] R10: ffff911b3fc07600 R11: 0000000000000001 R12: ffff9158796f3be0 [Wed Feb 6 08:12:20 2019][1691239.696538] R13: ffff914fcb37fc80 R14: ffff9158796f3ba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:20 2019][1691239.696540] FS: 00007f72eaa7c700(0000) GS:ffff9149ff780000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:20 2019][1691239.696541] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:20 2019][1691239.696542] CR2: 00007fcf5e129024 CR3: 000000403bd1e000 CR4: 00000000003407e0 [Wed Feb 6 08:12:20 2019][1691239.696543] Call Trace: [Wed Feb 6 08:12:20 2019][1691239.696550] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:20 2019][1691239.696554] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:20 2019][1691239.696600] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.696634] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.696672] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.696709] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.696745] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.696788] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.696828] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.696832] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:12:20 2019][1691239.696871] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.696910] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.696913] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:20 2019][1691239.696915] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:20 2019][1691239.696918] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:20 2019][1691239.696919] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:20 2019][1691239.696920] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:12:20 2019][1691239.776468] NMI watchdog: BUG: soft lockup - CPU#35 stuck for 22s! [ldlm_cn03_025:28518] [Wed Feb 6 08:12:20 2019][1691239.776469] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:20 2019][1691239.776539] CPU: 35 PID: 28518 Comm: ldlm_cn03_025 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:20 2019][1691239.776540] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:20 2019][1691239.776541] task: ffff9149d5fc9040 ti: ffff913ac391c000 task.ti: ffff913ac391c000 [Wed Feb 6 08:12:20 2019][1691239.776542] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:20 2019][1691239.776548] RSP: 0018:ffff913ac391fc38 EFLAGS: 00000246 [Wed Feb 6 08:12:20 2019][1691239.776549] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001190000 [Wed Feb 6 08:12:20 2019][1691239.776550] RDX: ffff9149ff8db780 RSI: 0000000001710001 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:20 2019][1691239.776550] RBP: ffff913ac391fc38 R08: ffff915a3f61b780 R09: 0000000000000000 [Wed Feb 6 08:12:20 2019][1691239.776551] R10: ffff911b3fc07600 R11: 0000000000003c00 R12: ffff913ac391fbe0 [Wed Feb 6 08:12:20 2019][1691239.776552] R13: ffff914fcb37fc80 R14: ffff913ac391fba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:20 2019][1691239.776553] FS: 00007f2b3eadb740(0000) GS:ffff915a3f600000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:20 2019][1691239.776554] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:20 2019][1691239.776555] CR2: 00007f2b3d82280d CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:12:20 2019][1691239.776556] Call Trace: [Wed Feb 6 08:12:20 2019][1691239.776560] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:20 2019][1691239.776564] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:20 2019][1691239.776609] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.776637] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.776669] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.776700] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.776730] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.776767] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.776801] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.776804] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:20 2019][1691239.776806] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:20 2019][1691239.776839] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.776872] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:20 2019][1691239.776874] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:20 2019][1691239.776876] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:20 2019][1691239.776878] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:21 2019][1691239.776880] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:21 2019][1691239.776880] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:12:21 2019] [Wed Feb 6 08:12:21 2019][1691240.778226] scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:21 2019][1691240.783877] CPU: 10 PID: 23319 Comm: ldlm_cn02_016 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:21 2019][1691240.796641] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:21 2019][1691240.804382] task: ffff9159e6619040 ti: ffff9158f7338000 task.ti: ffff9158f7338000 [Wed Feb 6 08:12:21 2019][1691240.812034] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:21 2019][1691240.822141] RSP: 0018:ffff9158f733bc38 EFLAGS: 00000246 [Wed Feb 6 08:12:21 2019][1691240.827626] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000510000 [Wed Feb 6 08:12:21 2019][1691240.834932] RDX: ffff9149ff81b780 RSI: 0000000001110001 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:21 2019][1691240.842237] RBP: ffff9158f733bc38 R08: ffff9149ff69b780 R09: 0000000000000000 [Wed Feb 6 08:12:21 2019][1691240.849543] R10: ffff911b3fc07600 R11: 00000000000001ce R12: ffff9158f733bbe0 [Wed Feb 6 08:12:21 2019][1691240.856851] R13: ffff914fcb37fc80 R14: ffff9158f733bba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:21 2019][1691240.864157] FS: 00007f72eaa7c700(0000) GS:ffff9149ff680000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:21 2019][1691240.872417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:21 2019][1691240.878336] CR2: 00007f0ff836e518 CR3: 000000403bd1e000 CR4: 00000000003407e0 [Wed Feb 6 08:12:21 2019][1691240.885641] Call Trace: [Wed Feb 6 08:12:21 2019][1691240.888271] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:21 2019][1691240.894709] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:21 2019][1691240.900403] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:21 2019][1691240.907134] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:21 2019][1691240.913870] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:21 2019][1691240.920952] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:21 2019][1691240.927859] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:21 2019][1691240.934944] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:21 2019][1691240.942804] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:21 2019][1691240.949762] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:21 2019][1691240.956200] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:21 2019][1691240.962243] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:21 2019][1691240.968718] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:21 2019][1691240.976281] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:21 2019][1691240.981333] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:21 2019][1691240.987601] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:21 2019][1691240.994213] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:21 2019][1691241.000479] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:12:22 2019][1691241.749617] LustreError: 27687:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(131072) req@ffff9129f261c050 x1624699216438832/t0(0) o4->64d3667d-1c77-7867-b1f3-ec9c4a6035ad@10.9.104.56@o2ib4:476/0 lens 488/448 e 0 to 0 dl 1549469611 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Feb 6 08:12:22 2019][1691241.774697] LustreError: 27687:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message [Wed Feb 6 08:12:23 2019][1691243.518564] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [ldlm_cn01_021:27821] [Wed Feb 6 08:12:23 2019][1691243.526742] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:23 2019][1691243.633511] CPU: 5 PID: 27821 Comm: ldlm_cn01_021 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:23 2019][1691243.646190] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:23 2019][1691243.653930] task: ffff9139f6230000 ti: ffff915751354000 task.ti: ffff915751354000 [Wed Feb 6 08:12:23 2019][1691243.661580] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:23 2019][1691243.671697] RSP: 0018:ffff915751357bb8 EFLAGS: 00000246 [Wed Feb 6 08:12:23 2019][1691243.677182] RAX: 0000000000000000 RBX: ffff915751357bf0 RCX: 0000000000290000 [Wed Feb 6 08:12:23 2019][1691243.684488] RDX: ffff9149ff89b780 RSI: 0000000001510000 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:23 2019][1691243.691795] RBP: ffff915751357bb8 R08: ffff9139ff65b780 R09: 0000000000000000 [Wed Feb 6 08:12:23 2019][1691243.699101] R10: ffff911b3fc07600 R11: ffffdbb25bf3a800 R12: 0000000000000000 [Wed Feb 6 08:12:23 2019][1691243.706407] R13: 0000000000000000 R14: 0000000000000002 R15: ffff91568e343f00 [Wed Feb 6 08:12:23 2019][1691243.713712] FS: 00007f35005b3880(0000) GS:ffff9139ff640000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:23 2019][1691243.721972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:23 2019][1691243.727891] CR2: 00007fbe3d5a4248 CR3: 0000001035820000 CR4: 00000000003407e0 [Wed Feb 6 08:12:23 2019][1691243.731571] NMI watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [ldlm_cn02_007:9032] [Wed Feb 6 08:12:23 2019][1691243.731618] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:24 2019][1691243.731639] CPU: 30 PID: 9032 Comm: ldlm_cn02_007 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:24 2019][1691243.731639] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:24 2019][1691243.731641] task: ffff9159b7552080 ti: ffff9159a1a78000 task.ti: ffff9159a1a78000 [Wed Feb 6 08:12:24 2019][1691243.731649] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:24 2019][1691243.731650] RSP: 0018:ffff9159a1a7bbb8 EFLAGS: 00000246 [Wed Feb 6 08:12:24 2019][1691243.731651] RAX: 0000000000000000 RBX: ffff9159a1a7bbf0 RCX: 0000000000f10000 [Wed Feb 6 08:12:24 2019][1691243.731652] RDX: ffff9139ff65b780 RSI: 0000000000290000 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:24 2019][1691243.731653] RBP: ffff9159a1a7bbb8 R08: ffff9149ff7db780 R09: 0000000000000000 [Wed Feb 6 08:12:24 2019][1691243.731654] R10: ffff911b3fc07600 R11: 0000000000000000 R12: 0000000000000000 [Wed Feb 6 08:12:24 2019][1691243.731654] R13: 0000000000000000 R14: 0000000000000002 R15: ffff9136fd709f80 [Wed Feb 6 08:12:24 2019][1691243.731656] FS: 00007f78f341b880(0000) GS:ffff9149ff7c0000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:24 2019][1691243.731657] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:24 2019][1691243.731658] CR2: 0000000000485000 CR3: 0000004033a9a000 CR4: 00000000003407e0 [Wed Feb 6 08:12:24 2019][1691243.731659] Call Trace: [Wed Feb 6 08:12:24 2019][1691243.731665] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:24 2019][1691243.731669] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:24 2019][1691243.731712] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.731745] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.731748] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:24 2019][1691243.731780] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.731817] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.731853] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.731888] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.731930] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.731969] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.731973] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:12:24 2019][1691243.732012] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.732050] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.732054] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:24 2019][1691243.732056] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:24 2019][1691243.732059] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:24 2019][1691243.732060] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:24 2019][1691243.732080] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:12:24 2019][1691243.810570] NMI watchdog: BUG: soft lockup - CPU#39 stuck for 22s! [ldlm_cn03_017:63248] [Wed Feb 6 08:12:24 2019][1691243.810627] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:24 2019][1691243.810651] CPU: 39 PID: 63248 Comm: ldlm_cn03_017 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:24 2019][1691243.810652] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:24 2019][1691243.810653] task: ffff9149d62d6180 ti: ffff9141a1abc000 task.ti: ffff9141a1abc000 [Wed Feb 6 08:12:24 2019][1691243.810661] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:24 2019][1691243.810662] RSP: 0018:ffff9141a1abfc38 EFLAGS: 00000246 [Wed Feb 6 08:12:24 2019][1691243.810664] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001390000 [Wed Feb 6 08:12:24 2019][1691243.810665] RDX: ffff9149ff89b780 RSI: 0000000001510000 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:24 2019][1691243.810665] RBP: ffff9141a1abfc38 R08: ffff915a3f65b780 R09: 0000000000000000 [Wed Feb 6 08:12:24 2019][1691243.810666] R10: ffff911b3fc07600 R11: ffffdbb2fecbc000 R12: ffff9141a1abfbe0 [Wed Feb 6 08:12:24 2019][1691243.810667] R13: ffff914fcb37fc80 R14: ffff9141a1abfba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:24 2019][1691243.810669] FS: 00007f4ff05b7880(0000) GS:ffff915a3f640000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:24 2019][1691243.810670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:24 2019][1691243.810671] CR2: 00007f53f6ea3248 CR3: 0000003f8eb5e000 CR4: 00000000003407e0 [Wed Feb 6 08:12:24 2019][1691243.810673] Call Trace: [Wed Feb 6 08:12:24 2019][1691243.810678] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:24 2019][1691243.810682] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:24 2019][1691243.810740] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.810777] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.810818] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.810859] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.810897] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.810944] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.810987] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.810991] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:12:24 2019][1691243.811033] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.811075] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.811079] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:24 2019][1691243.811081] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:24 2019][1691243.811084] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:24 2019][1691243.811087] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:24 2019][1691243.811110] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:12:24 2019][1691243.871573] NMI watchdog: BUG: soft lockup - CPU#46 stuck for 22s! [ldlm_cn02_017:23320] [Wed Feb 6 08:12:24 2019][1691243.871601] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:24 2019][1691243.871614] CPU: 46 PID: 23320 Comm: ldlm_cn02_017 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:24 2019][1691243.871614] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:24 2019][1691243.871615] task: ffff9159e661c100 ti: ffff91580b2b0000 task.ti: ffff91580b2b0000 [Wed Feb 6 08:12:24 2019][1691243.871619] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:24 2019][1691243.871620] RSP: 0018:ffff91580b2b3c38 EFLAGS: 00000246 [Wed Feb 6 08:12:24 2019][1691243.871621] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001710000 [Wed Feb 6 08:12:24 2019][1691243.871622] RDX: ffff9129fefdb780 RSI: 0000000000e10001 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:24 2019][1691243.871623] RBP: ffff91580b2b3c38 R08: ffff9149ff8db780 R09: 0000000000000000 [Wed Feb 6 08:12:24 2019][1691243.871624] R10: ffff911b3fc07600 R11: 0000000000000001 R12: ffff91580b2b3be0 [Wed Feb 6 08:12:24 2019][1691243.871624] R13: ffff914fcb37fc80 R14: ffff91580b2b3ba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:24 2019][1691243.871626] FS: 00007f277afbf700(0000) GS:ffff9149ff8c0000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:24 2019][1691243.871627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:24 2019][1691243.871628] CR2: 00007f275c005088 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:12:24 2019][1691243.871628] Call Trace: [Wed Feb 6 08:12:24 2019][1691243.871632] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:24 2019][1691243.871634] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:24 2019][1691243.871666] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.871699] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.871734] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.871770] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.871805] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.871845] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.871884] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.871886] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:12:24 2019][1691243.871924] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.871962] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:24 2019][1691243.871964] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:24 2019][1691243.871966] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:24 2019][1691243.871968] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:24 2019][1691243.871969] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:25 2019][1691243.871989] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:12:25 2019][1691244.771772] Call Trace: [Wed Feb 6 08:12:25 2019][1691244.774405] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:25 2019][1691244.780852] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:25 2019][1691244.786543] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:25 2019][1691244.793273] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:12:25 2019][1691244.800322] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:25 2019][1691244.807919] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:12:25 2019][1691244.814658] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:25 2019][1691244.821738] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:25 2019][1691244.828643] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:25 2019][1691244.835732] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:25 2019][1691244.843599] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:25 2019][1691244.850558] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:12:25 2019][1691244.856338] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:25 2019][1691244.862814] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:25 2019][1691244.870379] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:25 2019][1691244.875428] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:25 2019][1691244.881695] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:25 2019][1691244.888309] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:25 2019][1691244.894574] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:12:26 2019][1691246.188125] LustreError: 27700:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(17000) req@ffff913acba08050 x1624711763265168/t0(0) o4->56748acc-0114-c8d8-7a9c-c6362a34cc0f@10.8.4.3@o2ib6:479/0 lens 488/448 e 3 to 0 dl 1549469614 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:12:26 2019][1691246.212872] LustreError: 27700:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message [Wed Feb 6 08:12:27 2019][1691247.687669] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! [ldlm_cn01_000:6032] [Wed Feb 6 08:12:27 2019][1691247.695847] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:28 2019][1691247.802635] CPU: 25 PID: 6032 Comm: ldlm_cn01_000 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:28 2019][1691247.815310] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:28 2019][1691247.823051] task: ffff9129f3d26180 ti: ffff9159f97d0000 task.ti: ffff9159f97d0000 [Wed Feb 6 08:12:28 2019][1691247.830702] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:28 2019][1691247.840817] RSP: 0018:ffff9159f97d3c38 EFLAGS: 00000246 [Wed Feb 6 08:12:28 2019][1691247.846302] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000c90000 [Wed Feb 6 08:12:28 2019][1691247.853609] RDX: ffff9129fef5b780 RSI: 0000000000a10001 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:28 2019][1691247.860916] RBP: ffff9159f97d3c38 R08: ffff9139ff79b780 R09: 0000000000000000 [Wed Feb 6 08:12:28 2019][1691247.868221] R10: ffff911b3fc07600 R11: 0000000000000000 R12: ffff9159f97d3be0 [Wed Feb 6 08:12:28 2019][1691247.875527] R13: ffff914fcb37fc80 R14: ffff9159f97d3ba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:28 2019][1691247.882835] FS: 00007f277d7c4700(0000) GS:ffff9139ff780000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:28 2019][1691247.891094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:28 2019][1691247.897014] CR2: 00007f2781328000 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:12:28 2019][1691247.904318] Call Trace: [Wed Feb 6 08:12:28 2019][1691247.906950] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:28 2019][1691247.913396] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:28 2019][1691247.919087] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:28 2019][1691247.925819] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:28 2019][1691247.932556] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:28 2019][1691247.939637] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:28 2019][1691247.946543] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:28 2019][1691247.953632] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:28 2019][1691247.961499] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:28 2019][1691247.968457] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:12:28 2019][1691247.974238] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:28 2019][1691247.980712] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:28 2019][1691247.988276] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:28 2019][1691247.993328] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:28 2019][1691247.999595] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:28 2019][1691248.006207] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:28 2019][1691248.012474] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:12:31 2019][1691251.526767] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [ldlm_cn02_006:6776] [Wed Feb 6 08:12:31 2019][1691251.534856] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:31 2019][1691251.641624] CPU: 6 PID: 6776 Comm: ldlm_cn02_006 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:31 2019][1691251.654215] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:31 2019][1691251.661955] task: ffff9159a2766180 ti: ffff9159f3e34000 task.ti: ffff9159f3e34000 [Wed Feb 6 08:12:31 2019][1691251.669606] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:31 2019][1691251.679712] RSP: 0018:ffff9159f3e37c38 EFLAGS: 00000246 [Wed Feb 6 08:12:31 2019][1691251.685200] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000310000 [Wed Feb 6 08:12:31 2019][1691251.692505] RDX: ffff9139ff7db780 RSI: 0000000000e90001 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:31 2019][1691251.699812] RBP: ffff9159f3e37c38 R08: ffff9149ff65b780 R09: 0000000000000000 [Wed Feb 6 08:12:31 2019][1691251.707118] R10: ffff911b3fc07600 R11: ffff9149fa30ea20 R12: ffff9159f3e37be0 [Wed Feb 6 08:12:31 2019][1691251.714424] R13: ffff914fcb37fc80 R14: ffff9159f3e37ba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:31 2019][1691251.721730] FS: 00007f747490e700(0000) GS:ffff9149ff640000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:31 2019][1691251.729991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:31 2019][1691251.735910] CR2: 00007f74895f3f48 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:12:31 2019][1691251.743215] Call Trace: [Wed Feb 6 08:12:31 2019][1691251.745848] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:31 2019][1691251.752291] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:32 2019][1691251.757773] NMI watchdog: BUG: soft lockup - CPU#33 stuck for 21s! [ldlm_cn01_023:28088] [Wed Feb 6 08:12:32 2019][1691251.757822] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:32 2019][1691251.757844] CPU: 33 PID: 28088 Comm: ldlm_cn01_023 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:32 2019][1691251.757845] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:32 2019][1691251.757847] task: ffff915929b4c100 ti: ffff91578b654000 task.ti: ffff91578b654000 [Wed Feb 6 08:12:32 2019][1691251.757855] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:32 2019][1691251.757856] RSP: 0018:ffff91578b657c38 EFLAGS: 00000246 [Wed Feb 6 08:12:32 2019][1691251.757857] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001090000 [Wed Feb 6 08:12:32 2019][1691251.757858] RDX: ffff9149ff79b780 RSI: 0000000000d10001 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:32 2019][1691251.757859] RBP: ffff91578b657c38 R08: ffff9139ff81b780 R09: 0000000000000000 [Wed Feb 6 08:12:32 2019][1691251.757860] R10: ffff911b3fc07600 R11: ffffdbb2243ed400 R12: ffff91578b657be0 [Wed Feb 6 08:12:32 2019][1691251.757860] R13: ffff914fcb37fc80 R14: ffff91578b657ba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:32 2019][1691251.757862] FS: 00007f277c7c2700(0000) GS:ffff9139ff800000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:32 2019][1691251.757863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:32 2019][1691251.757864] CR2: 00007f2781328000 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:12:32 2019][1691251.757865] Call Trace: [Wed Feb 6 08:12:32 2019][1691251.757871] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:32 2019][1691251.757874] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:32 2019][1691251.757911] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.757944] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.757981] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.758019] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.758054] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.758096] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.758136] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.758139] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:12:32 2019][1691251.758178] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.758216] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.758219] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:32 2019][1691251.758221] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:32 2019][1691251.758224] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:32 2019][1691251.758225] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:32 2019][1691251.758245] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:12:32 2019][1691251.819774] NMI watchdog: BUG: soft lockup - CPU#40 stuck for 22s! [ldlm_cn00_015:14886] [Wed Feb 6 08:12:32 2019][1691251.819821] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:32 2019][1691251.819841] CPU: 40 PID: 14886 Comm: ldlm_cn00_015 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:32 2019][1691251.819842] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:32 2019][1691251.819844] task: ffff912fe0b65140 ti: ffff91593178c000 task.ti: ffff91593178c000 [Wed Feb 6 08:12:32 2019][1691251.819851] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:32 2019][1691251.819853] RSP: 0018:ffff91593178fc38 EFLAGS: 00000246 [Wed Feb 6 08:12:32 2019][1691251.819853] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001410000 [Wed Feb 6 08:12:32 2019][1691251.819854] RDX: ffff915a3f61b780 RSI: 0000000001190001 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:32 2019][1691251.819855] RBP: ffff91593178fc38 R08: ffff9129ff09b780 R09: 0000000000000000 [Wed Feb 6 08:12:32 2019][1691251.819856] R10: ffff911b3fc07600 R11: 000000000000015d R12: ffff91593178fbe0 [Wed Feb 6 08:12:32 2019][1691251.819856] R13: ffff914fcb37fc80 R14: ffff91593178fba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:32 2019][1691251.819858] FS: 00007f78f341b880(0000) GS:ffff9129ff080000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:32 2019][1691251.819859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:32 2019][1691251.819859] CR2: 00007f63fb343000 CR3: 0000004033a9a000 CR4: 00000000003407e0 [Wed Feb 6 08:12:32 2019][1691251.819860] Call Trace: [Wed Feb 6 08:12:32 2019][1691251.819866] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:32 2019][1691251.819870] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:32 2019][1691251.819913] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.819946] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.819982] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.820018] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.820052] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.820093] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.820131] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.820134] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:32 2019][1691251.820137] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:32 2019][1691251.820174] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.820212] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691251.820214] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:32 2019][1691251.820216] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:32 2019][1691251.820219] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:32 2019][1691251.820220] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:32 2019][1691251.820240] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:12:32 2019][1691252.446150] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691252.452882] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691252.459622] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691252.466701] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691252.473608] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691252.480692] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691252.488552] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691252.495511] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:32 2019][1691252.501950] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:32 2019][1691252.507990] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691252.514464] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:32 2019][1691252.522030] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:32 2019][1691252.527082] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:32 2019][1691252.533349] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:32 2019][1691252.539962] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:32 2019][1691252.546227] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:12:34 2019][1691254.503558] Lustre: fir-OST0020-osc-MDT0000: Connection to fir-OST0020 (at 10.0.10.105@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 08:12:34 2019][1691254.519800] Lustre: Skipped 19 previous similar messages [Wed Feb 6 08:12:35 2019][1691255.470302] Lustre: 23301:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 9s req@ffff9136171fd100 x1624709881794816/t0(0) o35->ca1af5b2-4b74-b03d-4a2b-13a823b2dc8f@10.8.15.10@o2ib6:0/0 lens 392/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 [Wed Feb 6 08:12:35 2019][1691255.483864] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_cn01_029:28330] [Wed Feb 6 08:12:35 2019][1691255.483911] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:35 2019][1691255.483931] CPU: 1 PID: 28330 Comm: ldlm_cn01_029 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:35 2019][1691255.483932] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:35 2019][1691255.483934] task: ffff9159aa3de180 ti: ffff915863708000 task.ti: ffff915863708000 [Wed Feb 6 08:12:35 2019][1691255.483941] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:35 2019][1691255.483943] RSP: 0018:ffff91586370bbb8 EFLAGS: 00000246 [Wed Feb 6 08:12:35 2019][1691255.483943] RAX: 0000000000000000 RBX: ffff91586370bbf0 RCX: 0000000000090000 [Wed Feb 6 08:12:35 2019][1691255.483944] RDX: ffff9139ff75b780 RSI: 0000000000a90000 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:35 2019][1691255.483945] RBP: ffff91586370bbb8 R08: ffff9139ff61b780 R09: 0000000000000000 [Wed Feb 6 08:12:35 2019][1691255.483946] R10: ffff911b3fc07600 R11: ffffdbb2243ed400 R12: 0000000000000000 [Wed Feb 6 08:12:35 2019][1691255.483946] R13: 0000000000000000 R14: 0000000000000002 R15: ffff91372d546e40 [Wed Feb 6 08:12:35 2019][1691255.483948] FS: 00007f63fb2d1700(0000) GS:ffff9139ff600000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:35 2019][1691255.483949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:35 2019][1691255.483949] CR2: 00007f63fb343000 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:12:35 2019][1691255.483951] Call Trace: [Wed Feb 6 08:12:35 2019][1691255.483956] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:35 2019][1691255.483959] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:35 2019][1691255.483995] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:35 2019][1691255.484028] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:12:35 2019][1691255.484030] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:35 2019][1691255.484062] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:12:35 2019][1691255.484097] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:35 2019][1691255.484132] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:36 2019][1691255.484167] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:36 2019][1691255.484206] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:36 2019][1691255.484243] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:36 2019][1691255.484247] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:12:36 2019][1691255.484284] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:36 2019][1691255.484322] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:36 2019][1691255.484324] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:36 2019][1691255.484326] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:36 2019][1691255.484329] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:36 2019][1691255.484330] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:36 2019][1691255.484349] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:12:36 2019][1691255.530673] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 7 seconds [Wed Feb 6 08:12:36 2019][1691255.530678] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.212@o2ib7 (12): c: 0, oc: 3, rc: 8 [Wed Feb 6 08:12:36 2019][1691255.537589] LustreError: 5873:0:(events.c:450:server_bulk_callback()) event type 5, status -110, desc ffff912e4a3b4e00 [Wed Feb 6 08:12:36 2019][1691255.784870] NMI watchdog: BUG: soft lockup - CPU#36 stuck for 22s! [ldlm_cn00_031:21243] [Wed Feb 6 08:12:36 2019][1691255.784920] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:36 2019][1691255.784942] CPU: 36 PID: 21243 Comm: ldlm_cn00_031 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:36 2019][1691255.784943] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:36 2019][1691255.784944] task: ffff9159aa3d9040 ti: ffff91591170c000 task.ti: ffff91591170c000 [Wed Feb 6 08:12:36 2019][1691255.784952] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:36 2019][1691255.784953] RSP: 0018:ffff91591170fc38 EFLAGS: 00000246 [Wed Feb 6 08:12:36 2019][1691255.784954] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001210000 [Wed Feb 6 08:12:36 2019][1691255.784955] RDX: ffff9129feedb780 RSI: 0000000000610000 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:36 2019][1691255.784956] RBP: ffff91591170fc38 R08: ffff9129ff05b780 R09: 0000000000000000 [Wed Feb 6 08:12:36 2019][1691255.784957] R10: ffff911b3fc07600 R11: ffffdbb27ef2a400 R12: ffff91591170fbe0 [Wed Feb 6 08:12:36 2019][1691255.784957] R13: ffff914fcb37fc80 R14: ffff91591170fba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:36 2019][1691255.784959] FS: 00007fb54d9fb740(0000) GS:ffff9129ff040000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:36 2019][1691255.784960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:36 2019][1691255.784961] CR2: 00007fb54d41a2e0 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:12:36 2019][1691255.784962] Call Trace: [Wed Feb 6 08:12:36 2019][1691255.784968] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:36 2019][1691255.784971] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:36 2019][1691255.785016] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:36 2019][1691255.785050] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:36 2019][1691255.785087] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:36 2019][1691255.785123] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:36 2019][1691255.785158] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:36 2019][1691255.785200] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:36 2019][1691255.785239] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:36 2019][1691255.785243] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:36 2019][1691255.785246] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:36 2019][1691255.785285] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:36 2019][1691255.785323] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:36 2019][1691255.785326] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:36 2019][1691255.785327] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:36 2019][1691255.785330] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:36 2019][1691255.785332] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:36 2019][1691255.785352] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:12:38 2019][1691257.951170] LustreError: 20150:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(17000) req@ffff913e00091c50 x1624669521058592/t0(0) o4->64c2e10f-595e-bb8e-efbd-0544992f523c@10.9.101.49@o2ib4:485/0 lens 488/448 e 1 to 0 dl 1549469620 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Feb 6 08:12:38 2019][1691257.976193] Lustre: fir-MDT0002: Bulk IO write error with 64c2e10f-595e-bb8e-efbd-0544992f523c (at 10.9.101.49@o2ib4), client will retry: rc = -110 [Wed Feb 6 08:12:38 2019][1691257.989577] Lustre: Skipped 4 previous similar messages [Wed Feb 6 08:12:38 2019][1691258.121949] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Wed Feb 6 08:12:38 2019][1691258.132206] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 12 previous similar messages [Wed Feb 6 08:12:38 2019][1691258.142562] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (11): c: 3, oc: 0, rc: 8 [Wed Feb 6 08:12:38 2019][1691258.154646] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 12 previous similar messages [Wed Feb 6 08:12:39 2019][1691259.610966] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 23s! [ldlm_cn00_023:20040] [Wed Feb 6 08:12:39 2019][1691259.619228] Modules linked in:[1691259.619967] NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [ldlm_cn01_018:23321] [Wed Feb 6 08:12:39 2019][1691259.619968] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:39 2019][1691259.620017] CPU: 17 PID: 23321 Comm: ldlm_cn01_018 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:39 2019][1691259.620018] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:39 2019][1691259.620020] task: ffff9159e661a080 ti: ffff9159163a0000 task.ti: ffff9159163a0000 [Wed Feb 6 08:12:39 2019][1691259.620021] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:40 2019][1691259.620025] RSP: 0018:ffff9159163a3bb8 EFLAGS: 00000246 [Wed Feb 6 08:12:40 2019][1691259.620026] RAX: 0000000000000000 RBX: ffff9159163a3bf0 RCX: 0000000000890000 [Wed Feb 6 08:12:40 2019][1691259.620027] RDX: ffff9139ff85b780 RSI: 0000000001290000 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:40 2019][1691259.620028] RBP: ffff9159163a3bb8 R08: ffff9139ff71b780 R09: 0000000000000000 [Wed Feb 6 08:12:40 2019][1691259.620028] R10: ffff911b3fc07600 R11: ffffdbb26de05800 R12: 0000000000000000 [Wed Feb 6 08:12:40 2019][1691259.620029] R13: 0000000000000000 R14: 0000000000000002 R15: ffff91214b5d0900 [Wed Feb 6 08:12:40 2019][1691259.620030] FS: 00007f0ff898a880(0000) GS:ffff9139ff700000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:40 2019][1691259.620032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:40 2019][1691259.620032] CR2: 00007f85b8df4000 CR3: 000000202de1c000 CR4: 00000000003407e0 [Wed Feb 6 08:12:40 2019][1691259.620033] Call Trace: [Wed Feb 6 08:12:40 2019][1691259.620036] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:40 2019][1691259.620039] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:40 2019][1691259.620072] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691259.620105] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691259.620108] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:40 2019][1691259.620140] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691259.620176] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691259.620212] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691259.620247] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691259.620287] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691259.620327] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691259.620329] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:40 2019][1691259.620332] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:40 2019][1691259.620371] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691259.620410] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691259.620411] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:40 2019][1691259.620413] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:40 2019][1691259.620415] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:40 2019][1691259.620417] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:40 2019][1691259.620417] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:12:40 2019] [Wed Feb 6 08:12:40 2019][1691259.988913] osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:40 2019][1691260.094297] CPU: 16 PID: 20040 Comm: ldlm_cn00_023 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:40 2019][1691260.107061] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:40 2019][1691260.114799] task: ffff9149ca785140 ti: ffff915909ff0000 task.ti: ffff915909ff0000 [Wed Feb 6 08:12:40 2019][1691260.122454] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:40 2019][1691260.132557] RSP: 0018:ffff915909ff3bb8 EFLAGS: 00000246 [Wed Feb 6 08:12:40 2019][1691260.138044] RAX: 0000000000000000 RBX: ffff915909ff3bf0 RCX: 0000000000810000 [Wed Feb 6 08:12:40 2019][1691260.145351] RDX: ffff9149ff6db780 RSI: 0000000000710000 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:40 2019][1691260.152657] RBP: ffff915909ff3bb8 R08: ffff9129fef1b780 R09: 0000000000000000 [Wed Feb 6 08:12:40 2019][1691260.159962] R10: ffff911b3fc07600 R11: ffff9127df78b660 R12: 0000000000000000 [Wed Feb 6 08:12:40 2019][1691260.167268] R13: 0000000000000000 R14: 0000000000000002 R15: ffff912bd05b3600 [Wed Feb 6 08:12:40 2019][1691260.174577] FS: 00007f78f341b880(0000) GS:ffff9129fef00000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:40 2019][1691260.182835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:40 2019][1691260.188753] CR2: 00007f63fb343000 CR3: 0000004033a9a000 CR4: 00000000003407e0 [Wed Feb 6 08:12:40 2019][1691260.196060] Call Trace: [Wed Feb 6 08:12:40 2019][1691260.198689] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:40 2019][1691260.205128] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:40 2019][1691260.210827] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691260.217562] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691260.224610] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:40 2019][1691260.232205] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691260.238936] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691260.246018] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691260.252924] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691260.260009] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691260.267877] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691260.274837] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:40 2019][1691260.281274] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:40 2019][1691260.287315] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691260.293789] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:40 2019][1691260.301357] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:40 2019][1691260.306408] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:40 2019][1691260.312675] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:40 2019][1691260.319287] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:40 2019][1691260.325551] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:12:43 2019][1691263.122127] LustreError: 5873:0:(events.c:450:server_bulk_callback()) event type 5, status -110, desc ffff914ee02c5200 [Wed Feb 6 08:12:43 2019][1691263.646068] NMI watchdog: BUG: soft lockup - CPU#20 stuck for 21s! [ldlm_cn00_010:11333] [Wed Feb 6 08:12:43 2019][1691263.654334] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:44 2019][1691263.761173] CPU: 20 PID: 11333 Comm: ldlm_cn00_010 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:44 2019][1691263.773937] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:44 2019][1691263.781676] task: ffff9149d7ab9040 ti: ffff915974bf8000 task.ti: ffff915974bf8000 [Wed Feb 6 08:12:44 2019][1691263.789329] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:44 2019][1691263.799443] RSP: 0018:ffff915974bfbbb8 EFLAGS: 00000246 [Wed Feb 6 08:12:44 2019][1691263.804930] RAX: 0000000000000000 RBX: ffff915974bfbbf0 RCX: 0000000000a10000 [Wed Feb 6 08:12:44 2019][1691263.812234] RDX: ffff9129ff09b780 RSI: 0000000001410000 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:44 2019][1691263.819540] RBP: ffff915974bfbbb8 R08: ffff9129fef5b780 R09: 0000000000000000 [Wed Feb 6 08:12:44 2019][1691263.826847] R10: ffff911b3fc07600 R11: ffff911a1a092508 R12: 0000000000000000 [Wed Feb 6 08:12:44 2019][1691263.834155] R13: 0000000000000000 R14: 0000000000000002 R15: ffff912f7c2ab600 [Wed Feb 6 08:12:44 2019][1691263.841461] FS: 00007f277c7c2700(0000) GS:ffff9129fef40000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:44 2019][1691263.849814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:44 2019][1691263.855735] CR2: 00007f27740dc0e8 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:12:44 2019][1691263.863041] Call Trace: [Wed Feb 6 08:12:44 2019][1691263.865672] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:44 2019][1691263.872119] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:44 2019][1691263.877816] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:44 2019][1691263.884549] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:12:44 2019][1691263.891600] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:44 2019][1691263.899195] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:12:44 2019][1691263.905928] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:44 2019][1691263.913006] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:44 2019][1691263.919912] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:44 2019][1691263.927001] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:44 2019][1691263.934865] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:44 2019][1691263.941825] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:44 2019][1691263.948262] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:44 2019][1691263.954307] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:44 2019][1691263.960780] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:44 2019][1691263.968345] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:44 2019][1691263.973398] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:44 2019][1691263.979673] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:44 2019][1691263.986293] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:44 2019][1691263.992583] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:12:44 2019][1691264.024781] LustreError: 28956:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff914af1fb0e00 [Wed Feb 6 08:12:44 2019][1691264.035788] LustreError: 28995:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff913ac5deec00 [Wed Feb 6 08:12:44 2019][1691264.046750] LustreError: 28965:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff9159a5e65400 [Wed Feb 6 08:12:44 2019][1691264.057871] LustreError: 28988:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff913ac5ded000 [Wed Feb 6 08:12:44 2019][1691264.068915] LustreError: 28980:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff915972f00e00 [Wed Feb 6 08:12:44 2019][1691264.080008] LustreError: 28978:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff913ac5dece00 [Wed Feb 6 08:12:44 2019][1691264.090966] LustreError: 28983:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff915972f06e00 [Wed Feb 6 08:12:44 2019][1691264.101941] LustreError: 28979:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff9159a479e200 [Wed Feb 6 08:12:44 2019][1691264.112915] LustreError: 28987:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff9151d0a88200 [Wed Feb 6 08:12:44 2019][1691264.123906] LustreError: 28985:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff914af1fb1400 [Wed Feb 6 08:12:44 2019][1691264.134862] LustreError: 28991:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff915972f03800 [Wed Feb 6 08:12:44 2019][1691264.145905] LustreError: 28957:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff915972f01800 [Wed Feb 6 08:12:44 2019][1691264.156876] LustreError: 28981:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff915296bd4600 [Wed Feb 6 08:12:44 2019][1691264.167831] LustreError: 28989:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff914af1fb4400 [Wed Feb 6 08:12:44 2019][1691264.178846] LustreError: 28958:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff9159a5e65800 [Wed Feb 6 08:12:44 2019][1691264.189939] LustreError: 28982:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff915999fb2800 [Wed Feb 6 08:12:44 2019][1691264.200907] LustreError: 28993:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff914ad8e9e200 [Wed Feb 6 08:12:44 2019][1691264.211968] LustreError: 28992:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff9159a4798600 [Wed Feb 6 08:12:47 2019][1691266.980350] LustreError: 5881:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff912059437800 [Wed Feb 6 08:12:47 2019][1691266.991092] LustreError: 28990:0:(events.c:450:server_bulk_callback()) event type 5, status -113, desc ffff9137a7a5c000 [Wed Feb 6 08:12:47 2019][1691267.002076] LustreError: 5881:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff915999fb1a00 [Wed Feb 6 08:12:47 2019][1691267.100638] LustreError: 9036:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(19537) req@ffff91494a410050 x1624705111824720/t0(0) o4->70c3879e-e355-8a86-c7f6-86725077b527@10.8.3.28@o2ib6:503/0 lens 488/448 e 0 to 0 dl 1549469638 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:12:47 2019][1691267.125372] LustreError: 9036:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message [Wed Feb 6 08:12:47 2019][1691267.476164] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ldlm_cn00_029:21239] [Wed Feb 6 08:12:47 2019][1691267.484339] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver[1691267.510164] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [ldlm_cn00_018:14898] [Wed Feb 6 08:12:47 2019][1691267.510166] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:47 2019][1691267.510240] CPU: 4 PID: 14898 Comm: ldlm_cn00_018 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:47 2019][1691267.510241] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:47 2019][1691267.510242] task: ffff9159f6d1a080 ti: ffff91597a7a0000 task.ti: ffff91597a7a0000 [Wed Feb 6 08:12:47 2019][1691267.510244] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:47 2019][1691267.510251] RSP: 0018:ffff91597a7a3bb8 EFLAGS: 00000246 [Wed Feb 6 08:12:47 2019][1691267.510252] RAX: 0000000000000000 RBX: ffff91597a7a3bf0 RCX: 0000000000210000 [Wed Feb 6 08:12:47 2019][1691267.510253] RDX: ffff9149ff69b780 RSI: 0000000000510000 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:47 2019][1691267.510254] RBP: ffff91597a7a3bb8 R08: ffff9129fee5b780 R09: 0000000000000000 [Wed Feb 6 08:12:47 2019][1691267.510254] R10: ffff911b3fc07600 R11: ffffdbb20811cc00 R12: 0000000000000000 [Wed Feb 6 08:12:47 2019][1691267.510255] R13: 0000000000000000 R14: 0000000000000002 R15: ffff911c53fe3cc0 [Wed Feb 6 08:12:47 2019][1691267.510257] FS: 00007fe2d3435880(0000) GS:ffff9129fee40000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:47 2019][1691267.510258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:47 2019][1691267.510258] CR2: 00007fe2d3446000 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:12:47 2019][1691267.510260] Call Trace: [Wed Feb 6 08:12:47 2019][1691267.510265] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:47 2019][1691267.510269] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:47 2019][1691267.510312] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:47 2019][1691267.510346] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:12:47 2019][1691267.510348] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:48 2019][1691267.510381] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.510419] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.510455] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.510491] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.510535] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.510575] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.510578] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:48 2019][1691267.510580] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:48 2019][1691267.510621] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.510662] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.510665] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:48 2019][1691267.510667] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:48 2019][1691267.510669] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:48 2019][1691267.510670] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:48 2019][1691267.510671] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:12:48 2019][1691267.543168] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 21s! [ldlm_cn00_020:19871] [Wed Feb 6 08:12:48 2019][1691267.543169] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:48 2019][1691267.543240] CPU: 8 PID: 19871 Comm: ldlm_cn00_020 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:48 2019][1691267.543240] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:48 2019][1691267.543242] task: ffff914321218000 ti: ffff91592b768000 task.ti: ffff91592b768000 [Wed Feb 6 08:12:48 2019][1691267.543243] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:48 2019][1691267.543250] RSP: 0018:ffff91592b76bbb8 EFLAGS: 00000246 [Wed Feb 6 08:12:48 2019][1691267.543251] RAX: 0000000000000000 RBX: ffff91592b76bbf0 RCX: 0000000000410000 [Wed Feb 6 08:12:48 2019][1691267.543252] RDX: ffff9129fee5b780 RSI: 0000000000210000 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:48 2019][1691267.543253] RBP: ffff91592b76bbb8 R08: ffff9129fee9b780 R09: 0000000000000000 [Wed Feb 6 08:12:48 2019][1691267.543254] R10: ffff911b3fc07600 R11: ffffdbb2093e3c00 R12: 0000000000000000 [Wed Feb 6 08:12:48 2019][1691267.543254] R13: 0000000000000000 R14: 0000000000000002 R15: ffff9140bab38d80 [Wed Feb 6 08:12:48 2019][1691267.543256] FS: 00007fe32e264880(0000) GS:ffff9129fee80000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:48 2019][1691267.543257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:48 2019][1691267.543258] CR2: 00007f63fb343000 CR3: 000000103b274000 CR4: 00000000003407e0 [Wed Feb 6 08:12:48 2019][1691267.543259] Call Trace: [Wed Feb 6 08:12:48 2019][1691267.543264] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:48 2019][1691267.543268] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:48 2019][1691267.543311] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.543345] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.543347] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:48 2019][1691267.543380] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.543418] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.543455] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.543490] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.543534] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.543574] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.543577] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:48 2019][1691267.543580] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:48 2019][1691267.543619] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.543658] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.543661] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:48 2019][1691267.543663] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:48 2019][1691267.543666] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:48 2019][1691267.543667] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:48 2019][1691267.543668] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:12:48 2019][1691267.777172] NMI watchdog: BUG: soft lockup - CPU#35 stuck for 22s! [ldlm_cn03_025:28518] [Wed Feb 6 08:12:48 2019][1691267.777173] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:48 2019][1691267.777245] CPU: 35 PID: 28518 Comm: ldlm_cn03_025 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:48 2019][1691267.777246] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:48 2019][1691267.777246] task: ffff9149d5fc9040 ti: ffff913ac391c000 task.ti: ffff913ac391c000 [Wed Feb 6 08:12:48 2019][1691267.777247] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:48 2019][1691267.777254] RSP: 0018:ffff913ac391fc38 EFLAGS: 00000246 [Wed Feb 6 08:12:48 2019][1691267.777255] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001190000 [Wed Feb 6 08:12:48 2019][1691267.777255] RDX: ffff9129feedb780 RSI: 0000000000610001 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:48 2019][1691267.777256] RBP: ffff913ac391fc38 R08: ffff915a3f61b780 R09: 0000000000000000 [Wed Feb 6 08:12:48 2019][1691267.777257] R10: ffff911b3fc07600 R11: 0000000000003c00 R12: ffff913ac391fbe0 [Wed Feb 6 08:12:48 2019][1691267.777258] R13: ffff914fcb37fc80 R14: ffff913ac391fba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:48 2019][1691267.777259] FS: 00007f2b3eadb740(0000) GS:ffff915a3f600000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:48 2019][1691267.777260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:48 2019][1691267.777261] CR2: 00007f2b3d82280d CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:12:48 2019][1691267.777262] Call Trace: [Wed Feb 6 08:12:48 2019][1691267.777266] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:48 2019][1691267.777269] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:48 2019][1691267.777309] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.777337] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.777370] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.777401] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.777431] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.777469] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.777504] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.777507] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:48 2019][1691267.777509] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:48 2019][1691267.777543] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.777577] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:48 2019][1691267.777579] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:48 2019][1691267.777581] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:48 2019][1691267.777583] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:48 2019][1691267.777585] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:48 2019][1691267.777585] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:12:48 2019][1691267.837172] NMI watchdog: BUG: soft lockup - CPU#42 stuck for 21s! [ldlm_cn02_020:23325] [Wed Feb 6 08:12:48 2019][1691267.837173] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:48 2019][1691267.837251] CPU: 42 PID: 23325 Comm: ldlm_cn02_020 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:48 2019][1691267.837251] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:48 2019][1691267.837253] task: ffff91599ff99040 ti: ffff91580caf8000 task.ti: ffff91580caf8000 [Wed Feb 6 08:12:48 2019][1691267.837254] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:49 2019][1691267.837262] RSP: 0018:ffff91580cafbc38 EFLAGS: 00000246 [Wed Feb 6 08:12:49 2019][1691267.837263] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001510000 [Wed Feb 6 08:12:49 2019][1691267.837264] RDX: ffff9139ff75b780 RSI: 0000000000a90000 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:49 2019][1691267.837265] RBP: ffff91580cafbc38 R08: ffff9149ff89b780 R09: 0000000000000000 [Wed Feb 6 08:12:49 2019][1691267.837266] R10: ffff911b3fc07600 R11: ffffdbb2843e0a00 R12: ffff91580cafbbe0 [Wed Feb 6 08:12:49 2019][1691267.837267] R13: ffff914fcb37fc80 R14: ffff91580cafbba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:49 2019][1691267.837268] FS: 00007f72eaa7c700(0000) GS:ffff9149ff880000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:49 2019][1691267.837269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:49 2019][1691267.837270] CR2: 00007f78e136d464 CR3: 000000403bd1e000 CR4: 00000000003407e0 [Wed Feb 6 08:12:49 2019][1691267.837271] Call Trace: [Wed Feb 6 08:12:49 2019][1691267.837277] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:49 2019][1691267.837281] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:49 2019][1691267.837324] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691267.837359] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691267.837397] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691267.837435] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691267.837471] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691267.837514] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691267.837555] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691267.837559] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:49 2019][1691267.837561] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:49 2019][1691267.837601] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691267.837641] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691267.837644] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:49 2019][1691267.837646] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:49 2019][1691267.837649] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:49 2019][1691267.837650] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:49 2019][1691267.837651] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:12:49 2019] [Wed Feb 6 08:12:49 2019][1691268.947582] nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:49 2019][1691269.029756] CPU: 0 PID: 21239 Comm: ldlm_cn00_029 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:49 2019][1691269.042433] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:49 2019][1691269.050174] task: ffff9159f828b0c0 ti: ffff9159537d8000 task.ti: ffff9159537d8000 [Wed Feb 6 08:12:49 2019][1691269.057827] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:49 2019][1691269.067931] RSP: 0018:ffff9159537dbc38 EFLAGS: 00000246 [Wed Feb 6 08:12:49 2019][1691269.073418] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000010000 [Wed Feb 6 08:12:49 2019][1691269.080723] RDX: ffff9149ff79b780 RSI: 0000000000d10001 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:49 2019][1691269.088029] RBP: ffff9159537dbc38 R08: ffff9129fee1b780 R09: 0000000000000000 [Wed Feb 6 08:12:49 2019][1691269.095335] R10: ffff911b3fc07600 R11: 0000000000000000 R12: ffff9159537dbbe0 [Wed Feb 6 08:12:49 2019][1691269.102641] R13: ffff914fcb37fc80 R14: ffff9159537dbba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:49 2019][1691269.109949] FS: 00007fe32e264880(0000) GS:ffff9129fee00000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:49 2019][1691269.118207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:49 2019][1691269.124127] CR2: 00007fe2d3446000 CR3: 000000103b274000 CR4: 00000000003407f0 [Wed Feb 6 08:12:49 2019][1691269.131433] Call Trace: [Wed Feb 6 08:12:49 2019][1691269.134062] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:49 2019][1691269.140500] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:49 2019][1691269.146191] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691269.152926] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691269.159664] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691269.166742] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691269.173650] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691269.180737] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691269.188605] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691269.195571] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:12:49 2019][1691269.201354] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691269.207827] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:49 2019][1691269.215390] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:49 2019][1691269.220444] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:49 2019][1691269.226710] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:49 2019][1691269.233323] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:49 2019][1691269.239588] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:12:51 2019][1691271.519268] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [ldlm_cn01_021:27821] [Wed Feb 6 08:12:51 2019][1691271.527443] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:51 2019][1691271.634214] CPU: 5 PID: 27821 Comm: ldlm_cn01_021 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:51 2019][1691271.646890] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:51 2019][1691271.654629] task: ffff9139f6230000 ti: ffff915751354000 task.ti: ffff915751354000 [Wed Feb 6 08:12:51 2019][1691271.662282] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:51 2019][1691271.672388] RSP: 0018:ffff915751357c10 EFLAGS: 00000246 [Wed Feb 6 08:12:51 2019][1691271.677874] RAX: 0000000000000000 RBX: ffff915751357ba0 RCX: 0000000000290000 [Wed Feb 6 08:12:51 2019][1691271.685180] RDX: ffff9149ff81b780 RSI: 0000000001110001 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:51 2019][1691271.692487] RBP: ffff915751357c10 R08: ffff9139ff65b780 R09: 0000000000000000 [Wed Feb 6 08:12:51 2019][1691271.699791] R10: ffff911b3fc07600 R11: ffffdbb25bf3a800 R12: ffff915751357bb0 [Wed Feb 6 08:12:51 2019][1691271.707099] R13: ffff912b6ac432dc R14: 0000000000510000 R15: ffff9149ff69b780 [Wed Feb 6 08:12:51 2019][1691271.714404] FS: 00007f35005b3880(0000) GS:ffff9139ff640000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:51 2019][1691271.722663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:51 2019][1691271.728583] CR2: 00007fbe3d5a4248 CR3: 0000001035820000 CR4: 00000000003407e0 [Wed Feb 6 08:12:51 2019][1691271.735889] Call Trace: [Wed Feb 6 08:12:51 2019][1691271.738520] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:51 2019][1691271.744959] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:51 2019][1691271.750654] [] __ldlm_reprocess_all+0xfa/0x380 [ptlrpc] [Wed Feb 6 08:12:51 2019][1691271.757728] [] ldlm_reprocess_all+0x10/0x20 [ptlrpc] [Wed Feb 6 08:12:51 2019][1691271.764551] [] ldlm_request_cancel+0x3ed/0x740 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.771632] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.778540] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.785626] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.793494] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.800453] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:12:52 2019][1691271.806232] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.811273] NMI watchdog: BUG: soft lockup - CPU#39 stuck for 22s! [ldlm_cn03_017:63248] [Wed Feb 6 08:12:52 2019][1691271.811322] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:52 2019][1691271.811342] CPU: 39 PID: 63248 Comm: ldlm_cn03_017 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:52 2019][1691271.811342] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:52 2019][1691271.811343] task: ffff9149d62d6180 ti: ffff9141a1abc000 task.ti: ffff9141a1abc000 [Wed Feb 6 08:12:52 2019][1691271.811349] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:52 2019][1691271.811350] RSP: 0018:ffff9141a1abfc38 EFLAGS: 00000246 [Wed Feb 6 08:12:52 2019][1691271.811351] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001390000 [Wed Feb 6 08:12:52 2019][1691271.811352] RDX: ffff9149ff81b780 RSI: 0000000001110000 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:52 2019][1691271.811352] RBP: ffff9141a1abfc38 R08: ffff915a3f65b780 R09: 0000000000000000 [Wed Feb 6 08:12:52 2019][1691271.811353] R10: ffff911b3fc07600 R11: ffffdbb2fecbc000 R12: ffff9141a1abfbe0 [Wed Feb 6 08:12:52 2019][1691271.811353] R13: ffff914fcb37fc80 R14: ffff9141a1abfba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:52 2019][1691271.811355] FS: 00007f4ff05b7880(0000) GS:ffff915a3f640000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:52 2019][1691271.811356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:52 2019][1691271.811356] CR2: 00007f53f6ea3248 CR3: 0000003f8eb5e000 CR4: 00000000003407e0 [Wed Feb 6 08:12:52 2019][1691271.811357] Call Trace: [Wed Feb 6 08:12:52 2019][1691271.811361] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:52 2019][1691271.811364] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:52 2019][1691271.811404] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.811432] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.811463] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.811493] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.811522] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.811557] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.811590] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.811593] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:12:52 2019][1691271.811625] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.811657] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.811659] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:52 2019][1691271.811661] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:52 2019][1691271.811663] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:52 2019][1691271.811665] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:52 2019][1691271.811682] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:12:52 2019][1691271.828275] NMI watchdog: BUG: soft lockup - CPU#41 stuck for 22s! [ldlm_cn01_013:19974] [Wed Feb 6 08:12:52 2019][1691271.828321] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:52 2019][1691271.828344] CPU: 41 PID: 19974 Comm: ldlm_cn01_013 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:52 2019][1691271.828345] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:52 2019][1691271.828347] task: ffff9129ee3ca080 ti: ffff91590ceb0000 task.ti: ffff91590ceb0000 [Wed Feb 6 08:12:52 2019][1691271.828355] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:52 2019][1691271.828356] RSP: 0018:ffff91590ceb3bb8 EFLAGS: 00000246 [Wed Feb 6 08:12:52 2019][1691271.828356] RAX: 0000000000000000 RBX: ffff91590ceb3bf0 RCX: 0000000001490000 [Wed Feb 6 08:12:52 2019][1691271.828357] RDX: ffff9139ff61b780 RSI: 0000000000090001 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:52 2019][1691271.828358] RBP: ffff91590ceb3bb8 R08: ffff9139ff89b780 R09: 0000000000000000 [Wed Feb 6 08:12:52 2019][1691271.828359] R10: ffff911b3fc07600 R11: ffff912fe4374c00 R12: 0000000000000000 [Wed Feb 6 08:12:52 2019][1691271.828359] R13: 0000000000000000 R14: 0000000000000002 R15: ffff913af0d1dc40 [Wed Feb 6 08:12:52 2019][1691271.828361] FS: 00007fc9e2d79880(0000) GS:ffff9139ff880000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:52 2019][1691271.828362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:52 2019][1691271.828363] CR2: 00007f85b8df4000 CR3: 000000303a262000 CR4: 00000000003407e0 [Wed Feb 6 08:12:52 2019][1691271.828363] Call Trace: [Wed Feb 6 08:12:52 2019][1691271.828370] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:52 2019][1691271.828374] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:52 2019][1691271.828410] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.828443] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.828446] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:52 2019][1691271.828478] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.828514] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.828549] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.828584] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.828623] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.828661] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.828665] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:52 2019][1691271.828667] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:52 2019][1691271.828705] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.828743] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.828745] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:52 2019][1691271.828747] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:52 2019][1691271.828750] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:52 2019][1691271.828752] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:52 2019][1691271.828771] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:12:52 2019][1691271.855277] NMI watchdog: BUG: soft lockup - CPU#44 stuck for 24s! [ldlm_cn00_014:14885] [Wed Feb 6 08:12:52 2019][1691271.855325] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:52 2019][1691271.855345] CPU: 44 PID: 14885 Comm: ldlm_cn00_014 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:52 2019][1691271.855346] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:52 2019][1691271.855347] task: ffff912fe0b61040 ti: ffff9159fb438000 task.ti: ffff9159fb438000 [Wed Feb 6 08:12:52 2019][1691271.855353] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x128/0x200 [Wed Feb 6 08:12:52 2019][1691271.855354] RSP: 0018:ffff9159fb43bc38 EFLAGS: 00000246 [Wed Feb 6 08:12:52 2019][1691271.855355] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001610000 [Wed Feb 6 08:12:52 2019][1691271.855356] RDX: ffff915a3f69b780 RSI: 0000000001590001 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:52 2019][1691271.855357] RBP: ffff9159fb43bc38 R08: ffff9129ff0db780 R09: 0000000000000000 [Wed Feb 6 08:12:52 2019][1691271.855357] R10: ffff911b3fc07600 R11: ffffdbb262b28400 R12: ffff9159fb43bbe0 [Wed Feb 6 08:12:52 2019][1691271.855358] R13: ffff914fcb37fc80 R14: ffff9159fb43bba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:12:52 2019][1691271.855359] FS: 00007f2781319740(0000) GS:ffff9129ff0c0000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:52 2019][1691271.855360] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:52 2019][1691271.855361] CR2: 00007f277416c0e8 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:12:52 2019][1691271.855362] Call Trace: [Wed Feb 6 08:12:52 2019][1691271.855367] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:52 2019][1691271.855370] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:52 2019][1691271.855413] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.855446] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:12:52 2019][1691271.855483] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.855519] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.855554] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.855595] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.855633] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.855636] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:12:53 2019][1691271.855638] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:12:53 2019][1691271.855676] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.855713] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.855716] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:53 2019][1691271.855718] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:53 2019][1691271.855720] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:53 2019][1691271.855722] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:53 2019][1691271.855741] Code: 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 85 c0 <74> f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 [Wed Feb 6 08:12:53 2019][1691271.872278] NMI watchdog: BUG: soft lockup - CPU#46 stuck for 22s! [ldlm_cn02_017:23320] [Wed Feb 6 08:12:53 2019][1691271.872326] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:53 2019][1691271.872346] CPU: 46 PID: 23320 Comm: ldlm_cn02_017 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:53 2019][1691271.872346] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:53 2019][1691271.872348] task: ffff9159e661c100 ti: ffff91580b2b0000 task.ti: ffff91580b2b0000 [Wed Feb 6 08:12:53 2019][1691271.872356] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:53 2019][1691271.872357] RSP: 0018:ffff91580b2b3bb8 EFLAGS: 00000246 [Wed Feb 6 08:12:53 2019][1691271.872358] RAX: 0000000000000000 RBX: ffff91580b2b3bf0 RCX: 0000000001710000 [Wed Feb 6 08:12:53 2019][1691271.872358] RDX: ffff9139ff6db780 RSI: 0000000000690000 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:53 2019][1691271.872359] RBP: ffff91580b2b3bb8 R08: ffff9149ff8db780 R09: 0000000000000000 [Wed Feb 6 08:12:53 2019][1691271.872360] R10: ffff911b3fc07600 R11: 0000000000000001 R12: 0000000000000000 [Wed Feb 6 08:12:53 2019][1691271.872360] R13: 0000000000000000 R14: 0000000000000002 R15: ffff915652e0b3c0 [Wed Feb 6 08:12:53 2019][1691271.872362] FS: 00007f277afbf700(0000) GS:ffff9149ff8c0000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:53 2019][1691271.872363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:53 2019][1691271.872363] CR2: 00007f275c005088 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:12:53 2019][1691271.872364] Call Trace: [Wed Feb 6 08:12:53 2019][1691271.872370] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:53 2019][1691271.872374] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:53 2019][1691271.872416] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.872449] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.872452] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:53 2019][1691271.872484] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.872521] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.872556] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.872591] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.872633] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.872671] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.872674] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:12:53 2019][1691271.872712] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.872750] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691271.872753] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:53 2019][1691271.872754] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:53 2019][1691271.872757] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:53 2019][1691271.872759] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:53 2019][1691271.872779] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:12:53 2019][1691273.218188] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:53 2019][1691273.225753] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:53 2019][1691273.230805] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:53 2019][1691273.237071] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:53 2019][1691273.243685] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:53 2019][1691273.249948] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:12:55 2019][1691275.551369] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [ldlm_cn01_005:6768] [Wed Feb 6 08:12:55 2019][1691275.559453] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:55 2019][1691275.666223] CPU: 9 PID: 6768 Comm: ldlm_cn01_005 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:55 2019][1691275.678814] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:55 2019][1691275.686553] task: ffff9159f33cd140 ti: ffff9159f2724000 task.ti: ffff9159f2724000 [Wed Feb 6 08:12:55 2019][1691275.694207] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:55 2019][1691275.704313] RSP: 0018:ffff9159f2727bb8 EFLAGS: 00000246 [Wed Feb 6 08:12:55 2019][1691275.709798] RAX: 0000000000000000 RBX: ffff9159f2727bf0 RCX: 0000000000490000 [Wed Feb 6 08:12:55 2019][1691275.717104] RDX: ffff9129fef9b780 RSI: 0000000000c10000 RDI: ffff912b6ac432dc [Wed Feb 6 08:12:55 2019][1691275.724411] RBP: ffff9159f2727bb8 R08: ffff9139ff69b780 R09: 0000000000000000 [Wed Feb 6 08:12:55 2019][1691275.731717] R10: ffff911b3fc07600 R11: ffffdbb25228ec00 R12: 0000000000000000 [Wed Feb 6 08:12:55 2019][1691275.739024] R13: 0000000000000000 R14: 0000000000000002 R15: ffff913b6c182f40 [Wed Feb 6 08:12:55 2019][1691275.746329] FS: 00007f277c7c2700(0000) GS:ffff9139ff680000(0000) knlGS:0000000000000000 [Wed Feb 6 08:12:55 2019][1691275.754589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:12:55 2019][1691275.760508] CR2: 00007f2781328000 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:12:56 2019][1691275.767813] Call Trace: [Wed Feb 6 08:12:56 2019][1691275.770442] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:12:56 2019][1691275.776881] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:12:56 2019][1691275.782573] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:12:56 2019][1691275.789306] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:12:56 2019][1691275.796355] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:12:56 2019][1691275.803953] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:12:56 2019][1691275.810690] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:12:56 2019][1691275.817770] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:12:56 2019][1691275.824677] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:12:56 2019][1691275.831762] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:12:56 2019][1691275.839623] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:12:56 2019][1691275.846581] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:12:56 2019][1691275.852363] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:12:56 2019][1691275.858836] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:12:56 2019][1691275.866402] [] kthread+0xd1/0xe0 [Wed Feb 6 08:12:56 2019][1691275.871453] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:56 2019][1691275.877721] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:12:56 2019][1691275.884333] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:12:56 2019][1691275.890599] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:12:59 2019][1691279.527469] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [ldlm_cn02_006:6776] [Wed Feb 6 08:12:59 2019][1691279.535557] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:12:59 2019][1691279.642326] CPU: 6 PID: 6776 Comm: ldlm_cn02_006 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:12:59 2019][1691279.654915] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:12:59 2019][1691279.662656] task: ffff9159a2766180 ti: ffff9159f3e34000 task.ti: ffff9159f3e34000 [Wed Feb 6 08:12:59 2019][1691279.670309] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:12:59 2019][1691279.680415] RSP: 0018:ffff9159f3e37c38 EFLAGS: 00000246 [Wed Feb 6 08:12:59 2019][1691279.680471] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 21s! [ldlm_cn00_006:6777] [Wed Feb 6 08:12:59 2019][1691279.680519] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:13:00 2019][1691279.680541] CPU: 24 PID: 6777 Comm: ldlm_cn00_006 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:13:00 2019][1691279.680542] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:13:00 2019][1691279.680543] task: ffff9159a2765140 ti: ffff9159ee60c000 task.ti: ffff9159ee60c000 [Wed Feb 6 08:13:00 2019][1691279.680551] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:13:00 2019][1691279.680552] RSP: 0018:ffff9159ee60fbb8 EFLAGS: 00000246 [Wed Feb 6 08:13:00 2019][1691279.680553] RAX: 0000000000000000 RBX: ffff9159ee60fbf0 RCX: 0000000000c10000 [Wed Feb 6 08:13:00 2019][1691279.680554] RDX: ffff9149ff89b780 RSI: 0000000001510000 RDI: ffff912b6ac432dc [Wed Feb 6 08:13:00 2019][1691279.680554] RBP: ffff9159ee60fbb8 R08: ffff9129fef9b780 R09: 0000000000000000 [Wed Feb 6 08:13:00 2019][1691279.680555] R10: ffff911b3fc07600 R11: ffffdbb275e2d800 R12: 0000000000000000 [Wed Feb 6 08:13:00 2019][1691279.680556] R13: 0000000000000000 R14: 0000000000000002 R15: ffff913adb084ec0 [Wed Feb 6 08:13:00 2019][1691279.680557] FS: 00007f886f401880(0000) GS:ffff9129fef80000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:00 2019][1691279.680558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:00 2019][1691279.680559] CR2: 00007ff74389080d CR3: 00000020dae10000 CR4: 00000000003407e0 [Wed Feb 6 08:13:00 2019][1691279.680560] Call Trace: [Wed Feb 6 08:13:00 2019][1691279.680566] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:13:00 2019][1691279.680570] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:13:00 2019][1691279.680613] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.680646] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.680649] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:13:00 2019][1691279.680681] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.680719] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.680754] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.680789] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.680831] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.680869] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.680873] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:13:00 2019][1691279.680875] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:13:00 2019][1691279.680913] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.680951] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.680954] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:00 2019][1691279.680956] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:00 2019][1691279.680959] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:00 2019][1691279.680960] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:00 2019][1691279.680980] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:13:00 2019][1691279.758472] NMI watchdog: BUG: soft lockup - CPU#33 stuck for 23s! [ldlm_cn01_023:28088] [Wed Feb 6 08:13:00 2019][1691279.758518] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:13:00 2019][1691279.758539] CPU: 33 PID: 28088 Comm: ldlm_cn01_023 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:13:00 2019][1691279.758540] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:13:00 2019][1691279.758541] task: ffff915929b4c100 ti: ffff91578b654000 task.ti: ffff91578b654000 [Wed Feb 6 08:13:00 2019][1691279.758547] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:13:00 2019][1691279.758548] RSP: 0018:ffff91578b657c38 EFLAGS: 00000246 [Wed Feb 6 08:13:00 2019][1691279.758549] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001090000 [Wed Feb 6 08:13:00 2019][1691279.758550] RDX: ffff9149ff81b780 RSI: 0000000001110001 RDI: ffff912b6ac432dc [Wed Feb 6 08:13:00 2019][1691279.758550] RBP: ffff91578b657c38 R08: ffff9139ff81b780 R09: 0000000000000000 [Wed Feb 6 08:13:00 2019][1691279.758551] R10: ffff911b3fc07600 R11: ffffdbb2243ed400 R12: ffff91578b657be0 [Wed Feb 6 08:13:00 2019][1691279.758552] R13: ffff914fcb37fc80 R14: ffff91578b657ba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:13:00 2019][1691279.758553] FS: 00007f277c7c2700(0000) GS:ffff9139ff800000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:00 2019][1691279.758554] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:00 2019][1691279.758555] CR2: 00007f2781328000 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:13:00 2019][1691279.758556] Call Trace: [Wed Feb 6 08:13:00 2019][1691279.758559] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:13:00 2019][1691279.758563] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:13:00 2019][1691279.758603] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.758636] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.758671] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.758707] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.758742] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.758780] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.758818] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.758821] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:13:00 2019][1691279.758858] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.758896] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.758899] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:00 2019][1691279.758901] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:00 2019][1691279.758903] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:00 2019][1691279.758905] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:00 2019][1691279.758924] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:13:00 2019][1691279.820473] NMI watchdog: BUG: soft lockup - CPU#40 stuck for 23s! [ldlm_cn00_015:14886] [Wed Feb 6 08:13:00 2019][1691279.820501] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:13:00 2019][1691279.820513] CPU: 40 PID: 14886 Comm: ldlm_cn00_015 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:13:00 2019][1691279.820514] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:13:00 2019][1691279.820515] task: ffff912fe0b65140 ti: ffff91593178c000 task.ti: ffff91593178c000 [Wed Feb 6 08:13:00 2019][1691279.820519] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:13:00 2019][1691279.820520] RSP: 0018:ffff91593178fbb8 EFLAGS: 00000246 [Wed Feb 6 08:13:00 2019][1691279.820520] RAX: 0000000000000000 RBX: ffff91593178fbf0 RCX: 0000000001410000 [Wed Feb 6 08:13:00 2019][1691279.820521] RDX: ffff9149ff6db780 RSI: 0000000000710001 RDI: ffff912b6ac432dc [Wed Feb 6 08:13:00 2019][1691279.820522] RBP: ffff91593178fbb8 R08: ffff9129ff09b780 R09: 0000000000000000 [Wed Feb 6 08:13:00 2019][1691279.820523] R10: ffff911b3fc07600 R11: 000000000000015d R12: 0000000000000000 [Wed Feb 6 08:13:00 2019][1691279.820523] R13: 0000000000000000 R14: 0000000000000002 R15: ffff912430572640 [Wed Feb 6 08:13:00 2019][1691279.820525] FS: 00007f78f341b880(0000) GS:ffff9129ff080000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:00 2019][1691279.820525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:00 2019][1691279.820526] CR2: 00007f63fb343000 CR3: 0000004033a9a000 CR4: 00000000003407e0 [Wed Feb 6 08:13:00 2019][1691279.820527] Call Trace: [Wed Feb 6 08:13:00 2019][1691279.820529] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:13:00 2019][1691279.820531] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:13:00 2019][1691279.820563] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.820596] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.820598] [] ? native_queued_spin_lock_slowpath+0x156/0x200 [Wed Feb 6 08:13:00 2019][1691279.820630] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.820665] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.820700] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.820734] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.820772] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.820809] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.820811] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:13:00 2019][1691279.820812] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:13:00 2019][1691279.820849] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.820887] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:00 2019][1691279.820888] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:00 2019][1691279.820890] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:00 2019][1691279.820892] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:00 2019][1691279.820893] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:00 2019][1691279.820912] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:13:01 2019][1691280.750488] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000310000 [Wed Feb 6 08:13:01 2019][1691280.757793] RDX: ffff9149ff69b780 RSI: 0000000000510001 RDI: ffff912b6ac432dc [Wed Feb 6 08:13:01 2019][1691280.765099] RBP: ffff9159f3e37c38 R08: ffff9149ff65b780 R09: 0000000000000000 [Wed Feb 6 08:13:01 2019][1691280.772405] R10: ffff911b3fc07600 R11: ffff9149fa30ea20 R12: ffff9159f3e37be0 [Wed Feb 6 08:13:01 2019][1691280.779712] R13: ffff914fcb37fc80 R14: ffff9159f3e37ba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:13:01 2019][1691280.787017] FS: 00007f747490e700(0000) GS:ffff9149ff640000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:01 2019][1691280.795278] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:01 2019][1691280.801196] CR2: 00007f74895f3f48 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:13:01 2019][1691280.808503] Call Trace: [Wed Feb 6 08:13:01 2019][1691280.811134] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:13:01 2019][1691280.817571] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:13:01 2019][1691280.823269] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:13:01 2019][1691280.830004] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:13:01 2019][1691280.836741] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:13:01 2019][1691280.843821] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:13:01 2019][1691280.850727] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:13:01 2019][1691280.857814] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:01 2019][1691280.865673] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:01 2019][1691280.872632] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:13:01 2019][1691280.879071] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:13:01 2019][1691280.885111] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:01 2019][1691280.891585] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:01 2019][1691280.899153] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:01 2019][1691280.904202] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:01 2019][1691280.910470] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:01 2019][1691280.917082] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:01 2019][1691280.923348] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:13:03 2019][1691283.484566] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_cn01_029:28330] [Wed Feb 6 08:13:03 2019][1691283.492736] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:13:03 2019][1691283.599508] CPU: 1 PID: 28330 Comm: ldlm_cn01_029 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:13:03 2019][1691283.612183] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:13:03 2019][1691283.619925] task: ffff9159aa3de180 ti: ffff915863708000 task.ti: ffff915863708000 [Wed Feb 6 08:13:03 2019][1691283.627576] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:13:03 2019][1691283.637683] RSP: 0018:ffff91586370bbb8 EFLAGS: 00000246 [Wed Feb 6 08:13:03 2019][1691283.643168] RAX: 0000000000000000 RBX: ffff91586370bbf0 RCX: 0000000000090000 [Wed Feb 6 08:13:03 2019][1691283.650475] RDX: ffff9139ff89b780 RSI: 0000000001490000 RDI: ffff912b6ac432dc [Wed Feb 6 08:13:03 2019][1691283.657781] RBP: ffff91586370bbb8 R08: ffff9139ff61b780 R09: 0000000000000000 [Wed Feb 6 08:13:03 2019][1691283.665086] R10: ffff911b3fc07600 R11: ffffdbb2243ed400 R12: 0000000000000000 [Wed Feb 6 08:13:03 2019][1691283.672394] R13: 0000000000000000 R14: 0000000000000002 R15: ffff91326e736c00 [Wed Feb 6 08:13:03 2019][1691283.679701] FS: 00007f63fb2d1700(0000) GS:ffff9139ff600000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:03 2019][1691283.687958] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:03 2019][1691283.693879] CR2: 00007f63fb343000 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:13:03 2019][1691283.701184] Call Trace: [Wed Feb 6 08:13:03 2019][1691283.703815] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:13:03 2019][1691283.710252] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:13:03 2019][1691283.715944] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:13:03 2019][1691283.722675] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:13:03 2019][1691283.729718] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:13:03 2019][1691283.737313] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:13:03 2019][1691283.744044] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:13:03 2019][1691283.751124] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:13:04 2019][1691283.758029] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:13:04 2019][1691283.765115] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:04 2019][1691283.772975] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:04 2019][1691283.779934] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:13:04 2019][1691283.785716] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:04 2019][1691283.792190] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:04 2019][1691283.799756] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:04 2019][1691283.804806] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:04 2019][1691283.811072] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:04 2019][1691283.817684] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:04 2019][1691283.823951] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:13:07 2019][1691287.688671] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [ldlm_cn01_014:21074] [Wed Feb 6 08:13:07 2019][1691287.696934] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:13:08 2019][1691287.803720] CPU: 25 PID: 21074 Comm: ldlm_cn01_014 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:13:08 2019][1691287.816485] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:13:08 2019][1691287.824226] task: ffff914753371040 ti: ffff9158d76f4000 task.ti: ffff9158d76f4000 [Wed Feb 6 08:13:08 2019][1691287.831878] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:13:08 2019][1691287.841984] RSP: 0018:ffff9158d76f7c38 EFLAGS: 00000246 [Wed Feb 6 08:13:08 2019][1691287.847470] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000c90000 [Wed Feb 6 08:13:08 2019][1691287.854775] RDX: ffff9129fee5b780 RSI: 0000000000210001 RDI: ffff912b6ac432dc [Wed Feb 6 08:13:08 2019][1691287.862080] RBP: ffff9158d76f7c38 R08: ffff9139ff79b780 R09: 0000000000000000 [Wed Feb 6 08:13:08 2019][1691287.869387] R10: ffff911b3fc07600 R11: ffffdbb27f174600 R12: ffff9158d76f7be0 [Wed Feb 6 08:13:08 2019][1691287.876693] R13: ffff914fcb37fc80 R14: ffff9158d76f7ba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:13:08 2019][1691287.884000] FS: 00007f277d7c4700(0000) GS:ffff9139ff780000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:08 2019][1691287.892261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:08 2019][1691287.898197] CR2: 00007f2781328000 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:13:08 2019][1691287.905503] Call Trace: [Wed Feb 6 08:13:08 2019][1691287.908132] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:13:08 2019][1691287.914570] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:13:08 2019][1691287.920264] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:13:08 2019][1691287.926993] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:13:08 2019][1691287.933731] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:13:08 2019][1691287.940811] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:13:08 2019][1691287.947719] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:13:08 2019][1691287.954802] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:08 2019][1691287.962663] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:08 2019][1691287.969622] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:13:08 2019][1691287.976060] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:13:08 2019][1691287.982101] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:08 2019][1691287.988577] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:08 2019][1691287.996144] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:08 2019][1691288.001194] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:08 2019][1691288.007460] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:08 2019][1691288.014072] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:08 2019][1691288.020338] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:13:10 2019][1691290.334438] Lustre: 27688:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43:10s); client may timeout. req@ffff9120a5f25850 x1624705420246272/t14368976181(0) o4->cd2fc9d9-2801-0b42-234d-619ca30bf3c7@10.8.9.8@o2ib6:483/0 lens 488/416 e 0 to 0 dl 1549469618 ref 1 fl Complete:/0/0 rc 0/0 [Wed Feb 6 08:13:10 2019][1691290.642157] LustreError: 9840:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(23452) req@ffff9134fde40850 x1624712309498800/t0(0) o4->3e106753-f830-b044-4cf1-87da4380b4a5@10.8.6.12@o2ib6:562/0 lens 488/448 e 0 to 0 dl 1549469697 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:13:10 2019][1691290.666896] LustreError: 9840:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message [Wed Feb 6 08:13:10 2019][1691290.676670] Lustre: fir-MDT0002: Bulk IO write error with 3e106753-f830-b044-4cf1-87da4380b4a5 (at 10.8.6.12@o2ib6), client will retry: rc = -110 [Wed Feb 6 08:13:10 2019][1691290.689870] Lustre: Skipped 3 previous similar messages [Wed Feb 6 08:13:11 2019][1691291.417777] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 08:13:11 2019][1691291.598534] LNetError: 7282:0:(o2iblnd_cb.c:3281:kiblnd_cm_callback()) 10.0.10.201@o2ib7 DISCONNECTED [Wed Feb 6 08:13:11 2019][1691291.617571] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 39 seconds [Wed Feb 6 08:13:11 2019][1691291.627835] LNet: 5873:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 140 previous similar messages [Wed Feb 6 08:13:11 2019][1691291.637508] LustreError: 5873:0:(events.c:450:server_bulk_callback()) event type 5, status -110, desc ffff91564f7d1200 [Wed Feb 6 08:13:11 2019][1691291.648394] LustreError: 5873:0:(events.c:450:server_bulk_callback()) event type 5, status -110, desc ffff91514d7cc600 [Wed Feb 6 08:13:11 2019][1691291.659281] LustreError: 5873:0:(events.c:450:server_bulk_callback()) event type 5, status -110, desc ffff91514d7ce200 [Wed Feb 6 08:13:11 2019][1691291.670166] LustreError: 5873:0:(events.c:450:server_bulk_callback()) event type 5, status -110, desc ffff912b62c7fa00 [Wed Feb 6 08:13:11 2019][1691291.681337] LustreError: 5873:0:(events.c:450:server_bulk_callback()) event type 5, status -110, desc ffff915972f07200 [Wed Feb 6 08:13:11 2019][1691291.692914] LustreError: 5873:0:(events.c:450:server_bulk_callback()) event type 5, status -110, desc ffff915972f06a00 [Wed Feb 6 08:13:11 2019][1691291.715771] NMI watchdog: BUG: soft lockup - CPU#28 stuck for 21s! [ldlm_cn00_024:20043] [Wed Feb 6 08:13:11 2019][1691291.724031] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:13:12 2019][1691291.830801] CPU: 28 PID: 20043 Comm: ldlm_cn00_024 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:13:12 2019][1691291.843565] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:13:12 2019][1691291.851304] task: ffff9139f700a080 ti: ffff9158ffab0000 task.ti: ffff9158ffab0000 [Wed Feb 6 08:13:12 2019][1691291.858957] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:13:12 2019][1691291.869062] RSP: 0018:ffff9158ffab3c38 EFLAGS: 00000246 [Wed Feb 6 08:13:12 2019][1691291.874549] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000e10000 [Wed Feb 6 08:13:12 2019][1691291.881854] RDX: ffff9129ff05b780 RSI: 0000000001210001 RDI: ffff912b6ac432dc [Wed Feb 6 08:13:12 2019][1691291.889162] RBP: ffff9158ffab3c38 R08: ffff9129fefdb780 R09: 0000000000000000 [Wed Feb 6 08:13:12 2019][1691291.896469] R10: ffff911b3fc07600 R11: ffffdbb22e892400 R12: ffff9158ffab3be0 [Wed Feb 6 08:13:12 2019][1691291.903773] R13: ffff914fcb37fc80 R14: ffff9158ffab3ba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:13:12 2019][1691291.911081] FS: 00007fbb7a00a740(0000) GS:ffff9129fefc0000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:12 2019][1691291.919341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:12 2019][1691291.925259] CR2: 00007fbb79a63d10 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:13:12 2019][1691291.932565] Call Trace: [Wed Feb 6 08:13:12 2019][1691291.935196] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:13:12 2019][1691291.941634] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:13:12 2019][1691291.947332] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:13:12 2019][1691291.954064] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:13:12 2019][1691291.960795] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:13:12 2019][1691291.967875] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:13:12 2019][1691291.974780] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:13:12 2019][1691291.981867] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:12 2019][1691291.989725] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:12 2019][1691291.996687] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:13:12 2019][1691292.002466] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:12 2019][1691292.008940] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:12 2019][1691292.016505] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:12 2019][1691292.021558] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:12 2019][1691292.027825] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:12 2019][1691292.034436] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:12 2019][1691292.040701] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:13:15 2019][1691295.510868] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [ldlm_cn00_018:14898] [Wed Feb 6 08:13:15 2019][1691295.519040] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:13:15 2019][1691295.625807] CPU: 4 PID: 14898 Comm: ldlm_cn00_018 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:13:15 2019][1691295.638486] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:13:15 2019][1691295.646226] task: ffff9159f6d1a080 ti: ffff91597a7a0000 task.ti: ffff91597a7a0000 [Wed Feb 6 08:13:15 2019][1691295.653879] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:13:15 2019][1691295.663983] RSP: 0018:ffff91597a7a3bb8 EFLAGS: 00000246 [Wed Feb 6 08:13:15 2019][1691295.669469] RAX: 0000000000000000 RBX: ffff91597a7a3bf0 RCX: 0000000000210000 [Wed Feb 6 08:13:15 2019][1691295.676774] RDX: ffff9139ff81b780 RSI: 0000000001090000 RDI: ffff912b6ac432dc [Wed Feb 6 08:13:15 2019][1691295.684091] RBP: ffff91597a7a3bb8 R08: ffff9129fee5b780 R09: 0000000000000000 [Wed Feb 6 08:13:15 2019][1691295.691397] R10: ffff911b3fc07600 R11: ffffdbb20811cc00 R12: 0000000000000000 [Wed Feb 6 08:13:15 2019][1691295.697875] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [ldlm_cn02_000:6035] [Wed Feb 6 08:13:15 2019][1691295.697920] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:13:16 2019][1691295.697940] CPU: 26 PID: 6035 Comm: ldlm_cn02_000 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:13:16 2019][1691295.697941] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:13:16 2019][1691295.697942] task: ffff9129f3d25140 ti: ffff9159b50f4000 task.ti: ffff9159b50f4000 [Wed Feb 6 08:13:16 2019][1691295.697950] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:13:16 2019][1691295.697951] RSP: 0018:ffff9159b50f7c38 EFLAGS: 00000246 [Wed Feb 6 08:13:16 2019][1691295.697952] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000000d10000 [Wed Feb 6 08:13:16 2019][1691295.697953] RDX: ffff9129ff01b780 RSI: 0000000001010001 RDI: ffff912b6ac432dc [Wed Feb 6 08:13:16 2019][1691295.697953] RBP: ffff9159b50f7c38 R08: ffff9149ff79b780 R09: 0000000000000000 [Wed Feb 6 08:13:16 2019][1691295.697954] R10: ffff911b3fc07600 R11: 0000000000000000 R12: ffff9159b50f7be0 [Wed Feb 6 08:13:16 2019][1691295.697955] R13: ffff914fcb37fc80 R14: ffff9159b50f7ba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:13:16 2019][1691295.697956] FS: 00007f72eaa7c700(0000) GS:ffff9149ff780000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:16 2019][1691295.697957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:16 2019][1691295.697958] CR2: 00007fcf5e129024 CR3: 000000403bd1e000 CR4: 00000000003407e0 [Wed Feb 6 08:13:16 2019][1691295.697959] Call Trace: [Wed Feb 6 08:13:16 2019][1691295.697965] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:13:16 2019][1691295.697969] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:13:16 2019][1691295.698012] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.698044] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.698080] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.698115] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.698149] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.698191] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.698228] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.698231] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:13:16 2019][1691295.698234] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:13:16 2019][1691295.698271] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.698308] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.698311] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:16 2019][1691295.698312] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:16 2019][1691295.698315] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:16 2019][1691295.698317] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:16 2019][1691295.698336] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:13:16 2019][1691295.777875] NMI watchdog: BUG: soft lockup - CPU#35 stuck for 22s! [ldlm_cn03_025:28518] [Wed Feb 6 08:13:16 2019][1691295.777915] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:13:16 2019][1691295.777933] CPU: 35 PID: 28518 Comm: ldlm_cn03_025 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:13:16 2019][1691295.777934] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:13:16 2019][1691295.777935] task: ffff9149d5fc9040 ti: ffff913ac391c000 task.ti: ffff913ac391c000 [Wed Feb 6 08:13:16 2019][1691295.777940] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:13:16 2019][1691295.777941] RSP: 0018:ffff913ac391fbb8 EFLAGS: 00000246 [Wed Feb 6 08:13:16 2019][1691295.777942] RAX: 0000000000000000 RBX: ffff913ac391fbf0 RCX: 0000000001190000 [Wed Feb 6 08:13:16 2019][1691295.777942] RDX: ffff9139ff89b780 RSI: 0000000001490000 RDI: ffff912b6ac432dc [Wed Feb 6 08:13:16 2019][1691295.777943] RBP: ffff913ac391fbb8 R08: ffff915a3f61b780 R09: 0000000000000000 [Wed Feb 6 08:13:16 2019][1691295.777944] R10: ffff911b3fc07600 R11: 0000000000003c00 R12: 0000000000000000 [Wed Feb 6 08:13:16 2019][1691295.777944] R13: 0000000000000000 R14: 0000000000000002 R15: ffff911c7a054ec0 [Wed Feb 6 08:13:16 2019][1691295.777946] FS: 00007f2b3eadb740(0000) GS:ffff915a3f600000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:16 2019][1691295.777946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:16 2019][1691295.777947] CR2: 00007f2b3d82280d CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:13:16 2019][1691295.777948] Call Trace: [Wed Feb 6 08:13:16 2019][1691295.777953] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:13:16 2019][1691295.777955] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:13:16 2019][1691295.777996] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.778023] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.778025] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:13:16 2019][1691295.778051] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.778081] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.778111] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.778139] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.778175] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.778206] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.778209] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:13:16 2019][1691295.778210] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:13:16 2019][1691295.778242] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.778273] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.778275] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:16 2019][1691295.778277] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:16 2019][1691295.778279] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:16 2019][1691295.778280] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:16 2019][1691295.778298] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:13:16 2019][1691295.837875] NMI watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [ldlm_cn02_020:23325] [Wed Feb 6 08:13:16 2019][1691295.837901] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:13:16 2019][1691295.837914] CPU: 42 PID: 23325 Comm: ldlm_cn02_020 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:13:16 2019][1691295.837915] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:13:16 2019][1691295.837916] task: ffff91599ff99040 ti: ffff91580caf8000 task.ti: ffff91580caf8000 [Wed Feb 6 08:13:16 2019][1691295.837919] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:13:16 2019][1691295.837920] RSP: 0018:ffff91580cafbc38 EFLAGS: 00000246 [Wed Feb 6 08:13:16 2019][1691295.837921] RAX: 0000000000000000 RBX: 00000001c0e40fd2 RCX: 0000000001510000 [Wed Feb 6 08:13:16 2019][1691295.837921] RDX: ffff9129fef5b780 RSI: 0000000000a10001 RDI: ffff912b6ac432dc [Wed Feb 6 08:13:16 2019][1691295.837922] RBP: ffff91580cafbc38 R08: ffff9149ff89b780 R09: 0000000000000000 [Wed Feb 6 08:13:16 2019][1691295.837923] R10: ffff911b3fc07600 R11: ffffdbb2843e0a00 R12: ffff91580cafbbe0 [Wed Feb 6 08:13:16 2019][1691295.837923] R13: ffff914fcb37fc80 R14: ffff91580cafbba0 R15: ffffffffc0ac8378 [Wed Feb 6 08:13:16 2019][1691295.837925] FS: 00007f72eaa7c700(0000) GS:ffff9149ff880000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:16 2019][1691295.837926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:16 2019][1691295.837926] CR2: 00007f78e136d464 CR3: 000000403bd1e000 CR4: 00000000003407e0 [Wed Feb 6 08:13:16 2019][1691295.837927] Call Trace: [Wed Feb 6 08:13:16 2019][1691295.837930] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:13:16 2019][1691295.837932] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:13:16 2019][1691295.837963] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.837994] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.838027] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.838061] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.838095] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.838132] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.838168] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.838170] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:13:16 2019][1691295.838171] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:13:16 2019][1691295.838208] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.838244] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:16 2019][1691295.838245] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:16 2019][1691295.838247] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:16 2019][1691295.838249] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:16 2019][1691295.838250] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:16 2019][1691295.838269] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:13:17 2019][1691296.755378] R13: 0000000000000000 R14: 0000000000000002 R15: ffff914f65f09200 [Wed Feb 6 08:13:17 2019][1691296.762685] FS: 00007fe2d3435880(0000) GS:ffff9129fee40000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:17 2019][1691296.770943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:17 2019][1691296.776863] CR2: 00007fe2d3446000 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:13:17 2019][1691296.784167] Call Trace: [Wed Feb 6 08:13:17 2019][1691296.786799] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:13:17 2019][1691296.793237] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:13:17 2019][1691296.798934] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [Wed Feb 6 08:13:17 2019][1691296.805667] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [Wed Feb 6 08:13:17 2019][1691296.812710] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:13:17 2019][1691296.820306] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [Wed Feb 6 08:13:17 2019][1691296.827034] [] ldlm_request_cancel+0x19b/0x740 [ptlrpc] [Wed Feb 6 08:13:17 2019][1691296.834115] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:13:17 2019][1691296.841020] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:13:17 2019][1691296.848108] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:17 2019][1691296.855964] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:17 2019][1691296.862926] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:13:17 2019][1691296.869366] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:13:17 2019][1691296.875404] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:17 2019][1691296.881880] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:17 2019][1691296.889447] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:17 2019][1691296.894499] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:17 2019][1691296.900766] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:17 2019][1691296.907378] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:17 2019][1691296.913642] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:13:19 2019][1691299.379722] LNet: 5873:0:(o2iblnd_cb.c:1484:kiblnd_reconnect_peer()) Abort reconnection of 10.0.10.201@o2ib7: accepting [Wed Feb 6 08:13:19 2019][1691299.379782] LNetError: 7282:0:(o2iblnd_cb.c:3281:kiblnd_cm_callback()) 10.0.10.210@o2ib7 DISCONNECTED [Wed Feb 6 08:13:19 2019][1691299.560969] NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [ldlm_cn02_025:28509] [Wed Feb 6 08:13:19 2019][1691299.569234] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver[1691299.594971] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 21s! [ldlm_cn02_029:28945] [Wed Feb 6 08:13:19 2019][1691299.594973] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:13:19 2019][1691299.595039] CPU: 14 PID: 28945 Comm: ldlm_cn02_029 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:13:19 2019][1691299.595039] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:13:19 2019][1691299.595041] task: ffff913ac8551040 ti: ffff9145661a4000 task.ti: ffff9145661a4000 [Wed Feb 6 08:13:19 2019][1691299.595042] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:13:19 2019][1691299.595050] RSP: 0018:ffff9145661a7c10 EFLAGS: 00000246 [Wed Feb 6 08:13:19 2019][1691299.595051] RAX: 0000000000000000 RBX: ffff9145661a7ba0 RCX: 0000000000710000 [Wed Feb 6 08:13:19 2019][1691299.595051] RDX: ffff9129ff09b780 RSI: 0000000001410001 RDI: ffff912b6ac432dc [Wed Feb 6 08:13:20 2019][1691299.595052] RBP: ffff9145661a7c10 R08: ffff9149ff6db780 R09: 0000000000000000 [Wed Feb 6 08:13:20 2019][1691299.595053] R10: ffff911b3fc07600 R11: ffffdbb2ac6bac00 R12: ffff9145661a7bb0 [Wed Feb 6 08:13:20 2019][1691299.595053] R13: ffff912b6ac432dc R14: 0000000001590000 R15: ffff915a3f69b780 [Wed Feb 6 08:13:20 2019][1691299.595055] FS: 00007fef6a3cd740(0000) GS:ffff9149ff6c0000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:20 2019][1691299.595056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:20 2019][1691299.595056] CR2: 00007f2b3106c010 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:13:20 2019][1691299.595058] Call Trace: [Wed Feb 6 08:13:20 2019][1691299.595064] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:13:20 2019][1691299.595068] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:13:20 2019][1691299.595107] [] __ldlm_reprocess_all+0xfa/0x380 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.595132] [] ldlm_reprocess_all+0x10/0x20 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.595162] [] ldlm_request_cancel+0x3ed/0x740 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.595191] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.595219] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.595254] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.595286] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.595289] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:13:20 2019][1691299.595320] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.595352] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.595354] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:20 2019][1691299.595356] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:20 2019][1691299.595358] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:20 2019][1691299.595359] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:20 2019][1691299.595360] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:13:20 2019][1691299.855975] NMI watchdog: BUG: soft lockup - CPU#44 stuck for 22s! [ldlm_cn00_014:14885] [Wed Feb 6 08:13:20 2019][1691299.855976] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:13:20 2019][1691299.856046] CPU: 44 PID: 14885 Comm: ldlm_cn00_014 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:13:20 2019][1691299.856046] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:13:20 2019][1691299.856047] task: ffff912fe0b61040 ti: ffff9159fb438000 task.ti: ffff9159fb438000 [Wed Feb 6 08:13:20 2019][1691299.856048] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [Wed Feb 6 08:13:20 2019][1691299.856055] RSP: 0018:ffff9159fb43bc10 EFLAGS: 00000246 [Wed Feb 6 08:13:20 2019][1691299.856056] RAX: 0000000000000000 RBX: ffff9159fb43bba0 RCX: 0000000001610000 [Wed Feb 6 08:13:20 2019][1691299.856056] RDX: ffff9129fee1b780 RSI: 0000000000010001 RDI: ffff912b6ac432dc [Wed Feb 6 08:13:20 2019][1691299.856057] RBP: ffff9159fb43bc10 R08: ffff9129ff0db780 R09: 0000000000000000 [Wed Feb 6 08:13:20 2019][1691299.856057] R10: ffff911b3fc07600 R11: ffffdbb262b28400 R12: ffff9159fb43bbb0 [Wed Feb 6 08:13:20 2019][1691299.856058] R13: ffff912b6ac432dc R14: 0000000000d10001 R15: ffff9149ff79b780 [Wed Feb 6 08:13:20 2019][1691299.856059] FS: 00007f2781319740(0000) GS:ffff9129ff0c0000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:20 2019][1691299.856060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:20 2019][1691299.856061] CR2: 00007f277416c0e8 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:13:20 2019][1691299.856062] Call Trace: [Wed Feb 6 08:13:20 2019][1691299.856067] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:13:20 2019][1691299.856070] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:13:20 2019][1691299.856111] [] __ldlm_reprocess_all+0xfa/0x380 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.856137] [] ldlm_reprocess_all+0x10/0x20 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.856166] [] ldlm_request_cancel+0x3ed/0x740 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.856195] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.856222] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.856255] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.856285] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.856288] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:13:20 2019][1691299.856290] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:13:20 2019][1691299.856320] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.856350] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691299.856352] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:20 2019][1691299.856354] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:20 2019][1691299.856356] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:20 2019][1691299.856358] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:20 2019][1691299.856358] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [Wed Feb 6 08:13:20 2019] [Wed Feb 6 08:13:20 2019][1691300.293698] nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:13:20 2019][1691300.375870] CPU: 10 PID: 28509 Comm: ldlm_cn02_025 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:13:20 2019][1691300.388633] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:13:20 2019][1691300.396374] task: ffff9149f83fd140 ti: ffff913a92ef8000 task.ti: ffff913a92ef8000 [Wed Feb 6 08:13:20 2019][1691300.404028] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [Wed Feb 6 08:13:20 2019][1691300.414133] RSP: 0018:ffff913a92efbc10 EFLAGS: 00000246 [Wed Feb 6 08:13:20 2019][1691300.419617] RAX: 0000000000000000 RBX: ffff913a92efbba0 RCX: 0000000000510000 [Wed Feb 6 08:13:20 2019][1691300.426925] RDX: ffff9139ff89b780 RSI: 0000000001490001 RDI: ffff912b6ac432dc [Wed Feb 6 08:13:20 2019][1691300.434232] RBP: ffff913a92efbc10 R08: ffff9149ff69b780 R09: 0000000000000000 [Wed Feb 6 08:13:20 2019][1691300.441536] R10: ffff911b3fc07600 R11: ffffdbb2ac6bac00 R12: ffff913a92efbbb0 [Wed Feb 6 08:13:20 2019][1691300.448843] R13: ffff912b6ac432dc R14: 0000000000210000 R15: ffff9129fee5b780 [Wed Feb 6 08:13:20 2019][1691300.456149] FS: 00007f72eaa7c700(0000) GS:ffff9149ff680000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:20 2019][1691300.464409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:20 2019][1691300.470327] CR2: 00007f0ff836e518 CR3: 000000403bd1e000 CR4: 00000000003407e0 [Wed Feb 6 08:13:20 2019][1691300.477635] Call Trace: [Wed Feb 6 08:13:20 2019][1691300.480267] [] queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 08:13:20 2019][1691300.486712] [] _raw_spin_lock+0x20/0x30 [Wed Feb 6 08:13:20 2019][1691300.492409] [] __ldlm_reprocess_all+0xfa/0x380 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691300.499485] [] ldlm_reprocess_all+0x10/0x20 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691300.506299] [] ldlm_request_cancel+0x3ed/0x740 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691300.513370] [] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691300.520268] [] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691300.527347] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691300.535204] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691300.542162] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:13:20 2019][1691300.548600] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:13:20 2019][1691300.554636] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691300.561108] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:20 2019][1691300.568674] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:20 2019][1691300.573726] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:20 2019][1691300.579995] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:20 2019][1691300.586604] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:20 2019][1691300.592868] Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 94 ae 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [Wed Feb 6 08:13:23 2019][1691303.111325] Lustre: fir-MDT0002: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 08:13:23 2019][1691303.122363] Lustre: Skipped 385 previous similar messages [Wed Feb 6 08:13:33 2019][1691313.395330] LNet: Service thread pid 17050 was inactive for 413.18s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 08:13:33 2019][1691313.412442] Pid: 17050, comm: mdt03_015 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:13:33 2019][1691313.422357] Call Trace: [Wed Feb 6 08:13:33 2019][1691313.424989] [] 0xffffffffffffffff [Wed Feb 6 08:13:33 2019][1691313.430198] LustreError: dumping log to /tmp/lustre-log.1549469651.17050 [Wed Feb 6 08:13:34 2019][1691314.614387] LustreError: 8956:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk READ req@ffff9134fde44c50 x1624705150646000/t0(0) o3->f0b653e6-2546-c659-97c8-5d3c41619c38@10.8.15.9@o2ib6:541/0 lens 488/440 e 0 to 0 dl 1549469676 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:13:34 2019][1691314.638900] LustreError: 8956:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 22 previous similar messages [Wed Feb 6 08:13:34 2019][1691314.648487] Lustre: fir-MDT0002: Bulk IO read error with f0b653e6-2546-c659-97c8-5d3c41619c38 (at 10.8.15.9@o2ib6), client will retry: rc -110 [Wed Feb 6 08:13:43 2019][1691323.612575] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [ldlm_bl_24:15987] [Wed Feb 6 08:13:43 2019][1691323.620578] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:13:43 2019][1691323.727348] CPU: 16 PID: 15987 Comm: ldlm_bl_24 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:13:43 2019][1691323.739851] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:13:43 2019][1691323.747590] task: ffff9149a0fce180 ti: ffff91461b228000 task.ti: ffff91461b228000 [Wed Feb 6 08:13:43 2019][1691323.755244] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 08:13:43 2019][1691323.765740] RSP: 0018:ffff91461b22bbb0 EFLAGS: 00000282 [Wed Feb 6 08:13:44 2019][1691323.771227] RAX: 0000000000000101 RBX: ffff913b50496120 RCX: ffff913b50496120 [Wed Feb 6 08:13:44 2019][1691323.778533] RDX: ffff91461b22bca8 RSI: ffff9151f0a12400 RDI: ffff912e64fd0240 [Wed Feb 6 08:13:44 2019][1691323.785840] RBP: ffff91461b22bc08 R08: ffff91461b22bca8 R09: 00000000c000173e [Wed Feb 6 08:13:44 2019][1691323.793144] R10: 000000000000003e R11: ffff9151f0a12400 R12: ffff91461b22bca8 [Wed Feb 6 08:13:44 2019][1691323.800450] R13: 00000000c000173e R14: 000000000000003e R15: ffff9151f0a12400 [Wed Feb 6 08:13:44 2019][1691323.807758] FS: 00007f78f341b880(0000) GS:ffff9129fef00000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:44 2019][1691323.816018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:44 2019][1691323.821935] CR2: 00007f63fb343000 CR3: 0000004033a9a000 CR4: 00000000003407e0 [Wed Feb 6 08:13:44 2019][1691323.829244] Call Trace: [Wed Feb 6 08:13:44 2019][1691323.831906] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 08:13:44 2019][1691323.839588] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 08:13:44 2019][1691323.847529] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 08:13:44 2019][1691323.854690] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 08:13:44 2019][1691323.861848] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 08:13:44 2019][1691323.870306] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:13:44 2019][1691323.878507] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:13:44 2019][1691323.885556] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:13:44 2019][1691323.891335] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:13:44 2019][1691323.898899] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:44 2019][1691323.903951] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:44 2019][1691323.910220] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:44 2019][1691323.916831] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:44 2019][1691323.923097] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 08:13:46 2019][1691326.081876] INFO: task mdt03_023:17218 blocked for more than 120 seconds. [Wed Feb 6 08:13:46 2019][1691326.088843] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Feb 6 08:13:46 2019][1691326.096856] mdt03_023 D ffff9159fa256180 0 17218 2 0x00000080 [Wed Feb 6 08:13:46 2019][1691326.104155] Call Trace: [Wed Feb 6 08:13:46 2019][1691326.106802] [] schedule+0x29/0x70 [Wed Feb 6 08:13:46 2019][1691326.111959] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 08:13:46 2019][1691326.118589] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 08:13:46 2019][1691326.125477] [] down_write+0x2d/0x3d [Wed Feb 6 08:13:46 2019][1691326.130819] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 08:13:46 2019][1691326.137721] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 08:13:46 2019][1691326.144656] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 08:13:46 2019][1691326.151814] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:13:46 2019][1691326.157639] [] ? ldlm_cli_enqueue_local+0x27d/0x870 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.165143] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 08:13:46 2019][1691326.173175] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 08:13:46 2019][1691326.180603] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 08:13:46 2019][1691326.187845] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 08:13:46 2019][1691326.194577] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:13:46 2019][1691326.201917] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 08:13:46 2019][1691326.208564] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 08:13:46 2019][1691326.215265] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.221909] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:13:46 2019][1691326.228551] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 08:13:46 2019][1691326.235235] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.242139] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 08:13:46 2019][1691326.249474] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 08:13:46 2019][1691326.256075] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.263379] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.271122] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.277480] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.284612] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.292380] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 08:13:46 2019][1691326.299674] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.307585] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.314564] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:13:46 2019][1691326.321021] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:13:46 2019][1691326.327106] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.333630] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.341215] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:46 2019][1691326.346282] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:46 2019][1691326.352572] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:46 2019][1691326.359206] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:46 2019][1691326.365493] INFO: task mdt03_034:24923 blocked for more than 120 seconds. [Wed Feb 6 08:13:46 2019][1691326.372460] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Feb 6 08:13:46 2019][1691326.380468] mdt03_034 D ffff912be6b9d140 0 24923 2 0x00000080 [Wed Feb 6 08:13:46 2019][1691326.387760] Call Trace: [Wed Feb 6 08:13:46 2019][1691326.390406] [] schedule+0x29/0x70 [Wed Feb 6 08:13:46 2019][1691326.395562] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 08:13:46 2019][1691326.402194] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 08:13:46 2019][1691326.409087] [] down_write+0x2d/0x3d [Wed Feb 6 08:13:46 2019][1691326.414427] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 08:13:46 2019][1691326.421338] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 08:13:46 2019][1691326.428274] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 08:13:46 2019][1691326.435422] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:13:46 2019][1691326.441244] [] ? ldlm_cli_enqueue_local+0x27d/0x870 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.448754] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 08:13:46 2019][1691326.456780] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 08:13:46 2019][1691326.464196] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 08:13:46 2019][1691326.471446] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 08:13:46 2019][1691326.478183] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:13:46 2019][1691326.485528] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 08:13:46 2019][1691326.492170] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 08:13:46 2019][1691326.498868] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.505516] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:13:46 2019][1691326.512153] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 08:13:46 2019][1691326.518841] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.525746] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 08:13:46 2019][1691326.533088] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 08:13:46 2019][1691326.539689] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.546988] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.554743] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.561103] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.568233] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.576002] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 08:13:46 2019][1691326.583298] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.591202] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.598178] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:13:46 2019][1691326.604634] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:13:46 2019][1691326.610724] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.617248] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:13:46 2019][1691326.624832] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:46 2019][1691326.629898] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:46 2019][1691326.636181] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:46 2019][1691326.642813] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:47 2019][1691327.716679] NMI watchdog: BUG: soft lockup - CPU#28 stuck for 22s! [ldlm_bl_16:15973] [Wed Feb 6 08:13:47 2019][1691327.724681] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:13:48 2019][1691327.831450] CPU: 28 PID: 15973 Comm: ldlm_bl_16 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:13:48 2019][1691327.843955] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:13:48 2019][1691327.851696] task: ffff9144c9204100 ti: ffff91461b308000 task.ti: ffff91461b308000 [Wed Feb 6 08:13:48 2019][1691327.859347] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 08:13:48 2019][1691327.869843] RSP: 0018:ffff91461b30bbb0 EFLAGS: 00000282 [Wed Feb 6 08:13:48 2019][1691327.875327] RAX: 0000000000000101 RBX: ffff912778124f20 RCX: ffff912778124f20 [Wed Feb 6 08:13:48 2019][1691327.882635] RDX: ffff91461b30bca8 RSI: ffff9120f5ce9440 RDI: ffff913b5de82f40 [Wed Feb 6 08:13:48 2019][1691327.889940] RBP: ffff91461b30bc08 R08: ffff91461b30bca8 R09: 00000000c00016e0 [Wed Feb 6 08:13:48 2019][1691327.897248] R10: 0000000000000060 R11: ffff9120f5ce9440 R12: ffff91461b30bca8 [Wed Feb 6 08:13:48 2019][1691327.904553] R13: 00000000c00016e0 R14: 0000000000000060 R15: ffff9120f5ce9440 [Wed Feb 6 08:13:48 2019][1691327.911861] FS: 00007fbb7a00a740(0000) GS:ffff9129fefc0000(0000) knlGS:0000000000000000 [Wed Feb 6 08:13:48 2019][1691327.920120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:13:48 2019][1691327.926039] CR2: 00007fbb79a63d10 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:13:48 2019][1691327.933346] Call Trace: [Wed Feb 6 08:13:48 2019][1691327.936010] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 08:13:48 2019][1691327.943691] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 08:13:48 2019][1691327.951632] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 08:13:48 2019][1691327.958790] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 08:13:48 2019][1691327.965951] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 08:13:48 2019][1691327.974425] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:13:48 2019][1691327.982638] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:13:48 2019][1691327.989684] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:13:48 2019][1691327.995454] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:13:48 2019][1691328.003021] [] kthread+0xd1/0xe0 [Wed Feb 6 08:13:48 2019][1691328.008072] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:48 2019][1691328.014339] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:13:48 2019][1691328.020951] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:13:48 2019][1691328.027214] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 08:13:59 2019][1691339.615055] LustreError: 28990:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk READ req@ffff9128ce605050 x1624711583760624/t0(0) o256->57a7251f-bc46-8d10-a8be-5212983a5be7@10.8.17.16@o2ib6:565/0 lens 304/240 e 0 to 0 dl 1549469700 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:13:59 2019][1691339.639707] LustreError: 28990:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 4 previous similar messages [Wed Feb 6 08:14:07 2019][1691346.963915] Lustre: fir-MDT0000: haven't heard from client 46174bc9-01a4-3188-2c93-b81d9709f367 (at 10.8.4.19@o2ib6) in 167 seconds. I think it's dead, and I am evicting it. exp ffff91260d767c00, cur 1549469685 expire 1549469535 last 1549469518 [Wed Feb 6 08:14:07 2019][1691346.985703] Lustre: Skipped 7 previous similar messages [Wed Feb 6 08:14:11 2019][1691351.613280] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [ldlm_bl_24:15987] [Wed Feb 6 08:14:11 2019][1691351.621280] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:14:11 2019][1691351.728049] CPU: 16 PID: 15987 Comm: ldlm_bl_24 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:14:11 2019][1691351.740552] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:14:11 2019][1691351.748292] task: ffff9149a0fce180 ti: ffff91461b228000 task.ti: ffff91461b228000 [Wed Feb 6 08:14:12 2019][1691351.755944] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 08:14:12 2019][1691351.766441] RSP: 0018:ffff91461b22bbb0 EFLAGS: 00000282 [Wed Feb 6 08:14:12 2019][1691351.771935] RAX: 0000000000000101 RBX: ffff913b50496120 RCX: ffff913b50496120 [Wed Feb 6 08:14:12 2019][1691351.779240] RDX: ffff91461b22bca8 RSI: ffff9151f0a12400 RDI: ffff9155b4b6ad00 [Wed Feb 6 08:14:12 2019][1691351.786548] RBP: ffff91461b22bc08 R08: ffff91461b22bca8 R09: 00000000c000173e [Wed Feb 6 08:14:12 2019][1691351.793854] R10: 000000000000003e R11: ffff9151f0a12400 R12: ffff91461b22bca8 [Wed Feb 6 08:14:12 2019][1691351.801160] R13: 00000000c000173e R14: 000000000000003e R15: ffff9151f0a12400 [Wed Feb 6 08:14:12 2019][1691351.808466] FS: 00007f78f341b880(0000) GS:ffff9129fef00000(0000) knlGS:0000000000000000 [Wed Feb 6 08:14:12 2019][1691351.816725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:14:12 2019][1691351.822645] CR2: 00007f63fb343000 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:14:12 2019][1691351.829951] Call Trace: [Wed Feb 6 08:14:12 2019][1691351.832613] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 08:14:12 2019][1691351.840286] [] ? cfs_hash_bd_from_key+0x33/0xb0 [libcfs] [Wed Feb 6 08:14:12 2019][1691351.847444] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 08:14:12 2019][1691351.855380] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 08:14:12 2019][1691351.862541] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 08:14:12 2019][1691351.869699] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 08:14:12 2019][1691351.878157] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:14:12 2019][1691351.886360] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:14:12 2019][1691351.893414] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:14:12 2019][1691351.899185] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:14:12 2019][1691351.906751] [] kthread+0xd1/0xe0 [Wed Feb 6 08:14:12 2019][1691351.911802] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:14:12 2019][1691351.918072] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:14:12 2019][1691351.924691] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:14:12 2019][1691351.930954] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 08:14:15 2019][1691355.717380] NMI watchdog: BUG: soft lockup - CPU#28 stuck for 22s! [ldlm_bl_16:15973] [Wed Feb 6 08:14:15 2019][1691355.725379] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:14:16 2019][1691355.832151] CPU: 28 PID: 15973 Comm: ldlm_bl_16 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:14:16 2019][1691355.844655] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:14:16 2019][1691355.852395] task: ffff9144c9204100 ti: ffff91461b308000 task.ti: ffff91461b308000 [Wed Feb 6 08:14:16 2019][1691355.860047] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 08:14:16 2019][1691355.870542] RSP: 0018:ffff91461b30bbb0 EFLAGS: 00000282 [Wed Feb 6 08:14:16 2019][1691355.876028] RAX: 0000000000000101 RBX: ffff912778124f20 RCX: ffff912778124f20 [Wed Feb 6 08:14:16 2019][1691355.883335] RDX: ffff91461b30bca8 RSI: ffff9120f5ce9440 RDI: ffff913708e772c0 [Wed Feb 6 08:14:16 2019][1691355.890640] RBP: ffff91461b30bc08 R08: ffff91461b30bca8 R09: 00000000c00016e0 [Wed Feb 6 08:14:16 2019][1691355.897947] R10: 0000000000000060 R11: ffff9120f5ce9440 R12: ffff91461b30bca8 [Wed Feb 6 08:14:16 2019][1691355.905254] R13: 00000000c00016e0 R14: 0000000000000060 R15: ffff9120f5ce9440 [Wed Feb 6 08:14:16 2019][1691355.912562] FS: 00007fbb7a00a740(0000) GS:ffff9129fefc0000(0000) knlGS:0000000000000000 [Wed Feb 6 08:14:16 2019][1691355.920818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:14:16 2019][1691355.926740] CR2: 00007fbb79a63d10 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:14:16 2019][1691355.934044] Call Trace: [Wed Feb 6 08:14:16 2019][1691355.936712] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 08:14:16 2019][1691355.944397] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 08:14:16 2019][1691355.952332] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 08:14:16 2019][1691355.959492] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 08:14:16 2019][1691355.966649] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 08:14:16 2019][1691355.975109] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:14:16 2019][1691355.983310] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:14:16 2019][1691355.990357] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:14:16 2019][1691355.996128] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:14:16 2019][1691356.003693] [] kthread+0xd1/0xe0 [Wed Feb 6 08:14:16 2019][1691356.008745] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:14:16 2019][1691356.015014] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:14:16 2019][1691356.021624] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:14:16 2019][1691356.027889] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 08:14:21 2019][1691361.271518] INFO: rcu_sched self-detected stall on CPU[1691361.272518] INFO: rcu_sched self-detected stall on CPU { 28} (t=60001 jiffies g=134910781 c=134910780 q=459152) [Wed Feb 6 08:14:21 2019][1691361.272531] Task dump for CPU 16: [Wed Feb 6 08:14:21 2019][1691361.272533] ldlm_bl_24 R running task 0 15987 2 0x00000088 [Wed Feb 6 08:14:21 2019][1691361.272536] Call Trace: [Wed Feb 6 08:14:21 2019][1691361.272584] [] ? ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.272614] [] ? ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.272619] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:14:21 2019][1691361.272647] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.272649] [] ? kthread+0xd1/0xe0 [Wed Feb 6 08:14:21 2019][1691361.272651] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:14:21 2019][1691361.272655] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:14:21 2019][1691361.272656] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:14:21 2019][1691361.272657] Task dump for CPU 28: [Wed Feb 6 08:14:21 2019][1691361.272657] ldlm_bl_16 R running task 0 15973 2 0x00000088 [Wed Feb 6 08:14:21 2019][1691361.272659] Call Trace: [Wed Feb 6 08:14:21 2019][1691361.272660] [] sched_show_task+0xa8/0x110 [Wed Feb 6 08:14:21 2019][1691361.272663] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 08:14:21 2019][1691361.272667] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 08:14:21 2019][1691361.272668] [] rcu_check_callbacks+0x442/0x730 [Wed Feb 6 08:14:21 2019][1691361.272671] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 08:14:21 2019][1691361.272674] [] update_process_times+0x46/0x80 [Wed Feb 6 08:14:21 2019][1691361.272675] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 08:14:21 2019][1691361.272677] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 08:14:21 2019][1691361.272679] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 08:14:21 2019][1691361.272681] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 08:14:21 2019][1691361.272684] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 08:14:21 2019][1691361.272686] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 08:14:21 2019][1691361.272688] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 08:14:21 2019][1691361.272689] [] ? ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.272746] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.272774] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.272800] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.272825] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.272851] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.272876] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.272904] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.272905] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:14:21 2019][1691361.272932] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.272934] [] kthread+0xd1/0xe0 [Wed Feb 6 08:14:21 2019][1691361.272935] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:14:21 2019][1691361.272936] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:14:21 2019][1691361.272938] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:14:21 2019] [Wed Feb 6 08:14:21 2019][1691361.547291] { 16} (t=60278 jiffies g=134910781 c=134910780 q=460797) [Wed Feb 6 08:14:21 2019][1691361.554277] Task dump for CPU 16: [Wed Feb 6 08:14:21 2019][1691361.557771] ldlm_bl_24 R running task 0 15987 2 0x00000088 [Wed Feb 6 08:14:21 2019][1691361.565041] Call Trace: [Wed Feb 6 08:14:21 2019][1691361.567670] [] sched_show_task+0xa8/0x110 [Wed Feb 6 08:14:21 2019][1691361.574136] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 08:14:21 2019][1691361.579708] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 08:14:21 2019][1691361.585800] [] rcu_check_callbacks+0x442/0x730 [Wed Feb 6 08:14:21 2019][1691361.592069] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 08:14:21 2019][1691361.598334] [] update_process_times+0x46/0x80 [Wed Feb 6 08:14:21 2019][1691361.604512] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 08:14:21 2019][1691361.610431] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 08:14:21 2019][1691361.616266] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 08:14:21 2019][1691361.622532] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 08:14:21 2019][1691361.628539] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 08:14:21 2019][1691361.635237] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 08:14:21 2019][1691361.641764] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 08:14:21 2019][1691361.648114] [] ? ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.656732] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.664393] [] ? cfs_hash_bd_from_key+0x33/0xb0 [libcfs] [Wed Feb 6 08:14:21 2019][1691361.671556] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.679496] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.686658] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.693814] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.702273] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.710474] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.717520] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:14:21 2019][1691361.723295] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:14:21 2019][1691361.730860] [] kthread+0xd1/0xe0 [Wed Feb 6 08:14:21 2019][1691361.735911] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:14:21 2019][1691361.742178] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:14:21 2019][1691361.748791] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:14:21 2019][1691361.755056] Task dump for CPU 28: [Wed Feb 6 08:14:21 2019][1691361.758547] ldlm_bl_16 R running task 0 15973 2 0x00000088 [Wed Feb 6 08:14:21 2019][1691361.765818] Call Trace: [Wed Feb 6 08:14:21 2019][1691361.768475] [] ? ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:14:22 2019][1691361.776845] [] ? ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:14:22 2019][1691361.784064] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:14:22 2019][1691361.789839] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:14:22 2019][1691361.797402] [] ? kthread+0xd1/0xe0 [Wed Feb 6 08:14:22 2019][1691361.802628] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:14:22 2019][1691361.808893] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:14:22 2019][1691361.815681] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:14:24 2019][1691364.596617] LNet: Service thread pid 8981 was inactive for 312.74s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 08:14:24 2019][1691364.613641] Pid: 8981, comm: mdt02_013 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:14:24 2019][1691364.623468] Call Trace: [Wed Feb 6 08:14:24 2019][1691364.626110] [] osp_precreate_reserve+0x2e8/0x800 [osp] [Wed Feb 6 08:14:24 2019][1691364.633120] [] osp_declare_create+0x199/0x5b0 [osp] [Wed Feb 6 08:14:24 2019][1691364.639870] [] lod_sub_declare_create+0xdf/0x210 [lod] [Wed Feb 6 08:14:24 2019][1691364.646903] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [Wed Feb 6 08:14:24 2019][1691364.654251] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [Wed Feb 6 08:14:24 2019][1691364.661780] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [Wed Feb 6 08:14:24 2019][1691364.668889] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 08:14:24 2019][1691364.677100] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 08:14:24 2019][1691364.684688] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 08:14:24 2019][1691364.692107] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 08:14:24 2019][1691364.698930] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 08:14:24 2019][1691364.705651] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 08:14:24 2019][1691364.712300] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:14:24 2019][1691364.718971] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:14:24 2019][1691364.725918] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:14:24 2019][1691364.733219] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:14:24 2019][1691364.739565] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:14:24 2019][1691364.746683] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:14:25 2019][1691364.754600] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:14:25 2019][1691364.761120] [] kthread+0xd1/0xe0 [Wed Feb 6 08:14:25 2019][1691364.766231] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:14:25 2019][1691364.772883] [] 0xffffffffffffffff [Wed Feb 6 08:14:25 2019][1691364.778099] LustreError: dumping log to /tmp/lustre-log.1549469702.8981 [Wed Feb 6 08:14:47 2019][1691387.125180] LNet: Service thread pid 9015 was inactive for 302.79s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 08:14:47 2019][1691387.142217] Pid: 9015, comm: mdt01_031 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:14:47 2019][1691387.152043] Call Trace: [Wed Feb 6 08:14:47 2019][1691387.154684] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 08:14:47 2019][1691387.161606] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 08:14:47 2019][1691387.168520] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 08:14:47 2019][1691387.175438] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 08:14:47 2019][1691387.183472] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 08:14:47 2019][1691387.190908] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 08:14:47 2019][1691387.198180] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 08:14:47 2019][1691387.204912] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 08:14:47 2019][1691387.211570] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 08:14:47 2019][1691387.218228] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:14:47 2019][1691387.224880] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:14:47 2019][1691387.231815] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:14:47 2019][1691387.239105] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:14:47 2019][1691387.245439] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:14:47 2019][1691387.252556] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:14:47 2019][1691387.260450] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:14:47 2019][1691387.266959] [] kthread+0xd1/0xe0 [Wed Feb 6 08:14:47 2019][1691387.272046] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:14:47 2019][1691387.278694] [] 0xffffffffffffffff [Wed Feb 6 08:14:47 2019][1691387.283886] LustreError: dumping log to /tmp/lustre-log.1549469725.9015 [Wed Feb 6 08:14:47 2019][1691387.718183] NMI watchdog: BUG: soft lockup - CPU#28 stuck for 22s! [ldlm_bl_16:15973] [Wed Feb 6 08:14:47 2019][1691387.726189] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:14:48 2019][1691387.833018] CPU: 28 PID: 15973 Comm: ldlm_bl_16 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:14:48 2019][1691387.845523] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:14:48 2019][1691387.853261] task: ffff9144c9204100 ti: ffff91461b308000 task.ti: ffff91461b308000 [Wed Feb 6 08:14:48 2019][1691387.860914] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 08:14:48 2019][1691387.871410] RSP: 0018:ffff91461b30bbb0 EFLAGS: 00000282 [Wed Feb 6 08:14:48 2019][1691387.876897] RAX: 0000000000000101 RBX: ffff912778124f20 RCX: ffff912778124f20 [Wed Feb 6 08:14:48 2019][1691387.884203] RDX: ffff91461b30bca8 RSI: ffff9120f5ce9440 RDI: ffff911fd1ba2d00 [Wed Feb 6 08:14:48 2019][1691387.891508] RBP: ffff91461b30bc08 R08: ffff91461b30bca8 R09: 00000000c00016e0 [Wed Feb 6 08:14:48 2019][1691387.898816] R10: 0000000000000060 R11: ffff9120f5ce9440 R12: ffff91461b30bca8 [Wed Feb 6 08:14:48 2019][1691387.906121] R13: 00000000c00016e0 R14: 0000000000000060 R15: ffff9120f5ce9440 [Wed Feb 6 08:14:48 2019][1691387.913429] FS: 00007fbb7a00a740(0000) GS:ffff9129fefc0000(0000) knlGS:0000000000000000 [Wed Feb 6 08:14:48 2019][1691387.921686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:14:48 2019][1691387.927605] CR2: 00007fbb79a63d10 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:14:48 2019][1691387.934912] Call Trace: [Wed Feb 6 08:14:48 2019][1691387.937579] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 08:14:48 2019][1691387.945265] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 08:14:48 2019][1691387.953204] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 08:14:48 2019][1691387.960412] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 08:14:48 2019][1691387.967683] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 08:14:48 2019][1691387.976141] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:14:48 2019][1691387.984342] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:14:48 2019][1691387.991390] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:14:48 2019][1691387.997162] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:14:48 2019][1691388.004726] [] kthread+0xd1/0xe0 [Wed Feb 6 08:14:48 2019][1691388.009778] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:14:48 2019][1691388.016047] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:14:48 2019][1691388.022658] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:14:48 2019][1691388.028921] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 08:15:15 2019][1691415.718886] NMI watchdog: BUG: soft lockup - CPU#28 stuck for 22s! [ldlm_bl_16:15973] [Wed Feb 6 08:15:15 2019][1691415.726896] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:15:16 2019][1691415.833676] CPU: 28 PID: 15973 Comm: ldlm_bl_16 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:15:16 2019][1691415.846179] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:15:16 2019][1691415.853920] task: ffff9144c9204100 ti: ffff91461b308000 task.ti: ffff91461b308000 [Wed Feb 6 08:15:16 2019][1691415.861573] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 08:15:16 2019][1691415.872068] RSP: 0018:ffff91461b30bbb0 EFLAGS: 00000282 [Wed Feb 6 08:15:16 2019][1691415.877552] RAX: 0000000000000101 RBX: ffff912778124f20 RCX: ffff912778124f20 [Wed Feb 6 08:15:16 2019][1691415.884860] RDX: ffff91461b30bca8 RSI: ffff9120f5ce9440 RDI: ffff9135e9460b40 [Wed Feb 6 08:15:16 2019][1691415.892165] RBP: ffff91461b30bc08 R08: ffff91461b30bca8 R09: 00000000c00016e0 [Wed Feb 6 08:15:16 2019][1691415.899472] R10: 0000000000000060 R11: ffff9120f5ce9440 R12: ffff91461b30bca8 [Wed Feb 6 08:15:16 2019][1691415.906778] R13: 00000000c00016e0 R14: 0000000000000060 R15: ffff9120f5ce9440 [Wed Feb 6 08:15:16 2019][1691415.914086] FS: 00007fbb7a00a740(0000) GS:ffff9129fefc0000(0000) knlGS:0000000000000000 [Wed Feb 6 08:15:16 2019][1691415.922343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:15:16 2019][1691415.928263] CR2: 00007fbb79a63d10 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:15:16 2019][1691415.935570] Call Trace: [Wed Feb 6 08:15:16 2019][1691415.938243] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 08:15:16 2019][1691415.945899] [] ? _raw_write_lock+0x10/0x20 [Wed Feb 6 08:15:16 2019][1691415.951851] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 08:15:16 2019][1691415.959794] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 08:15:16 2019][1691415.966954] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 08:15:16 2019][1691415.974111] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 08:15:16 2019][1691415.982572] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:15:16 2019][1691415.990771] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:15:16 2019][1691415.997818] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:15:16 2019][1691416.003589] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:15:16 2019][1691416.011155] [] kthread+0xd1/0xe0 [Wed Feb 6 08:15:16 2019][1691416.016208] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:15:16 2019][1691416.022474] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:15:16 2019][1691416.029087] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:15:16 2019][1691416.035349] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 08:15:23 2019][1691422.966199] LNet: Service thread pid 15228 was inactive for 302.37s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 08:15:23 2019][1691422.983318] Pid: 15228, comm: mdt00_036 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:15:23 2019][1691422.993241] Call Trace: [Wed Feb 6 08:15:23 2019][1691422.995869] [] 0xffffffffffffffff [Wed Feb 6 08:15:23 2019][1691423.001079] LustreError: dumping log to /tmp/lustre-log.1549469761.15228 [Wed Feb 6 08:15:43 2019][1691443.719589] NMI watchdog: BUG: soft lockup - CPU#28 stuck for 22s! [ldlm_bl_16:15973] [Wed Feb 6 08:15:43 2019][1691443.727589] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 08:15:44 2019][1691443.834356] CPU: 28 PID: 15973 Comm: ldlm_bl_16 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 08:15:44 2019][1691443.846863] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 08:15:44 2019][1691443.854601] task: ffff9144c9204100 ti: ffff91461b308000 task.ti: ffff91461b308000 [Wed Feb 6 08:15:44 2019][1691443.862253] RIP: 0010:[] [] ldlm_inodebits_compat_queue+0x188/0x440 [ptlrpc] [Wed Feb 6 08:15:44 2019][1691443.872749] RSP: 0018:ffff91461b30bbb0 EFLAGS: 00000282 [Wed Feb 6 08:15:44 2019][1691443.878236] RAX: 0000000000000101 RBX: ffff912778124f20 RCX: ffff912778124f20 [Wed Feb 6 08:15:44 2019][1691443.885541] RDX: ffff91461b30bca8 RSI: ffff9120f5ce9440 RDI: ffff913ba3e860c0 [Wed Feb 6 08:15:44 2019][1691443.892848] RBP: ffff91461b30bc08 R08: ffff91461b30bca8 R09: 00000000c00016e0 [Wed Feb 6 08:15:44 2019][1691443.900155] R10: 0000000000000060 R11: ffff9120f5ce9440 R12: ffff91461b30bca8 [Wed Feb 6 08:15:44 2019][1691443.907460] R13: 00000000c00016e0 R14: 0000000000000060 R15: ffff9120f5ce9440 [Wed Feb 6 08:15:44 2019][1691443.914767] FS: 00007fbb7a00a740(0000) GS:ffff9129fefc0000(0000) knlGS:0000000000000000 [Wed Feb 6 08:15:44 2019][1691443.923026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 08:15:44 2019][1691443.928946] CR2: 00007fbb79a63d10 CR3: 000000402c810000 CR4: 00000000003407e0 [Wed Feb 6 08:15:44 2019][1691443.936251] Call Trace: [Wed Feb 6 08:15:44 2019][1691443.938921] [] ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 08:15:44 2019][1691443.946607] [] ? ldlm_export_lock_object+0x10/0x10 [ptlrpc] [Wed Feb 6 08:15:44 2019][1691443.954031] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 08:15:44 2019][1691443.961968] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [Wed Feb 6 08:15:44 2019][1691443.969127] [] __ldlm_reprocess_all+0x10b/0x380 [ptlrpc] [Wed Feb 6 08:15:44 2019][1691443.976284] [] ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 08:15:44 2019][1691443.984744] [] ldlm_export_cancel_blocked_locks+0x121/0x200 [ptlrpc] [Wed Feb 6 08:15:44 2019][1691443.992945] [] ldlm_bl_thread_main+0x112/0x700 [ptlrpc] [Wed Feb 6 08:15:44 2019][1691443.999992] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:15:44 2019][1691444.005774] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 08:15:44 2019][1691444.013346] [] kthread+0xd1/0xe0 [Wed Feb 6 08:15:44 2019][1691444.018399] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:15:44 2019][1691444.024666] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:15:44 2019][1691444.031276] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:15:44 2019][1691444.037540] Code: 74 0e 4c 89 f2 48 89 de 4c 89 ff e8 d3 a2 fc ff 49 8b 87 10 02 00 00 4c 8d a8 f0 fd ff ff 4d 39 fd 74 2b 49 83 bd a8 00 00 00 00 <74> 0e 4c 89 f2 48 89 de 4c 89 ef e8 a8 a2 fc ff 4d 8b ad 10 02 [Wed Feb 6 08:15:46 2019][1691446.651871] INFO: task kworker/41:2:65939 blocked for more than 120 seconds. [Wed Feb 6 08:15:46 2019][1691446.659112] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Feb 6 08:15:46 2019][1691446.667120] kworker/41:2 D ffff9139f83ad140 0 65939 2 0x00000080 [Wed Feb 6 08:15:46 2019][1691446.674421] Workqueue: events key_garbage_collector [Wed Feb 6 08:15:46 2019][1691446.679500] Call Trace: [Wed Feb 6 08:15:46 2019][1691446.682137] [] schedule+0x29/0x70 [Wed Feb 6 08:15:46 2019][1691446.687289] [] schedule_timeout+0x221/0x2d0 [Wed Feb 6 08:15:46 2019][1691446.693304] [] ? cpumask_next_and+0x35/0x50 [Wed Feb 6 08:15:46 2019][1691446.699321] [] ? load_balance+0x1be/0x9a0 [Wed Feb 6 08:15:46 2019][1691446.705161] [] wait_for_completion+0xfd/0x140 [Wed Feb 6 08:15:46 2019][1691446.711366] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:15:46 2019][1691446.717121] [] ? __call_rcu+0x2c0/0x2c0 [Wed Feb 6 08:15:46 2019][1691446.722806] [] wait_rcu_gp+0x5e/0x80 [Wed Feb 6 08:15:46 2019][1691446.728221] [] ? ftrace_raw_output_rcu_utilization+0x70/0x70 [Wed Feb 6 08:15:46 2019][1691446.735747] [] synchronize_sched+0x3b/0x50 [Wed Feb 6 08:15:46 2019][1691446.741681] [] key_garbage_collector+0x1c5/0x390 [Wed Feb 6 08:15:46 2019][1691446.748158] [] process_one_work+0x17f/0x440 [Wed Feb 6 08:15:47 2019][1691446.754173] [] worker_thread+0x22c/0x3c0 [Wed Feb 6 08:15:47 2019][1691446.759926] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 08:15:47 2019][1691446.766648] [] kthread+0xd1/0xe0 [Wed Feb 6 08:15:47 2019][1691446.771713] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:15:47 2019][1691446.778002] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:15:47 2019][1691446.784643] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:15:47 2019][1691446.790992] INFO: task systemd-logind:28626 blocked for more than 120 seconds. [Wed Feb 6 08:15:47 2019][1691446.798388] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Feb 6 08:15:47 2019][1691446.806423] systemd-logind D ffff914487471040 0 28626 1 0x00000084 [Wed Feb 6 08:15:47 2019][1691446.813713] Call Trace: [Wed Feb 6 08:15:47 2019][1691446.816365] [] ? ttwu_do_activate+0x6f/0x80 [Wed Feb 6 08:15:47 2019][1691446.822398] [] schedule+0x29/0x70 [Wed Feb 6 08:15:47 2019][1691446.827592] [] schedule_timeout+0x221/0x2d0 [Wed Feb 6 08:15:47 2019][1691446.833614] [] ? __slab_free+0x81/0x2f0 [Wed Feb 6 08:15:47 2019][1691446.839287] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:15:47 2019][1691446.845308] [] wait_for_completion+0xfd/0x140 [Wed Feb 6 08:15:47 2019][1691446.851503] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:15:47 2019][1691446.857270] [] ? __call_rcu+0x2c0/0x2c0 [Wed Feb 6 08:15:47 2019][1691446.862965] [] wait_rcu_gp+0x5e/0x80 [Wed Feb 6 08:15:47 2019][1691446.868376] [] ? ftrace_raw_output_rcu_utilization+0x70/0x70 [Wed Feb 6 08:15:47 2019][1691446.875900] [] ? umount_tree+0x184/0x2a0 [Wed Feb 6 08:15:47 2019][1691446.881670] [] synchronize_sched+0x3b/0x50 [Wed Feb 6 08:15:47 2019][1691446.887598] [] namespace_unlock+0x68/0x80 [Wed Feb 6 08:15:47 2019][1691446.893455] [] SyS_umount+0x239/0x410 [Wed Feb 6 08:15:47 2019][1691446.898952] [] ? ____fput+0xe/0x10 [Wed Feb 6 08:15:47 2019][1691446.904189] [] ? task_work_run+0xc0/0xe0 [Wed Feb 6 08:15:47 2019][1691446.909944] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 08:16:01 2019][1691460.999942] Lustre: MGS: haven't heard from client 875bd367-4e39-43fd-bd4a-a99cc08dc44b (at 10.8.26.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9156a57b5000, cur 1549469799 expire 1549469649 last 1549469572 [Wed Feb 6 08:16:01 2019][1691461.021131] Lustre: Skipped 26 previous similar messages [Wed Feb 6 08:16:03 2019][1691462.903084] LNet: Service thread pid 6687 was inactive for 292.05s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 08:16:03 2019][1691462.920146] Pid: 6687, comm: mdt02_004 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:16:03 2019][1691462.929986] Call Trace: [Wed Feb 6 08:16:03 2019][1691462.932635] [] osp_precreate_reserve+0x2e8/0x800 [osp] [Wed Feb 6 08:16:03 2019][1691462.939641] [] osp_declare_create+0x199/0x5b0 [osp] [Wed Feb 6 08:16:03 2019][1691462.946383] [] lod_sub_declare_create+0xdf/0x210 [lod] [Wed Feb 6 08:16:03 2019][1691462.953454] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [Wed Feb 6 08:16:03 2019][1691462.960878] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [Wed Feb 6 08:16:03 2019][1691462.968469] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [Wed Feb 6 08:16:03 2019][1691462.975497] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 08:16:03 2019][1691462.983565] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 08:16:03 2019][1691462.991017] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 08:16:03 2019][1691462.998280] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 08:16:03 2019][1691463.005014] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 08:16:03 2019][1691463.011687] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 08:16:03 2019][1691463.018354] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:16:03 2019][1691463.025009] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:16:03 2019][1691463.031963] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:16:03 2019][1691463.039266] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:16:03 2019][1691463.045619] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:16:03 2019][1691463.052761] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:16:03 2019][1691463.060675] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:16:03 2019][1691463.067210] [] kthread+0xd1/0xe0 [Wed Feb 6 08:16:03 2019][1691463.072305] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:16:03 2019][1691463.078952] [] 0xffffffffffffffff [Wed Feb 6 08:16:03 2019][1691463.084171] LustreError: dumping log to /tmp/lustre-log.1549469801.6687 [Wed Feb 6 08:16:11 2019][1691470.777288] LustreError: 15163:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549469508, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff912e93780240/0x21ed9351af886614 lrc: 3/0,1 mode: --/PW res: [0x2c0001697:0x86:0x0].0x0 bits 0x40/0x0 rrc: 378998 type: IBT flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 15163 timeout: 0 lvb_type: 0 [Wed Feb 6 08:16:11 2019][1691470.817197] LustreError: 15163:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 4 previous similar messages [Wed Feb 6 08:16:32 2019][1691492.288826] LustreError: 9184:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.113.15@tcp) failed to reply to blocking AST (req@ffff913a8f7b6600 x1623682096646208 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff9125b48e5580/0x21ed9351b58aee3d lrc: 4/0,0 mode: PR/PR res: [0x2c0001712:0x3:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e19945 expref: 13 pid: 6616 timeout: 1691598 lvb_type: 0 [Wed Feb 6 08:16:32 2019][1691492.331329] LustreError: 9184:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 5 previous similar messages [Wed Feb 6 08:16:32 2019][1691492.341631] LustreError: 9184:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff91265e098800 ns: mdt-fir-MDT0002_UUID lock: ffff913d568f8fc0/0x21ed9351b58aee83 lrc: 1/0,0 mode: --/PW res: [0x2c0001712:0x3:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x54a01000000000 nid: 10.10.113.15@tcp remote: 0xdf1b55af52e19953 expref: 3 pid: 9184 timeout: 0 lvb_type: 0 [Wed Feb 6 08:16:32 2019][1691492.375909] LustreError: 9184:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 2 previous similar messages [Wed Feb 6 08:16:50 2019][1691510.584300] Lustre: 17053:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-10), not sending early reply [Wed Feb 6 08:16:50 2019][1691510.584300] req@ffff9150b73bc800 x1624698083299920/t13605490973(0) o36->5fda523c-9891-cc11-7a0e-50a252e6fb83@10.9.103.8@o2ib4:718/0 lens 496/2888 e 4 to 0 dl 1549469853 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:16:53 2019][1691513.211031] LNet: Service thread pid 6687 completed after 342.36s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 08:16:53 2019][1691513.227275] LNet: Skipped 8 previous similar messages [Wed Feb 6 08:17:12 2019][1691532.522834] LustreError: 15184:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549469570, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff913ad0374380/0x21ed9351afc38d83 lrc: 3/0,1 mode: --/PW res: [0x2c0001702:0x51:0x0].0x0 bits 0x40/0x0 rrc: 422129 type: IBT flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 15184 timeout: 0 lvb_type: 0 [Wed Feb 6 08:17:51 2019][1691570.951833] Lustre: MGS: haven't heard from client ee4fa216-af85-5e91-1d49-785792c1542a (at 10.8.22.5@o2ib6) in 257 seconds. I think it's dead, and I am evicting it. exp ffff912da8012400, cur 1549469909 expire 1549469759 last 1549469652 [Wed Feb 6 08:17:51 2019][1691570.972938] Lustre: Skipped 6 previous similar messages [Wed Feb 6 08:18:13 2019][1691593.451346] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 08:18:13 2019][1691593.462039] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 84657539 previous similar messages [Wed Feb 6 08:18:21 2019][1691601.473553] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549469939/real 1549469939] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549469950 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 08:18:21 2019][1691601.501406] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 84499021 previous similar messages [Wed Feb 6 08:18:35 2019][1691615.291916] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 134s: evicting client at 10.9.102.39@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff9149166aec00/0x21ed9351b55a9166 lrc: 3/0,0 mode: PR/PR res: [0x2c000169a:0x46:0x0].0x0 bits 0x40/0x0 rrc: 455875 type: IBT flags: 0x60000400010020 nid: 10.9.102.39@o2ib4 remote: 0x8de480ae7403dd88 expref: 455966 pid: 15265 timeout: 1691572 lvb_type: 0 [Wed Feb 6 08:18:35 2019][1691615.330610] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 736640 previous similar messages [Wed Feb 6 08:19:23 2019][1691663.422115] LustreError: 6178:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549469701, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff9154e96df980/0x21ed9351b59ab035 lrc: 3/1,0 mode: --/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 6178 timeout: 0 lvb_type: 0 [Wed Feb 6 08:19:25 2019][1691665.394052] Lustre: fir-MDT0000: Client 46174bc9-01a4-3188-2c93-b81d9709f367 (at 10.8.4.19@o2ib6) reconnecting [Wed Feb 6 08:19:25 2019][1691665.404222] Lustre: Skipped 2110 previous similar messages [Wed Feb 6 08:19:39 2019][1691679.292524] Lustre: 9017:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 08:19:39 2019][1691679.292524] req@ffff912a4bae4800 x1624736386787408/t0(0) o101->045f5984-f159-09cd-da98-0d87730fa119@10.8.4.16@o2ib6:132/0 lens 376/1600 e 9 to 0 dl 1549470022 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:19:47 2019][1691686.921862] INFO: task mdt01_031:9015 blocked for more than 120 seconds. [Wed Feb 6 08:19:47 2019][1691686.928736] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Feb 6 08:19:47 2019][1691686.936740] mdt01_031 D ffff9159a4ead140 0 9015 2 0x00000080 [Wed Feb 6 08:19:47 2019][1691686.944014] Call Trace: [Wed Feb 6 08:19:47 2019][1691686.946659] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [Wed Feb 6 08:19:47 2019][1691686.953797] [] schedule+0x29/0x70 [Wed Feb 6 08:19:47 2019][1691686.958947] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 08:19:47 2019][1691686.965568] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 08:19:47 2019][1691686.972447] [] down_write+0x2d/0x3d [Wed Feb 6 08:19:47 2019][1691686.977776] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 08:19:47 2019][1691686.984665] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 08:19:47 2019][1691686.991578] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 08:19:47 2019][1691686.998716] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 08:19:47 2019][1691687.004476] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 08:19:47 2019][1691687.012485] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 08:19:47 2019][1691687.019894] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 08:19:47 2019][1691687.027121] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 08:19:47 2019][1691687.033838] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:19:47 2019][1691687.041156] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 08:19:47 2019][1691687.047788] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 08:19:47 2019][1691687.054456] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 08:19:47 2019][1691687.061093] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:19:47 2019][1691687.067721] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 08:19:47 2019][1691687.074371] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:19:47 2019][1691687.081262] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 08:19:47 2019][1691687.088580] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 08:19:47 2019][1691687.095147] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:19:47 2019][1691687.102405] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 08:19:47 2019][1691687.110110] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:19:47 2019][1691687.116430] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:19:47 2019][1691687.123518] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 08:19:47 2019][1691687.131275] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 08:19:47 2019][1691687.138536] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:19:47 2019][1691687.146404] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 08:19:47 2019][1691687.153368] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 08:19:47 2019][1691687.159818] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 08:19:47 2019][1691687.165865] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:19:47 2019][1691687.172345] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 08:19:47 2019][1691687.179924] [] kthread+0xd1/0xe0 [Wed Feb 6 08:19:47 2019][1691687.184987] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:19:47 2019][1691687.191260] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:19:47 2019][1691687.197880] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 08:19:48 2019][1691688.136736] LustreError: 6170:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549469726, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff91300dd38fc0/0x21ed9351b5b01c1b lrc: 3/1,0 mode: --/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 8 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 6170 timeout: 0 lvb_type: 0 [Wed Feb 6 08:20:37 2019][1691737.224353] LustreError: 17050:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff911f7a27a700 x1623682098185696/t0(0) o104->fir-MDT0002@10.9.103.8@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 08:20:37 2019][1691737.245530] LustreError: 17050:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 315994 previous similar messages [Wed Feb 6 08:21:12 2019][1691772.447591] Lustre: MGS: Connection restored to ba6e9243-f5e7-a9ca-10a1-87280ba6b45d (at 10.8.8.20@o2ib6) [Wed Feb 6 08:21:12 2019][1691772.457329] Lustre: Skipped 4087 previous similar messages [Wed Feb 6 08:21:32 2019][1691792.348351] LustreError: 17204:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549469830, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff91588430e9c0/0x21ed9351b5924267 lrc: 3/0,1 mode: --/CW res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x2/0x0 rrc: 9 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 17204 timeout: 0 lvb_type: 0 [Wed Feb 6 08:23:37 2019][1691917.570490] LNet: Service thread pid 17204 was inactive for 566.46s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 08:23:37 2019][1691917.587606] Pid: 17204, comm: mdt03_019 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:23:37 2019][1691917.597521] Call Trace: [Wed Feb 6 08:23:37 2019][1691917.600163] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 08:23:37 2019][1691917.607287] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 08:23:37 2019][1691917.614689] [] mdt_object_local_lock+0x438/0xb20 [mdt] [Wed Feb 6 08:23:37 2019][1691917.621701] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:23:37 2019][1691917.628894] [] mdt_object_lock+0x20/0x30 [mdt] [Wed Feb 6 08:23:37 2019][1691917.635202] [] mdt_create+0x569/0x1090 [mdt] [Wed Feb 6 08:23:37 2019][1691917.641337] [] mdt_reint_create+0x16b/0x360 [mdt] [Wed Feb 6 08:23:37 2019][1691917.647907] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 08:23:37 2019][1691917.654130] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 08:23:37 2019][1691917.660873] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 08:23:37 2019][1691917.666755] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:23:37 2019][1691917.673899] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:23:37 2019][1691917.681787] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:23:37 2019][1691917.688294] [] kthread+0xd1/0xe0 [Wed Feb 6 08:23:37 2019][1691917.693383] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:23:37 2019][1691917.700032] [] 0xffffffffffffffff [Wed Feb 6 08:23:37 2019][1691917.705233] LustreError: dumping log to /tmp/lustre-log.1549470255.17204 [Wed Feb 6 08:23:50 2019][1691929.858798] Pid: 6178, comm: mdt03_002 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:23:50 2019][1691929.868626] Call Trace: [Wed Feb 6 08:23:50 2019][1691929.871268] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 08:23:50 2019][1691929.878381] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 08:23:50 2019][1691929.885758] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 08:23:50 2019][1691929.892760] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:23:50 2019][1691929.899946] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 08:23:50 2019][1691929.907035] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 08:23:50 2019][1691929.913777] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:23:50 2019][1691929.920444] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:23:50 2019][1691929.927384] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:23:50 2019][1691929.934666] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:23:50 2019][1691929.941016] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:23:50 2019][1691929.948133] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:23:50 2019][1691929.956037] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:23:50 2019][1691929.962547] [] kthread+0xd1/0xe0 [Wed Feb 6 08:23:50 2019][1691929.967634] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:23:50 2019][1691929.974281] [] 0xffffffffffffffff [Wed Feb 6 08:23:50 2019][1691929.979474] LustreError: dumping log to /tmp/lustre-log.1549470268.6178 [Wed Feb 6 08:23:58 2019][1691938.257029] Lustre: 9030:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 08:23:58 2019][1691938.257029] req@ffff913a920bf800 x1624669511939440/t0(0) o101->4944a2ce-d92c-784e-9536-c95b01530191@10.9.101.48@o2ib4:391/0 lens 376/1600 e 10 to 0 dl 1549470281 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:24:06 2019][1691946.243211] LNet: Service thread pid 6170 was inactive for 558.09s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 08:24:06 2019][1691946.260234] LNet: Skipped 1 previous similar message [Wed Feb 6 08:24:06 2019][1691946.265383] Pid: 6170, comm: mdt01_000 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:24:06 2019][1691946.275207] Call Trace: [Wed Feb 6 08:24:06 2019][1691946.277841] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 08:24:06 2019][1691946.284956] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 08:24:06 2019][1691946.292321] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 08:24:06 2019][1691946.299325] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:24:06 2019][1691946.306502] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 08:24:06 2019][1691946.313590] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 08:24:06 2019][1691946.320333] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:24:06 2019][1691946.326991] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:24:06 2019][1691946.333924] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:24:06 2019][1691946.341203] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:24:06 2019][1691946.347540] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:24:06 2019][1691946.354656] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:24:06 2019][1691946.362543] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:24:06 2019][1691946.369043] [] kthread+0xd1/0xe0 [Wed Feb 6 08:24:06 2019][1691946.374129] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:24:06 2019][1691946.380768] [] 0xffffffffffffffff [Wed Feb 6 08:24:06 2019][1691946.385962] LustreError: dumping log to /tmp/lustre-log.1549470284.6170 [Wed Feb 6 08:24:06 2019][1691946.491224] Lustre: 17053:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 08:24:06 2019][1691946.491224] req@ffff9153e3a98900 x1624698083406176/t0(0) o36->5fda523c-9891-cc11-7a0e-50a252e6fb83@10.9.103.8@o2ib4:399/0 lens 512/448 e 1 to 0 dl 1549470289 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:24:18 2019][1691958.765533] Lustre: 17203:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 08:24:19 2019][1691958.765533] req@ffff9153ab67c200 x1624697932638640/t0(0) o101->2e6063a2-ca9b-ae5d-abce-e3daf4d673e4@10.9.103.23@o2ib4:411/0 lens 592/3264 e 1 to 0 dl 1549470301 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:24:27 2019][1691967.007743] LustreError: 15294:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549470005, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff91386170c800/0x21ed9351b69aed19 lrc: 3/1,0 mode: --/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 12 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 15294 timeout: 0 lvb_type: 0 [Wed Feb 6 08:24:27 2019][1691967.047482] LustreError: 15294:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Feb 6 08:24:43 2019][1691983.620165] Lustre: 15170:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 08:24:43 2019][1691983.620165] req@ffff9139fe94b900 x1624700284400688/t0(0) o101->77116eb9-21ab-2adf-64d6-a8cbcc3d83d7@10.9.102.39@o2ib4:436/0 lens 592/3264 e 1 to 0 dl 1549470326 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:24:45 2019][1691985.156191] LNet: Service thread pid 15511 was inactive for 641.80s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 08:24:45 2019][1691985.173343] Pid: 15511, comm: mdt02_049 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:24:45 2019][1691985.183255] Call Trace: [Wed Feb 6 08:24:45 2019][1691985.185896] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 08:24:45 2019][1691985.192814] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 08:24:45 2019][1691985.199732] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 08:24:45 2019][1691985.206639] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 08:24:45 2019][1691985.214683] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 08:24:45 2019][1691985.222117] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 08:24:45 2019][1691985.229384] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 08:24:45 2019][1691985.236125] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 08:24:45 2019][1691985.242779] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 08:24:45 2019][1691985.249440] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:24:45 2019][1691985.256100] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:24:45 2019][1691985.263044] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:24:45 2019][1691985.270342] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:24:45 2019][1691985.276696] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:24:45 2019][1691985.283838] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:24:45 2019][1691985.291753] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:24:45 2019][1691985.298293] [] kthread+0xd1/0xe0 [Wed Feb 6 08:24:45 2019][1691985.303379] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:24:45 2019][1691985.310030] [] 0xffffffffffffffff [Wed Feb 6 08:24:45 2019][1691985.315232] LustreError: dumping log to /tmp/lustre-log.1549470323.15511 [Wed Feb 6 08:25:48 2019][1692048.645786] Pid: 9174, comm: mdt00_023 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:25:48 2019][1692048.655613] Call Trace: [Wed Feb 6 08:25:48 2019][1692048.658258] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 08:25:48 2019][1692048.665171] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 08:25:48 2019][1692048.672100] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 08:25:48 2019][1692048.679019] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 08:25:48 2019][1692048.687079] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 08:25:48 2019][1692048.694522] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 08:25:48 2019][1692048.701804] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 08:25:48 2019][1692048.708557] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 08:25:48 2019][1692048.715228] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 08:25:48 2019][1692048.721884] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:25:48 2019][1692048.728537] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:25:48 2019][1692048.735481] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:25:48 2019][1692048.742773] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:25:48 2019][1692048.749123] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:25:48 2019][1692048.756246] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:25:49 2019][1692048.764153] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:25:49 2019][1692048.770686] [] kthread+0xd1/0xe0 [Wed Feb 6 08:25:49 2019][1692048.775785] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:25:49 2019][1692048.782438] [] 0xffffffffffffffff [Wed Feb 6 08:25:49 2019][1692048.787641] LustreError: dumping log to /tmp/lustre-log.1549470386.9174 [Wed Feb 6 08:25:49 2019][1692048.833675] LNet: Service thread pid 15223 was inactive for 556.29s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 08:25:53 2019][1692053.061911] Lustre: 48636:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 08:25:53 2019][1692053.061911] req@ffff913b22ec2400 x1624669508108736/t0(0) o101->e2d946d5-12d9-285b-e4ca-825370f3d1df@10.9.101.22@o2ib4:506/0 lens 376/1600 e 3 to 0 dl 1549470396 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:26:27 2019][1692087.302764] Lustre: 15170:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 08:26:27 2019][1692087.302764] req@ffff913521730c00 x1624701937500576/t0(0) o101->95f41e85-91f0-6aee-d80e-dc57a1b83666@10.10.113.15@tcp:540/0 lens 592/3264 e 1 to 0 dl 1549470430 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:26:30 2019][1692090.118822] LustreError: dumping log to /tmp/lustre-log.1549470428.15220 [Wed Feb 6 08:27:12 2019][1692132.096879] LustreError: 8573:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549470170, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff91511b2a3180/0x21ed9351b70914df lrc: 3/1,0 mode: --/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 13 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 8573 timeout: 0 lvb_type: 0 [Wed Feb 6 08:27:12 2019][1692132.136437] LustreError: 8573:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Feb 6 08:27:31 2019][1692151.048356] LNet: Service thread pid 48676 was inactive for 692.48s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 08:27:31 2019][1692151.061391] LNet: Skipped 1 previous similar message [Wed Feb 6 08:27:31 2019][1692151.066542] LustreError: dumping log to /tmp/lustre-log.1549470489.48676 [Wed Feb 6 08:28:13 2019][1692193.466412] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 08:28:13 2019][1692193.477111] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 83924391 previous similar messages [Wed Feb 6 08:28:21 2019][1692201.488607] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549470539/real 1549470539] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549470550 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 08:28:21 2019][1692201.516458] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 83925577 previous similar messages [Wed Feb 6 08:28:26 2019][1692205.833739] Lustre: 15181:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 08:28:26 2019][1692205.833739] req@ffff9156cd7ed700 x1624673497897456/t0(0) o101->1bdafb8f-098e-e990-f183-dce8ce68db0c@10.9.102.5@o2ib4:658/0 lens 376/1600 e 1 to 0 dl 1549470548 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:28:26 2019][1692205.862908] Lustre: 15181:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Feb 6 08:28:33 2019][1692213.228878] LNet: Service thread pid 9174 completed after 699.36s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 08:28:33 2019][1692213.228897] LustreError: 8981:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff91260d767c00 ns: mdt-fir-MDT0000_UUID lock: ffff914246735a00/0x21ed9351b54a5702 lrc: 1/0,0 mode: --/EX res: [0x200001870:0x1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x54801000000000 nid: 10.8.4.19@o2ib6 remote: 0xe23f7d714d1ab432 expref: 3 pid: 8981 timeout: 0 lvb_type: 3 [Wed Feb 6 08:28:33 2019][1692213.228934] Lustre: 8981:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (320:842s); client may timeout. req@ffff913dacc12d00 x1624736313391904/t14369072893(0) o101->46174bc9-01a4-3188-2c93-b81d9709f367@10.8.4.19@o2ib6:574/0 lens 376/1568 e 5 to 0 dl 1549469709 ref 1 fl Complete:/0/0 rc -107/-107 [Wed Feb 6 08:28:33 2019][1692213.309354] LNet: Skipped 1 previous similar message [Wed Feb 6 08:28:35 2019][1692215.306983] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 225s: evicting client at 10.9.102.39@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff915897aad580/0x21ed9351b54a8bf9 lrc: 3/0,0 mode: PR/PR res: [0x2c000169a:0x46:0x0].0x0 bits 0x40/0x0 rrc: 455294 type: IBT flags: 0x60000400010020 nid: 10.9.102.39@o2ib4 remote: 0x8de480ae74026946 expref: 455385 pid: 6758 timeout: 1692082 lvb_type: 0 [Wed Feb 6 08:28:35 2019][1692215.345585] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 15265 previous similar messages [Wed Feb 6 08:29:28 2019][1692268.075420] Lustre: fir-MDT0002: Client 54802b78-b8a3-6ef2-1c4c-1f299e55d7f1 (at 10.9.103.14@o2ib4) reconnecting [Wed Feb 6 08:29:28 2019][1692268.085768] Lustre: Skipped 10 previous similar messages [Wed Feb 6 08:30:04 2019][1692304.652204] LNet: Service thread pid 8985 was inactive for 535.26s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 08:30:04 2019][1692304.669226] LNet: Skipped 1 previous similar message [Wed Feb 6 08:30:04 2019][1692304.674375] Pid: 8985, comm: mdt00_016 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:30:04 2019][1692304.684200] Call Trace: [Wed Feb 6 08:30:04 2019][1692304.686836] [] osp_precreate_reserve+0x2e8/0x800 [osp] [Wed Feb 6 08:30:04 2019][1692304.693835] [] osp_declare_create+0x199/0x5b0 [osp] [Wed Feb 6 08:30:04 2019][1692304.700571] [] lod_sub_declare_create+0xdf/0x210 [lod] [Wed Feb 6 08:30:04 2019][1692304.707573] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [Wed Feb 6 08:30:04 2019][1692304.714827] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [Wed Feb 6 08:30:04 2019][1692304.722255] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [Wed Feb 6 08:30:04 2019][1692304.729249] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 08:30:04 2019][1692304.737284] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 08:30:04 2019][1692304.744711] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 08:30:04 2019][1692304.751965] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 08:30:04 2019][1692304.758699] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 08:30:04 2019][1692304.765355] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 08:30:05 2019][1692304.772012] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:30:05 2019][1692304.778659] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:30:05 2019][1692304.785603] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:30:05 2019][1692304.792882] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:30:05 2019][1692304.799225] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:30:05 2019][1692304.806333] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:30:05 2019][1692304.814220] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:30:05 2019][1692304.820720] [] kthread+0xd1/0xe0 [Wed Feb 6 08:30:05 2019][1692304.825808] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:30:05 2019][1692304.832454] [] 0xffffffffffffffff [Wed Feb 6 08:30:05 2019][1692304.837649] LustreError: dumping log to /tmp/lustre-log.1549470642.8985 [Wed Feb 6 08:30:37 2019][1692337.252912] LustreError: 15228:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff912142ab6900 x1623682100961888/t0(0) o104->fir-MDT0002@10.9.102.39@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 08:30:37 2019][1692337.274180] LustreError: 15228:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 49533 previous similar messages [Wed Feb 6 08:31:04 2019][1692364.429722] Lustre: 9182:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 08:31:04 2019][1692364.429722] req@ffff9126a829cb00 x1624705431067312/t0(0) o101->cd2fc9d9-2801-0b42-234d-619ca30bf3c7@10.8.9.8@o2ib6:62/0 lens 376/1600 e 1 to 0 dl 1549470707 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:31:04 2019][1692364.458531] Lustre: 9182:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Feb 6 08:31:05 2019][1692365.310735] LustreError: 9174:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.10.19.3@tcp) failed to reply to blocking AST (req@ffff911fe2ba0f00 x1623682100490304 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff9135db853cc0/0x21ed9351b7e4e465 lrc: 4/0,0 mode: PR/PR res: [0x20000177b:0xf:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.10.19.3@tcp remote: 0x136f0f14d5cb6954 expref: 21 pid: 15393 timeout: 1692471 lvb_type: 0 [Wed Feb 6 08:31:05 2019][1692365.352977] LustreError: 138-a: fir-MDT0000: A client on nid 10.10.19.3@tcp was evicted due to a lock blocking callback time out: rc -110 [Wed Feb 6 08:31:05 2019][1692365.365493] LustreError: Skipped 5 previous similar messages [Wed Feb 6 08:31:06 2019][1692366.000748] LustreError: 15279:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549470404, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff9125ba3306c0/0x21ed9351b7b1d8a0 lrc: 3/1,0 mode: --/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 14 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 15279 timeout: 0 lvb_type: 0 [Wed Feb 6 08:31:06 2019][1692366.040477] LustreError: 15279:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages [Wed Feb 6 08:31:27 2019][1692387.227316] Lustre: fir-MDT0000: Connection restored to a01a761e-2002-7435-92ff-dd4ad44b3544 (at 10.10.19.3@tcp) [Wed Feb 6 08:31:27 2019][1692387.237662] Lustre: Skipped 139 previous similar messages [Wed Feb 6 08:31:53 2019][1692413.551350] LNet: Service thread pid 15511 completed after 1070.19s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 08:31:53 2019][1692413.567897] LNet: Skipped 1 previous similar message [Wed Feb 6 08:33:33 2019][1692513.570790] LNet: Service thread pid 9015 completed after 1429.21s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 08:33:50 2019][1692529.937868] LNet: Service thread pid 9020 was inactive for 805.14s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 08:33:50 2019][1692529.954915] Pid: 9020, comm: mdt02_018 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:33:50 2019][1692529.964760] Call Trace: [Wed Feb 6 08:33:50 2019][1692529.967416] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 08:33:50 2019][1692529.974567] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 08:33:50 2019][1692529.981975] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 08:33:50 2019][1692529.989005] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:33:50 2019][1692529.996224] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 08:33:50 2019][1692530.003355] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 08:33:50 2019][1692530.010123] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:33:50 2019][1692530.016807] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:33:50 2019][1692530.023790] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:33:50 2019][1692530.031124] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:33:50 2019][1692530.037510] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:33:50 2019][1692530.044671] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:33:50 2019][1692530.052611] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:33:50 2019][1692530.059164] [] kthread+0xd1/0xe0 [Wed Feb 6 08:33:50 2019][1692530.064273] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:33:50 2019][1692530.070939] [] 0xffffffffffffffff [Wed Feb 6 08:33:50 2019][1692530.076146] LustreError: dumping log to /tmp/lustre-log.1549470868.9020 [Wed Feb 6 08:38:13 2019][1692793.481460] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 08:38:13 2019][1692793.492157] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 91388066 previous similar messages [Wed Feb 6 08:38:21 2019][1692801.503669] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549471139/real 1549471139] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549471150 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 08:38:21 2019][1692801.531529] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 91390193 previous similar messages [Wed Feb 6 08:38:35 2019][1692815.322064] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 225s: evicting client at 10.9.102.39@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff9159a97672c0/0x21ed9351b504e03f lrc: 3/0,0 mode: PR/PR res: [0x2c000169a:0x46:0x0].0x0 bits 0x40/0x0 rrc: 452906 type: IBT flags: 0x60000400010020 nid: 10.9.102.39@o2ib4 remote: 0x8de480ae73fe5073 expref: 452996 pid: 24923 timeout: 1692682 lvb_type: 0 [Wed Feb 6 08:38:35 2019][1692815.360757] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 79439 previous similar messages [Wed Feb 6 08:38:36 2019][1692816.665142] Lustre: 15281:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 08:38:36 2019][1692816.665142] req@ffff9126c0ac5700 x1624697987161840/t0(0) o101->d78401ad-d7f9-70f0-e382-808abf23c9bd@10.9.103.10@o2ib4:514/0 lens 592/3264 e 0 to 0 dl 1549471159 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 08:38:36 2019][1692816.694647] Lustre: 15281:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages [Wed Feb 6 08:38:56 2019][1692835.819541] LustreError: 6168:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549470873, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff91268e9d0900/0x21ed9351b82dcbf4 lrc: 3/1,0 mode: --/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 17 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 6168 timeout: 0 lvb_type: 0 [Wed Feb 6 08:38:56 2019][1692835.859099] LustreError: 6168:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Feb 6 08:39:28 2019][1692867.866340] LNet: Service thread pid 15294 was inactive for 1200.83s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 08:39:28 2019][1692867.883545] Pid: 15294, comm: mdt01_067 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:39:28 2019][1692867.893462] Call Trace: [Wed Feb 6 08:39:28 2019][1692867.896111] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 08:39:28 2019][1692867.903236] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 08:39:28 2019][1692867.910619] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 08:39:28 2019][1692867.917637] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:39:28 2019][1692867.924815] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 08:39:28 2019][1692867.931910] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 08:39:28 2019][1692867.938674] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:39:28 2019][1692867.945355] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:39:28 2019][1692867.952312] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:39:28 2019][1692867.959611] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:39:28 2019][1692867.965973] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:39:28 2019][1692867.973112] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:39:28 2019][1692867.981026] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:39:28 2019][1692867.987548] [] kthread+0xd1/0xe0 [Wed Feb 6 08:39:28 2019][1692867.992654] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:39:28 2019][1692867.999304] [] 0xffffffffffffffff [Wed Feb 6 08:39:28 2019][1692868.004508] LustreError: dumping log to /tmp/lustre-log.1549471206.15294 [Wed Feb 6 08:39:29 2019][1692869.168108] Lustre: fir-MDT0002: Client 54802b78-b8a3-6ef2-1c4c-1f299e55d7f1 (at 10.9.103.14@o2ib4) reconnecting [Wed Feb 6 08:39:29 2019][1692869.178463] Lustre: Skipped 11 previous similar messages [Wed Feb 6 08:40:37 2019][1692937.280207] LustreError: 17050:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff91510127f800 x1623682103902592/t0(0) o104->fir-MDT0002@10.9.103.8@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 08:40:37 2019][1692937.301387] LustreError: 17050:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 126030 previous similar messages [Wed Feb 6 08:41:35 2019][1692994.845533] Pid: 9171, comm: mdt02_021 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:41:35 2019][1692994.855364] Call Trace: [Wed Feb 6 08:41:35 2019][1692994.858013] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 08:41:35 2019][1692994.865121] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 08:41:35 2019][1692994.872486] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 08:41:35 2019][1692994.879488] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:41:35 2019][1692994.886673] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 08:41:35 2019][1692994.893764] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 08:41:35 2019][1692994.900508] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:41:35 2019][1692994.907163] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:41:35 2019][1692994.914096] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:41:35 2019][1692994.921376] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:41:35 2019][1692994.927719] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:41:35 2019][1692994.934838] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:41:35 2019][1692994.942725] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:41:35 2019][1692994.949222] [] kthread+0xd1/0xe0 [Wed Feb 6 08:41:35 2019][1692994.954309] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:41:35 2019][1692994.960957] [] 0xffffffffffffffff [Wed Feb 6 08:41:35 2019][1692994.966151] LustreError: dumping log to /tmp/lustre-log.1549471332.9171 [Wed Feb 6 08:42:16 2019][1693035.806552] Pid: 8573, comm: mdt03_011 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:42:16 2019][1693035.816382] Call Trace: [Wed Feb 6 08:42:16 2019][1693035.819027] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 08:42:16 2019][1693035.826157] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 08:42:16 2019][1693035.833534] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 08:42:16 2019][1693035.840542] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:42:16 2019][1693035.847719] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 08:42:16 2019][1693035.854806] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 08:42:16 2019][1693035.861549] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:42:16 2019][1693035.868205] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:42:16 2019][1693035.875139] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:42:16 2019][1693035.882419] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:42:16 2019][1693035.888754] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:42:16 2019][1693035.895873] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:42:16 2019][1693035.903757] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:42:16 2019][1693035.910259] [] kthread+0xd1/0xe0 [Wed Feb 6 08:42:16 2019][1693035.915336] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:42:16 2019][1693035.921974] [] 0xffffffffffffffff [Wed Feb 6 08:42:16 2019][1693035.927170] LustreError: dumping log to /tmp/lustre-log.1549471373.8573 [Wed Feb 6 08:43:09 2019][1693089.055892] Pid: 8972, comm: mdt01_019 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:43:09 2019][1693089.065720] Call Trace: [Wed Feb 6 08:43:09 2019][1693089.068361] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 08:43:09 2019][1693089.075472] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 08:43:09 2019][1693089.082839] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 08:43:09 2019][1693089.089839] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:43:09 2019][1693089.097014] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 08:43:09 2019][1693089.104096] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 08:43:09 2019][1693089.110829] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:43:09 2019][1693089.117477] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:43:09 2019][1693089.124411] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:43:09 2019][1693089.131692] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:43:09 2019][1693089.138027] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:43:09 2019][1693089.145143] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:43:09 2019][1693089.153031] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:43:09 2019][1693089.159565] [] kthread+0xd1/0xe0 [Wed Feb 6 08:43:09 2019][1693089.164650] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:43:09 2019][1693089.171289] [] 0xffffffffffffffff [Wed Feb 6 08:43:09 2019][1693089.176484] LustreError: dumping log to /tmp/lustre-log.1549471427.8972 [Wed Feb 6 08:44:15 2019][1693155.302698] Lustre: fir-MDT0002: Connection restored to efd3e0f8-d811-5264-1834-175e20ae309e (at 10.9.103.8@o2ib4) [Wed Feb 6 08:44:15 2019][1693155.313225] Lustre: Skipped 15 previous similar messages [Wed Feb 6 08:44:51 2019][1693191.458466] Pid: 15279, comm: mdt00_042 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:44:51 2019][1693191.468384] Call Trace: [Wed Feb 6 08:44:51 2019][1693191.471031] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 08:44:51 2019][1693191.478149] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 08:44:51 2019][1693191.485518] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 08:44:51 2019][1693191.492537] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:44:51 2019][1693191.499732] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 08:44:51 2019][1693191.506838] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 08:44:51 2019][1693191.513596] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:44:51 2019][1693191.520262] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:44:51 2019][1693191.527208] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:44:51 2019][1693191.534503] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:44:51 2019][1693191.540855] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:44:51 2019][1693191.547989] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:44:51 2019][1693191.555909] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:44:51 2019][1693191.562426] [] kthread+0xd1/0xe0 [Wed Feb 6 08:44:51 2019][1693191.567521] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:44:51 2019][1693191.574167] [] 0xffffffffffffffff [Wed Feb 6 08:44:51 2019][1693191.579371] LustreError: dumping log to /tmp/lustre-log.1549471529.15279 [Wed Feb 6 08:47:23 2019][1693343.014333] Pid: 15222, comm: mdt01_046 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:47:23 2019][1693343.024253] Call Trace: [Wed Feb 6 08:47:23 2019][1693343.026903] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 08:47:23 2019][1693343.034033] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 08:47:23 2019][1693343.041435] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 08:47:23 2019][1693343.048463] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:47:23 2019][1693343.055667] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 08:47:23 2019][1693343.062771] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 08:47:23 2019][1693343.069533] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:47:23 2019][1693343.076206] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:47:23 2019][1693343.083167] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:47:23 2019][1693343.090471] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:47:23 2019][1693343.096843] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:47:23 2019][1693343.103993] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:47:23 2019][1693343.111916] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:47:23 2019][1693343.118449] [] kthread+0xd1/0xe0 [Wed Feb 6 08:47:23 2019][1693343.123545] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:47:23 2019][1693343.130200] [] 0xffffffffffffffff [Wed Feb 6 08:47:23 2019][1693343.135412] LustreError: dumping log to /tmp/lustre-log.1549471681.15222 [Wed Feb 6 08:48:13 2019][1693393.496523] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 08:48:13 2019][1693393.507224] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 84585083 previous similar messages [Wed Feb 6 08:48:21 2019][1693401.518723] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549471739/real 1549471739] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549471750 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 08:48:21 2019][1693401.546578] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 84577975 previous similar messages [Wed Feb 6 08:48:35 2019][1693415.337139] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 225s: evicting client at 10.9.102.39@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff913b132e3a80/0x21ed9351b41673e2 lrc: 3/0,0 mode: PR/PR res: [0x2c000169a:0x46:0x0].0x0 bits 0x40/0x0 rrc: 447248 type: IBT flags: 0x60000400010020 nid: 10.9.102.39@o2ib4 remote: 0x8de480ae73f7b1ee expref: 447338 pid: 15331 timeout: 1693281 lvb_type: 0 [Wed Feb 6 08:48:35 2019][1693415.376042] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 156048 previous similar messages [Wed Feb 6 08:49:30 2019][1693470.261018] Lustre: fir-MDT0002: Client 54802b78-b8a3-6ef2-1c4c-1f299e55d7f1 (at 10.9.103.14@o2ib4) reconnecting [Wed Feb 6 08:49:30 2019][1693470.271375] Lustre: Skipped 12 previous similar messages [Wed Feb 6 08:50:34 2019][1693533.996922] Lustre: fir-MDT0002: haven't heard from client 40ad8941-e56d-16a8-06a4-9f03d85f8e6b (at 10.9.107.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff913b70d0c800, cur 1549471872 expire 1549471722 last 1549471645 [Wed Feb 6 08:50:34 2019][1693534.018894] Lustre: Skipped 4 previous similar messages [Wed Feb 6 08:50:37 2019][1693537.335301] LustreError: 15228:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff912933066f00 x1623682108020400/t0(0) o104->fir-MDT0002@10.9.102.39@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 08:50:37 2019][1693537.356572] LustreError: 15228:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 206658 previous similar messages [Wed Feb 6 08:53:56 2019][1693736.240134] LNet: Service thread pid 6168 was inactive for 1200.39s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 08:53:56 2019][1693736.257245] LNet: Skipped 5 previous similar messages [Wed Feb 6 08:53:56 2019][1693736.262483] Pid: 6168, comm: mdt00_001 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:53:56 2019][1693736.272308] Call Trace: [Wed Feb 6 08:53:56 2019][1693736.274945] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 08:53:56 2019][1693736.282066] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 08:53:56 2019][1693736.289432] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 08:53:56 2019][1693736.296434] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:53:56 2019][1693736.303608] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 08:53:56 2019][1693736.310709] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 08:53:56 2019][1693736.317450] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:53:56 2019][1693736.324098] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:53:56 2019][1693736.331032] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:53:56 2019][1693736.338313] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:53:56 2019][1693736.344648] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:53:56 2019][1693736.351765] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:53:56 2019][1693736.359649] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:53:56 2019][1693736.366152] [] kthread+0xd1/0xe0 [Wed Feb 6 08:53:56 2019][1693736.371237] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:53:56 2019][1693736.377885] [] 0xffffffffffffffff [Wed Feb 6 08:53:56 2019][1693736.383079] LustreError: dumping log to /tmp/lustre-log.1549472074.6168 [Wed Feb 6 08:54:16 2019][1693756.395618] Lustre: fir-MDT0002: Connection restored to efd3e0f8-d811-5264-1834-175e20ae309e (at 10.9.103.8@o2ib4) [Wed Feb 6 08:54:16 2019][1693756.406143] Lustre: Skipped 14 previous similar messages [Wed Feb 6 08:54:33 2019][1693773.105054] Pid: 6952, comm: mdt01_008 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:54:33 2019][1693773.114879] Call Trace: [Wed Feb 6 08:54:33 2019][1693773.117520] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 08:54:33 2019][1693773.124638] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 08:54:33 2019][1693773.132004] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 08:54:33 2019][1693773.139007] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:54:33 2019][1693773.146183] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 08:54:33 2019][1693773.153271] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 08:54:33 2019][1693773.160015] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:54:33 2019][1693773.166670] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:54:33 2019][1693773.173604] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:54:33 2019][1693773.180885] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:54:33 2019][1693773.187222] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:54:33 2019][1693773.194337] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:54:33 2019][1693773.202223] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:54:33 2019][1693773.208723] [] kthread+0xd1/0xe0 [Wed Feb 6 08:54:33 2019][1693773.213810] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:54:33 2019][1693773.220457] [] 0xffffffffffffffff [Wed Feb 6 08:54:33 2019][1693773.225661] LustreError: dumping log to /tmp/lustre-log.1549472111.6952 [Wed Feb 6 08:54:37 2019][1693777.201162] Pid: 9009, comm: mdt01_027 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 08:54:37 2019][1693777.210991] Call Trace: [Wed Feb 6 08:54:37 2019][1693777.213635] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 08:54:37 2019][1693777.220748] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 08:54:37 2019][1693777.228115] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 08:54:37 2019][1693777.235116] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 08:54:37 2019][1693777.242294] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 08:54:37 2019][1693777.249384] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 08:54:37 2019][1693777.256125] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 08:54:37 2019][1693777.262780] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 08:54:37 2019][1693777.269716] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 08:54:37 2019][1693777.276995] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 08:54:37 2019][1693777.283340] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 08:54:37 2019][1693777.290454] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 08:54:37 2019][1693777.298351] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 08:54:37 2019][1693777.304851] [] kthread+0xd1/0xe0 [Wed Feb 6 08:54:37 2019][1693777.309937] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 08:54:37 2019][1693777.316585] [] 0xffffffffffffffff [Wed Feb 6 08:54:37 2019][1693777.321778] LustreError: dumping log to /tmp/lustre-log.1549472115.9009 [Wed Feb 6 08:58:13 2019][1693993.511579] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 08:58:13 2019][1693993.522274] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 84586587 previous similar messages [Wed Feb 6 08:58:21 2019][1694001.533777] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549472339/real 1549472339] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549472350 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 08:58:21 2019][1694001.561630] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 84597194 previous similar messages [Wed Feb 6 08:58:35 2019][1694015.352226] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 225s: evicting client at 10.9.103.8@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff9150eef06300/0x21ed9351b0e94cb5 lrc: 3/0,0 mode: PR/PR res: [0x2c00016f0:0x4e:0x0].0x0 bits 0x40/0x0 rrc: 310220 type: IBT flags: 0x60000400010020 nid: 10.9.103.8@o2ib4 remote: 0x1ece216710a284f5 expref: 310298 pid: 23853 timeout: 1693972 lvb_type: 0 [Wed Feb 6 08:58:35 2019][1694015.390887] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 242304 previous similar messages [Wed Feb 6 08:59:31 2019][1694071.353975] Lustre: fir-MDT0002: Client 54802b78-b8a3-6ef2-1c4c-1f299e55d7f1 (at 10.9.103.14@o2ib4) reconnecting [Wed Feb 6 08:59:31 2019][1694071.364320] Lustre: Skipped 12 previous similar messages [Wed Feb 6 09:00:37 2019][1694137.368713] LustreError: 15228:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff912611c79e00 x1623682112529984/t0(0) o104->fir-MDT0002@10.9.102.39@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 09:00:37 2019][1694137.389981] LustreError: 15228:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 196983 previous similar messages [Wed Feb 6 09:01:12 2019][1694172.771087] LustreError: 17050:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549472210, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff9151f0a12400/0x21ed9351b48a1759 lrc: 3/0,1 mode: --/PW res: [0x2c00016f0:0x4e:0x0].0x0 bits 0x40/0x0 rrc: 307158 type: IBT flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 17050 timeout: 0 lvb_type: 0 [Wed Feb 6 09:01:12 2019][1694172.810987] LustreError: 17050:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages [Wed Feb 6 09:04:17 2019][1694357.489146] Lustre: fir-MDT0002: Connection restored to efd3e0f8-d811-5264-1834-175e20ae309e (at 10.9.103.8@o2ib4) [Wed Feb 6 09:04:17 2019][1694357.499668] Lustre: Skipped 16 previous similar messages [Wed Feb 6 09:04:43 2019][1694383.018116] Lustre: fir-MDT0002: haven't heard from client e7c11665-b408-cbb8-ba80-fcc49917f67e (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9124f863d000, cur 1549472721 expire 1549472571 last 1549472494 [Wed Feb 6 09:04:43 2019][1694383.039902] Lustre: Skipped 2 previous similar messages [Wed Feb 6 09:06:19 2019][1694479.682783] LNet: Service thread pid 15240 was inactive for 576.77s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 09:06:19 2019][1694479.699899] LNet: Skipped 2 previous similar messages [Wed Feb 6 09:06:19 2019][1694479.705130] Pid: 15240, comm: mdt00_038 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 09:06:19 2019][1694479.715042] Call Trace: [Wed Feb 6 09:06:19 2019][1694479.717677] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:06:19 2019][1694479.724820] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:06:19 2019][1694479.732184] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:06:19 2019][1694479.739185] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:06:19 2019][1694479.746363] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:06:19 2019][1694479.753467] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:06:19 2019][1694479.760202] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:06:19 2019][1694479.766866] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:06:19 2019][1694479.773809] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:06:19 2019][1694479.781096] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:06:19 2019][1694479.787447] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:06:19 2019][1694479.794586] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:06:19 2019][1694479.802485] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:06:19 2019][1694479.809003] [] kthread+0xd1/0xe0 [Wed Feb 6 09:06:20 2019][1694479.814100] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:06:20 2019][1694479.820755] [] 0xffffffffffffffff [Wed Feb 6 09:06:20 2019][1694479.825967] LustreError: dumping log to /tmp/lustre-log.1549472817.15240 [Wed Feb 6 09:06:37 2019][1694497.731247] Lustre: 8995:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 09:06:37 2019][1694497.731247] req@ffff911e1c2ec800 x1624673111292144/t0(0) o101->23391852-ef00-9fa9-e0d6-21dd26d1c3f4@10.9.102.42@o2ib4:685/0 lens 592/3264 e 6 to 0 dl 1549472840 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:06:37 2019][1694497.760403] Lustre: 8995:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages [Wed Feb 6 09:08:13 2019][1694593.526637] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 09:08:13 2019][1694593.537332] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 84669254 previous similar messages [Wed Feb 6 09:08:21 2019][1694601.548836] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549472939/real 1549472939] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549472950 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 09:08:21 2019][1694601.576694] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 84668557 previous similar messages [Wed Feb 6 09:08:35 2019][1694615.367241] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 134s: evicting client at 10.9.102.39@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff9154ffb74ec0/0x21ed9351b211fbf0 lrc: 3/0,0 mode: PR/PR res: [0x2c000169a:0x46:0x0].0x0 bits 0x40/0x0 rrc: 418682 type: IBT flags: 0x60000400010020 nid: 10.9.102.39@o2ib4 remote: 0x8de480ae73de95d3 expref: 418771 pid: 17204 timeout: 1694572 lvb_type: 0 [Wed Feb 6 09:08:35 2019][1694615.405937] LustreError: 6043:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 153364 previous similar messages [Wed Feb 6 09:09:32 2019][1694672.446975] Lustre: fir-MDT0002: Client 54802b78-b8a3-6ef2-1c4c-1f299e55d7f1 (at 10.9.103.14@o2ib4) reconnecting [Wed Feb 6 09:09:32 2019][1694672.457322] Lustre: Skipped 11 previous similar messages [Wed Feb 6 09:10:37 2019][1694737.409191] LustreError: 15228:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff911ed975d400 x1623682115408384/t0(0) o104->fir-MDT0002@10.9.102.39@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Feb 6 09:10:37 2019][1694737.430466] LustreError: 15228:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 131522 previous similar messages [Wed Feb 6 09:13:14 2019][1694893.858826] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 09:13:14 2019][1694893.869869] Lustre: Skipped 1108 previous similar messages [Wed Feb 6 09:14:12 2019][1694951.993505] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.102.18@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 09:14:12 2019][1694952.011053] LustreError: Skipped 3 previous similar messages [Wed Feb 6 09:14:12 2019][1694952.521219] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.104.20@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 09:14:13 2019][1694953.612335] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.107.32@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 09:14:13 2019][1694953.629879] LustreError: Skipped 4 previous similar messages [Wed Feb 6 09:14:15 2019][1694955.628867] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.105.8@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 09:14:15 2019][1694955.646321] LustreError: Skipped 20 previous similar messages [Wed Feb 6 09:14:17 2019][1694957.822814] Lustre: fir-MDT0000: Connection restored to d669b8c2-1597-51fa-d1ed-fc3284ffdc84 (at 10.9.107.54@o2ib4) [Wed Feb 6 09:14:17 2019][1694957.833416] Lustre: Skipped 32 previous similar messages [Wed Feb 6 09:14:20 2019][1694959.856848] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.102.23@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 09:14:20 2019][1694959.874396] LustreError: Skipped 11 previous similar messages [Wed Feb 6 09:14:28 2019][1694967.973544] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.107.9@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 09:14:28 2019][1694967.991001] LustreError: Skipped 31 previous similar messages [Wed Feb 6 09:14:57 2019][1694997.574888] Lustre: MGS: Received new LWP connection from 10.9.106.6@o2ib4, removing former export from same NID [Wed Feb 6 09:15:06 2019][1695006.552025] Lustre: fir-MDT0001-osp-MDT0002: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 09:15:06 2019][1695006.568180] Lustre: Skipped 52 previous similar messages [Wed Feb 6 09:15:27 2019][1695027.275585] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.101.24@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 09:15:27 2019][1695027.293125] LustreError: Skipped 17 previous similar messages [Wed Feb 6 09:15:39 2019][1695039.598952] Lustre: MGS: Received new LWP connection from 10.9.106.6@o2ib4, removing former export from same NID [Wed Feb 6 09:15:59 2019][1695059.344512] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.24.19@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 09:15:59 2019][1695059.361964] LustreError: Skipped 175 previous similar messages [Wed Feb 6 09:17:17 2019][1695137.384460] Lustre: DEBUG MARKER: Wed Feb 6 09:17:55 2019 [Wed Feb 6 09:17:17 2019][1695137.384460] [Wed Feb 6 09:17:56 2019][1695175.855259] LustreError: 15228:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549473213, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff9120f5ce9440/0x21ed9351b580689b lrc: 3/0,1 mode: --/PW res: [0x2c000169a:0x46:0x0].0x0 bits 0x40/0x0 rrc: 403430 type: IBT flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 15228 timeout: 0 lvb_type: 0 [Wed Feb 6 09:17:56 2019][1695175.895184] LustreError: 15228:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Feb 6 09:18:13 2019][1695193.541693] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 09:18:13 2019][1695193.552388] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 84559765 previous similar messages [Wed Feb 6 09:18:21 2019][1695201.563896] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549473539/real 1549473539] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549473550 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 09:18:21 2019][1695201.591749] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 84563838 previous similar messages [Wed Feb 6 09:18:33 2019][1695213.794213] LustreError: 6171:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549473251, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff9139bbc31b00/0x21ed9351bad1f146 lrc: 3/1,0 mode: --/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 23 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 6171 timeout: 0 lvb_type: 0 [Wed Feb 6 09:19:25 2019][1695265.110496] LNet: Service thread pid 35530 was inactive for 350.85s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 09:19:25 2019][1695265.127609] Pid: 35530, comm: mdt02_024 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 09:19:25 2019][1695265.137522] Call Trace: [Wed Feb 6 09:19:25 2019][1695265.140184] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.147339] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.154723] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:19:25 2019][1695265.161731] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:19:25 2019][1695265.168906] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:19:25 2019][1695265.176002] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:19:25 2019][1695265.182750] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:19:25 2019][1695265.189403] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.196349] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.203643] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.210001] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.217133] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.225045] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.231563] [] kthread+0xd1/0xe0 [Wed Feb 6 09:19:25 2019][1695265.236657] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:19:25 2019][1695265.243306] [] 0xffffffffffffffff [Wed Feb 6 09:19:25 2019][1695265.248518] LustreError: dumping log to /tmp/lustre-log.1549473603.35530 [Wed Feb 6 09:19:33 2019][1695273.540000] Lustre: fir-MDT0002: Client 54802b78-b8a3-6ef2-1c4c-1f299e55d7f1 (at 10.9.103.14@o2ib4) reconnecting [Wed Feb 6 09:19:33 2019][1695273.550345] Lustre: Skipped 17 previous similar messages [Wed Feb 6 09:20:15 2019][1695315.287756] Pid: 23853, comm: mdt03_030 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 09:20:15 2019][1695315.297670] Call Trace: [Wed Feb 6 09:20:15 2019][1695315.300321] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:20:15 2019][1695315.307453] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:20:15 2019][1695315.314865] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:20:15 2019][1695315.321874] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:20:15 2019][1695315.329060] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:20:15 2019][1695315.336157] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:20:15 2019][1695315.342926] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:20:15 2019][1695315.349590] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:20:15 2019][1695315.356550] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:20:15 2019][1695315.363839] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:20:15 2019][1695315.370218] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:20:15 2019][1695315.377378] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:20:15 2019][1695315.385298] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:20:15 2019][1695315.391816] [] kthread+0xd1/0xe0 [Wed Feb 6 09:20:15 2019][1695315.396903] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:20:15 2019][1695315.403550] [] 0xffffffffffffffff [Wed Feb 6 09:20:15 2019][1695315.408769] LustreError: dumping log to /tmp/lustre-log.1549473653.23853 [Wed Feb 6 09:20:52 2019][1695352.152685] Pid: 8949, comm: mdt01_015 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 09:20:52 2019][1695352.162511] Call Trace: [Wed Feb 6 09:20:52 2019][1695352.165154] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.172275] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.179651] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:20:52 2019][1695352.186657] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:20:52 2019][1695352.193837] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:20:52 2019][1695352.200926] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:20:52 2019][1695352.207670] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:20:52 2019][1695352.214316] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.221261] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.228548] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.234893] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.242017] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.249914] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.256422] [] kthread+0xd1/0xe0 [Wed Feb 6 09:20:52 2019][1695352.261507] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:20:52 2019][1695352.268156] [] 0xffffffffffffffff [Wed Feb 6 09:20:52 2019][1695352.273358] LustreError: dumping log to /tmp/lustre-log.1549473690.8949 [Wed Feb 6 09:20:52 2019][1695352.306836] Pid: 6171, comm: mdt01_001 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 09:20:52 2019][1695352.316947] Call Trace: [Wed Feb 6 09:20:52 2019][1695352.319601] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.326824] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.334271] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:20:52 2019][1695352.341359] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:20:52 2019][1695352.348542] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:20:52 2019][1695352.355629] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:20:52 2019][1695352.362374] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:20:52 2019][1695352.369029] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.375971] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.383262] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.389604] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.396730] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.404625] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:20:52 2019][1695352.411134] [] kthread+0xd1/0xe0 [Wed Feb 6 09:20:52 2019][1695352.416220] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:20:52 2019][1695352.422858] [] 0xffffffffffffffff [Wed Feb 6 09:20:55 2019][1695355.224760] Pid: 15385, comm: mdt01_071 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 09:20:55 2019][1695355.234669] Call Trace: [Wed Feb 6 09:20:55 2019][1695355.237317] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:20:55 2019][1695355.244438] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:20:55 2019][1695355.251840] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:20:55 2019][1695355.258850] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:20:55 2019][1695355.266035] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:20:55 2019][1695355.273141] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:20:55 2019][1695355.279893] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:20:55 2019][1695355.286548] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:20:55 2019][1695355.293482] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:20:55 2019][1695355.300774] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:20:55 2019][1695355.307125] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:20:55 2019][1695355.314273] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:20:55 2019][1695355.322169] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:20:55 2019][1695355.328680] [] kthread+0xd1/0xe0 [Wed Feb 6 09:20:55 2019][1695355.333774] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:20:55 2019][1695355.340423] [] 0xffffffffffffffff [Wed Feb 6 09:20:55 2019][1695355.345632] LustreError: dumping log to /tmp/lustre-log.1549473693.15385 [Wed Feb 6 09:21:23 2019][1695382.873460] LNet: Service thread pid 24899 was inactive for 400.21s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Feb 6 09:21:23 2019][1695382.886503] LustreError: dumping log to /tmp/lustre-log.1549473720.24899 [Wed Feb 6 09:22:55 2019][1695475.814907] LNet: Service thread pid 24899 completed after 493.15s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Feb 6 09:22:56 2019][1695475.831232] LNet: Skipped 2 previous similar messages [Wed Feb 6 09:23:28 2019][1695508.572656] Lustre: 8970:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 09:23:28 2019][1695508.572656] req@ffff913484babf00 x1624697977302400/t0(0) o101->819a2974-4f03-e3b0-601d-2d97317e6637@10.9.103.3@o2ib4:186/0 lens 592/3264 e 3 to 0 dl 1549473851 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:24:07 2019][1695547.561604] LustreError: 15239:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549473585, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff912a4b390d80/0x21ed9351bb89e5e9 lrc: 3/1,0 mode: --/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 24 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 15239 timeout: 0 lvb_type: 0 [Wed Feb 6 09:24:07 2019][1695547.601344] LustreError: 15239:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 4 previous similar messages [Wed Feb 6 09:24:19 2019][1695559.633222] Lustre: fir-MDT0002: Connection restored to efd3e0f8-d811-5264-1834-175e20ae309e (at 10.9.103.8@o2ib4) [Wed Feb 6 09:24:19 2019][1695559.643740] Lustre: Skipped 25 previous similar messages [Wed Feb 6 09:24:43 2019][1695583.462389] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.105.37@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 09:24:43 2019][1695583.479937] LustreError: Skipped 80 previous similar messages [Wed Feb 6 09:26:03 2019][1695663.456509] Lustre: 24899:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply [Wed Feb 6 09:26:03 2019][1695663.456509] req@ffff9159e6228f00 x1624697978360368/t0(0) o101->5fd68af5-0c2b-1947-5fc1-6504b55b60fb@10.9.103.16@o2ib4:341/0 lens 592/3264 e 0 to 0 dl 1549474006 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:26:03 2019][1695663.486006] Lustre: 24899:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [Wed Feb 6 09:26:51 2019][1695711.636395] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.107.12@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Feb 6 09:26:51 2019][1695711.653946] LustreError: Skipped 22 previous similar messages [Wed Feb 6 09:27:21 2019][1695741.234442] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [Wed Feb 6 09:27:21 2019][1695741.244716] LNetError: 5873:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 1 previous similar message [Wed Feb 6 09:27:21 2019][1695741.254920] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (5): c: 0, oc: 1, rc: 8 [Wed Feb 6 09:27:21 2019][1695741.266904] LNetError: 5873:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 1 previous similar message [Wed Feb 6 09:27:24 2019][1695743.929009] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 09:27:24 2019][1695743.940054] Lustre: Skipped 1 previous similar message [Wed Feb 6 09:27:57 2019][1695777.787381] Lustre: fir-MDT0003-osp-MDT0002: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 09:27:57 2019][1695777.803541] Lustre: Skipped 1 previous similar message [Wed Feb 6 09:28:01 2019][1695781.355458] Lustre: fir-MDT0003-osp-MDT0000: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 09:28:02 2019][1695782.275115] Lustre: fir-MDT0002: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 09:28:02 2019][1695782.286156] Lustre: Skipped 3 previous similar messages [Wed Feb 6 09:28:05 2019][1695785.691569] Lustre: fir-MDT0001-osp-MDT0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Feb 6 09:28:13 2019][1695793.556749] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 09:28:13 2019][1695793.567444] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 84806372 previous similar messages [Wed Feb 6 09:28:21 2019][1695801.578952] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549474139/real 1549474139] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549474150 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 09:28:21 2019][1695801.606805] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 84807216 previous similar messages [Wed Feb 6 09:29:03 2019][1695843.173015] Lustre: 9015:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/4), not sending early reply [Wed Feb 6 09:29:03 2019][1695843.173015] req@ffff913d4765e600 x1624697947926560/t0(0) o101->3f8f2dad-883d-00e9-c850-0ee136ec2184@10.9.103.13@o2ib4:520/0 lens 592/3264 e 1 to 0 dl 1549474185 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:29:03 2019][1695843.202167] Lustre: 9015:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Feb 6 09:29:34 2019][1695874.625459] Lustre: fir-MDT0002: Client 54802b78-b8a3-6ef2-1c4c-1f299e55d7f1 (at 10.9.103.14@o2ib4) reconnecting [Wed Feb 6 09:29:34 2019][1695874.635805] Lustre: Skipped 20 previous similar messages [Wed Feb 6 09:29:54 2019][1695894.246296] Lustre: 9015:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Feb 6 09:29:54 2019][1695894.246296] req@ffff9132bc6eda00 x1624736178143008/t0(0) o101->c7d587fd-2878-3a53-bb0b-89a81458bb83@10.8.6.5@o2ib6:572/0 lens 376/1600 e 1 to 0 dl 1549474237 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Feb 6 09:34:20 2019][1696160.749058] Lustre: fir-MDT0002: Connection restored to efd3e0f8-d811-5264-1834-175e20ae309e (at 10.9.103.8@o2ib4) [Wed Feb 6 09:34:20 2019][1696160.759578] Lustre: Skipped 36 previous similar messages [Wed Feb 6 09:37:05 2019][1696325.067413] Lustre: fir-MDT0002: haven't heard from client 961b1d3a-e8f6-ab43-e9b3-f245b8a015f0 (at 10.8.10.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9126493abc00, cur 1549474663 expire 1549474513 last 1549474436 [Wed Feb 6 09:37:05 2019][1696325.089293] Lustre: Skipped 2 previous similar messages [Wed Feb 6 09:38:13 2019][1696393.571807] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) no route to 10.10.114.10@tcp from [Wed Feb 6 09:38:13 2019][1696393.582504] LNetError: 6615:0:(lib-move.c:1980:lnet_handle_find_routed_path()) Skipped 84744457 previous similar messages [Wed Feb 6 09:38:21 2019][1696401.594010] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1549474739/real 1549474739] req@ffff912521032d00 x1623682053684704/t0(0) o106->fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549474750 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 [Wed Feb 6 09:38:21 2019][1696401.621865] Lustre: 6615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 84742232 previous similar messages [Wed Feb 6 09:38:21 2019][1696401.732258] SysRq : Show State [Wed Feb 6 09:38:21 2019][1696401.735516] task PC stack pid father [Wed Feb 6 09:38:21 2019][1696401.741616] systemd S ffff913aa9e88000 0 1 0 0x00000000 [Wed Feb 6 09:38:21 2019][1696401.748906] Call Trace: [Wed Feb 6 09:38:21 2019][1696401.751563] [] schedule+0x29/0x70 [Wed Feb 6 09:38:21 2019][1696401.756707] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:21 2019][1696401.763951] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Wed Feb 6 09:38:21 2019][1696401.770922] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:21 2019][1696401.777470] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:38:21 2019][1696401.782708] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:21 2019][1696401.788476] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:38:21 2019][1696401.794235] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:21 2019][1696401.800435] kthreadd S ffff913aa9e89040 0 2 0 0x00000000 [Wed Feb 6 09:38:21 2019][1696401.807722] Call Trace: [Wed Feb 6 09:38:21 2019][1696401.810364] [] schedule+0x29/0x70 [Wed Feb 6 09:38:21 2019][1696401.815511] [] kthreadd+0x2f5/0x300 [Wed Feb 6 09:38:21 2019][1696401.820846] [] ? kthread_create_on_cpu+0x60/0x60 [Wed Feb 6 09:38:21 2019][1696401.827295] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:21 2019][1696401.833919] [] ? kthread_create_on_cpu+0x60/0x60 [Wed Feb 6 09:38:21 2019][1696401.840381] ksoftirqd/0 S ffff911b29b90000 0 3 2 0x00000000 [Wed Feb 6 09:38:21 2019][1696401.847671] Call Trace: [Wed Feb 6 09:38:21 2019][1696401.850319] [] schedule+0x29/0x70 [Wed Feb 6 09:38:21 2019][1696401.855463] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:21 2019][1696401.861496] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696401.867509] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696401.872575] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696401.878864] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696401.885488] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696401.891775] kworker/0:0H S ffff911b29b92080 0 5 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696401.899068] Call Trace: [Wed Feb 6 09:38:22 2019][1696401.901716] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:22 2019][1696401.907902] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696401.913065] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:22 2019][1696401.918820] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:22 2019][1696401.925542] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696401.930608] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696401.936895] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696401.943513] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696401.949801] migration/0 S ffff911b29b95140 0 8 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696401.957094] Call Trace: [Wed Feb 6 09:38:22 2019][1696401.959740] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696401.964882] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:22 2019][1696401.970916] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696401.976933] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696401.981994] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696401.988283] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696401.994908] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.001194] rcu_bh S ffff911b29b96180 0 9 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.008467] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.011121] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:22 2019][1696402.016789] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696402.021953] [] rcu_gp_kthread+0xb2/0x710 [Wed Feb 6 09:38:22 2019][1696402.027712] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:22 2019][1696402.033727] [] ? rcu_gp_fqs+0x90/0x90 [Wed Feb 6 09:38:22 2019][1696402.039236] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696402.044299] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.050589] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696402.057215] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.063501] rcu_sched S ffff911b29be0000 0 10 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.070769] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.073417] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696402.078569] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:38:22 2019][1696402.084600] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:38:22 2019][1696402.091137] [] ? prepare_to_wait+0x56/0x90 [Wed Feb 6 09:38:22 2019][1696402.097076] [] rcu_gp_kthread+0x349/0x710 [Wed Feb 6 09:38:22 2019][1696402.102910] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:22 2019][1696402.108942] [] ? rcu_gp_fqs+0x90/0x90 [Wed Feb 6 09:38:22 2019][1696402.114438] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696402.119505] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.125792] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696402.132417] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.138703] lru-add-drain S ffff911b29be1040 0 11 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.145977] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.148625] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696402.153771] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:22 2019][1696402.159627] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:22 2019][1696402.165819] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696402.170893] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.177174] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696402.183807] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.190086] watchdog/0 S ffff911b29be2080 0 12 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.197379] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.200025] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:22 2019][1696402.205521] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696402.210684] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:22 2019][1696402.216700] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.222730] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696402.227791] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.234084] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696402.240705] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.246981] watchdog/1 S ffff912aa989a080 0 13 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.254287] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.256925] [] ? watchdog_cleanup+0x10/0x10 [Wed Feb 6 09:38:22 2019][1696402.262938] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:22 2019][1696402.268448] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696402.273600] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:22 2019][1696402.279627] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.285633] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696402.290713] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.296993] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696402.303610] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.309902] migration/1 S ffff912aa989b0c0 0 14 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.317177] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.319807] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696402.324956] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:22 2019][1696402.330990] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.337006] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696402.342068] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.348357] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696402.354984] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.361267] ksoftirqd/1 S ffff912aa989c100 0 15 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.368539] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.371180] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696402.376325] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:22 2019][1696402.382359] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.388371] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696402.393428] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.399716] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696402.406341] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.412631] kworker/1:0H S ffff912aa98b9040 0 17 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.419920] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.422567] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:22 2019][1696402.428752] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696402.433915] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:22 2019][1696402.439664] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:22 2019][1696402.446375] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696402.451448] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.457718] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696402.464352] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.470624] watchdog/2 S ffff913aa9fcb0c0 0 19 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.477914] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.480544] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:22 2019][1696402.486044] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696402.491206] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:22 2019][1696402.497221] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.503251] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696402.508315] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.514588] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696402.521228] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.527502] migration/2 S ffff913aa9800000 0 20 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.534799] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.537452] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696402.542600] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:22 2019][1696402.548629] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.554638] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696402.559698] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.565990] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696402.572615] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.578891] ksoftirqd/2 S ffff913aa9801040 0 21 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.586189] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.588817] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696402.593958] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:22 2019][1696402.599989] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.605997] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696402.611060] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.617350] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696402.623968] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.630242] kworker/2:0H S ffff913aa98030c0 0 23 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.637546] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.640179] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:22 2019][1696402.646373] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696402.651533] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:22 2019][1696402.657291] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:22 2019][1696402.664009] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696402.669070] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.675357] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696402.681974] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.688263] watchdog/3 S ffff914aa9f7a080 0 25 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.695548] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.698191] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:22 2019][1696402.703681] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696402.708845] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:22 2019][1696402.714856] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.720871] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696402.725946] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.732213] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696402.738849] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.745117] migration/3 S ffff914aa9f7b0c0 0 26 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.752385] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.755016] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 09:38:22 2019][1696402.761655] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696402.766806] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:22 2019][1696402.772821] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.778847] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696402.783904] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.790194] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696402.796817] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.803109] ksoftirqd/3 S ffff914aa9f7c100 0 27 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.810393] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.813023] [] schedule+0x29/0x70 [Wed Feb 6 09:38:22 2019][1696402.818173] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:22 2019][1696402.824200] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.830213] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:22 2019][1696402.835274] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.841563] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:22 2019][1696402.848188] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:22 2019][1696402.854477] kworker/3:0H S ffff914aa9f99040 0 29 2 0x00000000 [Wed Feb 6 09:38:22 2019][1696402.861756] Call Trace: [Wed Feb 6 09:38:22 2019][1696402.864397] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:23 2019][1696402.870580] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696402.875743] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:23 2019][1696402.881495] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:23 2019][1696402.888205] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696402.893281] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696402.899554] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696402.906189] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696402.912458] watchdog/4 S ffff911b2950b0c0 0 31 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696402.919734] Call Trace: [Wed Feb 6 09:38:23 2019][1696402.922378] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:23 2019][1696402.927866] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696402.933025] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:23 2019][1696402.939046] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696402.945065] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696402.950138] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696402.956408] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696402.963052] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696402.969322] migration/4 S ffff911b29558000 0 32 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696402.976610] Call Trace: [Wed Feb 6 09:38:23 2019][1696402.979253] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696402.984393] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:23 2019][1696402.990423] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696402.996431] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.001493] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.007783] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.014402] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.020677] ksoftirqd/4 S ffff911b29559040 0 33 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.027967] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.030597] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.035750] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:23 2019][1696403.041777] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.047790] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.052859] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.059131] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.065754] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.072048] kworker/4:0H S ffff911b2955b0c0 0 35 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.079331] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.081969] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:23 2019][1696403.088170] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.093322] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:23 2019][1696403.099074] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:23 2019][1696403.105795] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.110856] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.117131] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.123765] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.130046] watchdog/5 S ffff912aa98be180 0 36 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.137338] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.139977] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:23 2019][1696403.145471] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.150631] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:23 2019][1696403.156644] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.162670] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.167727] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.174014] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.180634] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.186946] migration/5 S ffff912aa98bd140 0 37 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.194232] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.196865] [] ? cpu_stop_queue_work+0x70/0x70 [Wed Feb 6 09:38:23 2019][1696403.203157] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.208304] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:23 2019][1696403.214318] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.220348] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.225406] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.231682] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.238317] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.244594] ksoftirqd/5 S ffff912aa98bc100 0 38 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.251897] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.254535] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.259680] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:23 2019][1696403.265709] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.271717] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.276772] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.283061] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.289678] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.295957] kworker/5:0H S ffff912aa98f8000 0 40 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.303258] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.305892] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:23 2019][1696403.312083] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.317239] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:23 2019][1696403.322992] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:23 2019][1696403.329714] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.334773] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.341051] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.347683] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.353962] watchdog/6 S ffff913aa9806180 0 41 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.361260] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.363889] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:23 2019][1696403.369377] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.374537] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:23 2019][1696403.380546] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.386557] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.391632] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.397910] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.404547] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.410820] migration/6 S ffff9159fc6bb200 0 42 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.418107] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.420735] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.425884] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:23 2019][1696403.431914] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.437926] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.442988] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.449275] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.455893] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.462177] ksoftirqd/6 S ffff913aa9840000 0 43 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.469451] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.472092] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.477239] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:23 2019][1696403.483266] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.489277] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.494352] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.500631] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.507266] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.513542] kworker/6:0H S ffff913aa9842080 0 45 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.520839] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.523474] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:23 2019][1696403.529662] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.534826] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:23 2019][1696403.540578] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:23 2019][1696403.547286] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.552359] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.558632] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.565263] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.571532] watchdog/7 S ffff914aa9f9e180 0 46 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.578824] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.581456] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:23 2019][1696403.586956] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.592114] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:23 2019][1696403.598125] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.604153] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.609214] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.615497] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.622110] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.628378] migration/7 S ffff914aa9f9d140 0 47 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.635676] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.638313] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.643462] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:23 2019][1696403.649489] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.655499] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.660570] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.666845] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.673475] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.679748] ksoftirqd/7 S ffff914aa9f9c100 0 48 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.687044] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.689682] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.694831] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:23 2019][1696403.700854] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.706867] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.711942] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.718214] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.724851] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.731126] kworker/7:0H S ffff914aa9fe0000 0 50 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.738423] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.741058] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:23 2019][1696403.747248] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.752408] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:23 2019][1696403.758161] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:23 2019][1696403.764881] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.769945] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.776236] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.782859] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.789144] watchdog/8 S ffff911b2955e180 0 51 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.796416] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.799047] [] ? touch_nmi_watchdog+0x30/0x30 [Wed Feb 6 09:38:23 2019][1696403.805248] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:23 2019][1696403.810737] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.815880] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:23 2019][1696403.821912] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.827926] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:23 2019][1696403.832990] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.839271] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:23 2019][1696403.845891] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:23 2019][1696403.852180] migration/8 S ffff911b2955d140 0 52 2 0x00000000 [Wed Feb 6 09:38:23 2019][1696403.859453] Call Trace: [Wed Feb 6 09:38:23 2019][1696403.862082] [] schedule+0x29/0x70 [Wed Feb 6 09:38:23 2019][1696403.867230] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:24 2019][1696403.873260] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696403.879275] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696403.884352] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696403.890620] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696403.897242] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696403.903535] ksoftirqd/8 S ffff911b2955c100 0 53 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696403.910819] Call Trace: [Wed Feb 6 09:38:24 2019][1696403.913447] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696403.918588] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:24 2019][1696403.924618] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696403.930627] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696403.935688] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696403.941981] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696403.948607] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696403.954880] kworker/8:0H S ffff911b295a9040 0 55 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696403.962174] Call Trace: [Wed Feb 6 09:38:24 2019][1696403.964807] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:24 2019][1696403.970988] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696403.976152] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:24 2019][1696403.981899] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:24 2019][1696403.988616] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696403.993690] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696403.999968] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.006604] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.012878] watchdog/9 S ffff912aa98fe180 0 56 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.020181] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.022820] [] ? watchdog_cleanup+0x10/0x10 [Wed Feb 6 09:38:24 2019][1696404.028835] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:24 2019][1696404.034344] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.039494] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:24 2019][1696404.045526] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.051545] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696404.056617] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.062891] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.069528] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.075796] migration/9 S ffff912aa98fd140 0 57 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.083073] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.085716] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.090865] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:24 2019][1696404.096893] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.102910] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696404.107985] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.114251] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.120888] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.127158] ksoftirqd/9 S ffff912aa98fc100 0 58 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.134442] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.137089] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.142242] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:24 2019][1696404.148269] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.154276] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696404.159352] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.165620] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.172247] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.178533] kworker/9:0H S ffff912aa98fa080 0 60 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.185806] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.188454] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:24 2019][1696404.194637] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.199802] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:24 2019][1696404.205548] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:24 2019][1696404.212258] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696404.217333] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.223606] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.230227] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.236516] watchdog/10 S ffff913aa9846180 0 61 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.243800] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.246445] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:24 2019][1696404.251934] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.257097] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:24 2019][1696404.263105] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.269120] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696404.274196] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.280466] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.287092] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.293379] migration/10 S ffff913aa9845140 0 62 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.300656] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.303298] [] ? cpu_stop_unpark+0x50/0x50 [Wed Feb 6 09:38:24 2019][1696404.309222] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.314386] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:24 2019][1696404.320392] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.326400] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696404.331476] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.337749] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.344370] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.350657] ksoftirqd/10 S ffff913aa9844100 0 63 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.357943] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.360588] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.365735] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:24 2019][1696404.371760] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.377769] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696404.382847] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.389126] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.395747] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.402036] kworker/10:0H S ffff913aa9878000 0 65 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.409326] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.411960] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:24 2019][1696404.418161] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.423303] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:24 2019][1696404.429053] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:24 2019][1696404.435773] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696404.440829] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.447104] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.453739] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.460011] watchdog/11 S ffff914aa9fe6180 0 66 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.467305] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.469932] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:24 2019][1696404.475447] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.480592] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:24 2019][1696404.486611] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.492641] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696404.497701] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.503989] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.510606] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.516880] migration/11 S ffff914aa9fe5140 0 67 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.524170] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.526811] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 09:38:24 2019][1696404.533430] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.538591] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:24 2019][1696404.544611] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.550640] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696404.555699] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.561995] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.568618] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.574893] ksoftirqd/11 S ffff914aa9fe4100 0 68 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.582195] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.584830] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.589976] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:24 2019][1696404.596006] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.602015] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696404.607082] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.613366] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.619983] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.626262] kworker/11:0H S ffff914aa9fe2080 0 70 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.633565] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.636196] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:24 2019][1696404.642385] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.647551] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:24 2019][1696404.653308] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:24 2019][1696404.660014] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696404.665095] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.671372] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.677993] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.684280] watchdog/12 S ffff911b295ae180 0 71 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.691567] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.694208] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:24 2019][1696404.699702] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.704864] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:24 2019][1696404.710871] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.716883] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696404.721957] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.728235] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.734871] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.741149] migration/12 S ffff911b295ad140 0 72 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.748440] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.751069] [] ? cpu_stop_queue_work+0x70/0x70 [Wed Feb 6 09:38:24 2019][1696404.757343] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.762508] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:24 2019][1696404.768518] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.774534] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696404.779608] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.785878] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.792512] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.798782] ksoftirqd/12 S ffff911b295ac100 0 73 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.806073] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.808706] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.813856] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:24 2019][1696404.819885] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.825902] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:24 2019][1696404.830977] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.837255] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:24 2019][1696404.843888] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:24 2019][1696404.850161] kworker/12:0H S ffff911b295aa080 0 75 2 0x00000000 [Wed Feb 6 09:38:24 2019][1696404.857458] Call Trace: [Wed Feb 6 09:38:24 2019][1696404.860095] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:24 2019][1696404.866280] [] schedule+0x29/0x70 [Wed Feb 6 09:38:24 2019][1696404.871443] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:25 2019][1696404.877199] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:25 2019][1696404.883924] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696404.888988] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696404.895264] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696404.901899] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696404.908177] watchdog/13 S ffff912aa98f9040 0 76 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696404.915462] Call Trace: [Wed Feb 6 09:38:25 2019][1696404.918112] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:25 2019][1696404.923600] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696404.928762] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:25 2019][1696404.934772] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696404.940790] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696404.945866] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696404.952146] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696404.958780] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696404.965059] migration/13 S ffff912aa9958000 0 77 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696404.972349] Call Trace: [Wed Feb 6 09:38:25 2019][1696404.974977] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696404.980118] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:25 2019][1696404.986148] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696404.992160] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696404.997220] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.003509] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.010134] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.016428] ksoftirqd/13 S ffff912aa9959040 0 78 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.023709] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.026354] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.031498] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:25 2019][1696405.037526] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.043545] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696405.048619] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.054887] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.061526] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.067802] kworker/13:0H S ffff912aa995b0c0 0 80 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.075100] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.077753] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:25 2019][1696405.083942] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.089106] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:25 2019][1696405.094858] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:25 2019][1696405.101580] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696405.106636] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.112915] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.119550] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.125828] watchdog/14 R running task 0 81 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.133133] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.135766] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:25 2019][1696405.141265] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.146427] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:25 2019][1696405.152444] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.158473] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696405.163539] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.169827] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.176450] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.182741] migration/14 R running task 0 82 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.190017] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.192660] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.197806] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:25 2019][1696405.203835] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.209855] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696405.214928] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.221207] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.227842] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.234118] ksoftirqd/14 R running task 0 83 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.241419] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.244055] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.249201] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:25 2019][1696405.255230] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.261245] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696405.266325] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.272602] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.279221] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.285514] kworker/14:0H S ffff913aa987a080 0 85 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.292822] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.295457] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:25 2019][1696405.301646] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.306807] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:25 2019][1696405.312565] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:25 2019][1696405.319285] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696405.324344] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.330632] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.337258] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.343550] watchdog/15 S ffff914aa9fe1040 0 86 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.350834] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.353479] [] ? watchdog_cleanup+0x10/0x10 [Wed Feb 6 09:38:25 2019][1696405.359494] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:25 2019][1696405.365005] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.370156] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:25 2019][1696405.376187] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.382202] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696405.387280] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.393557] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.400194] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.406470] migration/15 S ffff914aa9428000 0 87 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.413762] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.416394] [] ? cpu_stop_unpark+0x50/0x50 [Wed Feb 6 09:38:25 2019][1696405.422324] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.427485] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:25 2019][1696405.433498] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.439530] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696405.444586] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.450862] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.457498] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.463776] ksoftirqd/15 S ffff914aa9429040 0 88 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.471075] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.473706] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.478854] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:25 2019][1696405.484886] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.490900] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696405.495963] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.502254] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.508878] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.515155] kworker/15:0H S ffff914aa942b0c0 0 90 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.522451] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.525081] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:25 2019][1696405.531262] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.536425] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:25 2019][1696405.542180] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:25 2019][1696405.548886] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696405.553959] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.560230] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.566863] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.573135] watchdog/16 S ffff911b29638000 0 91 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.580425] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.583055] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:25 2019][1696405.588555] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.593716] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:25 2019][1696405.599735] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.605761] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696405.610815] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.617110] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.623728] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.630004] migration/16 S ffff911b29639040 0 92 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.637301] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.639938] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.645087] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:25 2019][1696405.651121] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.657132] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696405.662210] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.668485] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.675114] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.681406] ksoftirqd/16 S ffff911b2963a080 0 93 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.688686] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.691329] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.696472] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:25 2019][1696405.702502] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.708521] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696405.713594] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.719866] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.726499] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.732772] kworker/16:0H S ffff911b2963c100 0 95 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.740074] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.742707] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:25 2019][1696405.748896] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.754064] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:25 2019][1696405.759820] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:25 2019][1696405.766525] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696405.771603] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.777883] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.784503] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.790791] watchdog/17 S ffff912aa995e180 0 96 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.798069] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.800712] [] ? touch_nmi_watchdog+0x30/0x30 [Wed Feb 6 09:38:25 2019][1696405.806894] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:25 2019][1696405.812406] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.817560] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:25 2019][1696405.823573] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.829599] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:25 2019][1696405.834657] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.840945] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:25 2019][1696405.847563] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:25 2019][1696405.853853] migration/17 S ffff912aa995d140 0 97 2 0x00000000 [Wed Feb 6 09:38:25 2019][1696405.861149] Call Trace: [Wed Feb 6 09:38:25 2019][1696405.863801] [] schedule+0x29/0x70 [Wed Feb 6 09:38:25 2019][1696405.868943] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:25 2019][1696405.874971] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696405.880980] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696405.886055] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696405.892324] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696405.898949] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696405.905237] ksoftirqd/17 S ffff912aa995c100 0 98 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696405.912520] Call Trace: [Wed Feb 6 09:38:26 2019][1696405.915183] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696405.920344] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:26 2019][1696405.926372] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696405.932383] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696405.937444] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696405.943737] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696405.950360] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696405.956637] kworker/17:0H S ffff912aa9991040 0 100 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696405.963932] Call Trace: [Wed Feb 6 09:38:26 2019][1696405.966566] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:26 2019][1696405.972756] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696405.977917] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:26 2019][1696405.983668] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:26 2019][1696405.990388] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696405.995448] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.001734] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.008361] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.014649] watchdog/18 S ffff913aa9879040 0 101 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.021943] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.024586] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:26 2019][1696406.030079] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.035241] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:26 2019][1696406.041261] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.047288] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696406.052349] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.058643] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.065267] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.071542] migration/18 S ffff913aa98c8000 0 102 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.078836] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.081473] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.086622] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:26 2019][1696406.092652] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.098670] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696406.103741] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.110012] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.116649] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.122924] ksoftirqd/18 S ffff913aa98c9040 0 103 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.130199] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.132844] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.137987] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:26 2019][1696406.144019] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.150038] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696406.155098] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.161388] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.168008] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.174299] kworker/18:0H S ffff913aa98cb0c0 0 105 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.181574] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.184213] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:26 2019][1696406.190401] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.195562] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:26 2019][1696406.201321] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:26 2019][1696406.208042] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696406.213108] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.219396] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.226021] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.232310] watchdog/19 S ffff914aa942e180 0 106 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.239605] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.242251] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:26 2019][1696406.247744] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.252910] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:26 2019][1696406.258929] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.264941] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696406.270016] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.276289] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.282909] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.289197] migration/19 S ffff914aa942d140 0 107 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.296485] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.299112] [] ? cpu_stop_queue_work+0x70/0x70 [Wed Feb 6 09:38:26 2019][1696406.305395] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.310540] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:26 2019][1696406.316552] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.322579] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696406.327641] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.333917] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.340549] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.346830] ksoftirqd/19 S ffff914aa942c100 0 108 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.354128] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.356756] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.361897] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:26 2019][1696406.367930] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.373945] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696406.379009] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.385297] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.391924] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.398209] kworker/19:0H S ffff914aa9479040 0 110 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.405494] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.408147] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:26 2019][1696406.414335] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.419498] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:26 2019][1696406.425256] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:26 2019][1696406.431975] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696406.437031] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.443321] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.449946] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.456220] watchdog/20 S ffff911b2963e180 0 111 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.463525] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.466165] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:26 2019][1696406.471659] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.476819] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:26 2019][1696406.482833] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.488863] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696406.493918] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.500188] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.506823] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.513102] migration/20 S ffff911b2963d140 0 112 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.520402] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.523035] [] ? cpu_stop_queue_work+0x70/0x70 [Wed Feb 6 09:38:26 2019][1696406.529305] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.534467] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:26 2019][1696406.540476] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.546496] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696406.551570] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.557850] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.564486] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.570759] ksoftirqd/20 S ffff911b29688000 0 113 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.578044] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.580675] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.585822] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:26 2019][1696406.591856] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.597872] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696406.602935] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.609225] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.615847] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.622138] kworker/20:0H S ffff911b2968a080 0 115 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.629427] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.632062] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:26 2019][1696406.638266] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.643417] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:26 2019][1696406.649171] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:26 2019][1696406.655895] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696406.660958] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.667247] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.673865] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.680148] watchdog/21 S ffff912aa9996180 0 116 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.687433] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.690072] [] ? watchdog_cleanup+0x10/0x10 [Wed Feb 6 09:38:26 2019][1696406.696100] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:26 2019][1696406.701598] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.706743] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:26 2019][1696406.712772] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.718781] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696406.723847] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.730142] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.736760] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.743048] migration/21 S ffff912aa9995140 0 117 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.750330] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.752966] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.758126] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:26 2019][1696406.764143] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.770162] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696406.775219] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.781510] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.788127] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.794410] ksoftirqd/21 S ffff912aa9994100 0 118 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.801715] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.804353] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.809501] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:26 2019][1696406.815526] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.821537] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:26 2019][1696406.826613] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.832881] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:26 2019][1696406.839503] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:26 2019][1696406.845792] kworker/21:0H S ffff912aa9992080 0 120 2 0x00000000 [Wed Feb 6 09:38:26 2019][1696406.853088] Call Trace: [Wed Feb 6 09:38:26 2019][1696406.855721] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:26 2019][1696406.861909] [] schedule+0x29/0x70 [Wed Feb 6 09:38:26 2019][1696406.867073] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:26 2019][1696406.872831] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:27 2019][1696406.879537] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696406.884611] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696406.890884] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696406.897518] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696406.903799] watchdog/22 S ffff913aa98ce180 0 121 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696406.911094] Call Trace: [Wed Feb 6 09:38:27 2019][1696406.913730] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:27 2019][1696406.919221] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696406.924387] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:27 2019][1696406.930403] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696406.936418] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696406.941493] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696406.947765] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696406.954401] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696406.960676] migration/22 S ffff913aa98cd140 0 122 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696406.967969] Call Trace: [Wed Feb 6 09:38:27 2019][1696406.970596] [] ? cpu_stop_queue_work+0x70/0x70 [Wed Feb 6 09:38:27 2019][1696406.976865] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696406.982027] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:27 2019][1696406.988036] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696406.994054] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696406.999117] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.005391] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.012025] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.018304] ksoftirqd/22 S ffff913aa98cc100 0 123 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.025593] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.028225] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.033374] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:27 2019][1696407.039403] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.045414] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.050489] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.056768] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.063399] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.069672] kworker/22:0H S ffff913aa9919040 0 125 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.076970] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.079602] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:27 2019][1696407.085795] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.090955] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:27 2019][1696407.096713] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:27 2019][1696407.103436] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.108500] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.114787] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.121405] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.127691] watchdog/23 S ffff914aa947e180 0 126 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.134983] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.137636] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:27 2019][1696407.143133] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.148301] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:27 2019][1696407.154312] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.160326] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.165405] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.171683] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.178303] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.184592] migration/23 S ffff914aa947d140 0 127 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.191889] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.194527] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.199676] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:27 2019][1696407.205705] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.211724] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.216785] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.223073] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.229698] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.235986] ksoftirqd/23 S ffff914aa947c100 0 128 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.243275] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.245912] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.251062] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:27 2019][1696407.257088] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.263100] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.268162] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.274446] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.281068] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.287354] kworker/23:0H S ffff914aa947a080 0 130 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.294644] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.297292] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:27 2019][1696407.303478] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.308636] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:27 2019][1696407.314392] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:27 2019][1696407.321113] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.326177] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.332449] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.339083] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.345358] watchdog/24 S ffff911b2968e180 0 131 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.352643] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.355292] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:27 2019][1696407.360784] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.365947] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:27 2019][1696407.371965] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.377994] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.383058] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.389345] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.395961] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.402247] migration/24 S ffff911b2968d140 0 132 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.409520] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.412155] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.417302] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:27 2019][1696407.423331] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.429347] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.434424] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.440701] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.447337] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.453614] ksoftirqd/24 S ffff911b2968c100 0 133 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.460888] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.463536] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.468688] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:27 2019][1696407.474714] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.480722] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.485797] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.492066] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.498691] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.504980] kworker/24:0H S ffff911b296d8000 0 135 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.512260] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.514897] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:27 2019][1696407.521101] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.526254] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:27 2019][1696407.532007] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:27 2019][1696407.538730] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.543790] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.550083] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.556708] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.562982] watchdog/25 S ffff912aa99d0000 0 136 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.570283] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.572912] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:27 2019][1696407.578407] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.583573] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:27 2019][1696407.589590] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.595602] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.600676] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.606947] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.613570] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.619861] migration/25 S ffff912aa99d1040 0 137 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.627146] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.629790] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.634934] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:27 2019][1696407.640963] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.646982] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.652057] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.658336] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.664970] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.671240] ksoftirqd/25 S ffff912aa99d2080 0 138 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.678531] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.681175] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.686318] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:27 2019][1696407.692347] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.698355] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.703408] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.709700] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.716324] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.722602] kworker/25:0H S ffff912aa99d4100 0 140 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.729909] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.732538] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:27 2019][1696407.738728] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.743890] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:27 2019][1696407.749638] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:27 2019][1696407.756347] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.761424] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.767703] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.774326] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.780615] watchdog/26 S ffff913aa991e180 0 141 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.787890] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.790532] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:27 2019][1696407.796025] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.801188] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:27 2019][1696407.807206] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.813223] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.818298] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.824577] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:27 2019][1696407.831211] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.837489] migration/26 S ffff9159fc6bb200 0 142 2 0x00000000 [Wed Feb 6 09:38:27 2019][1696407.844788] Call Trace: [Wed Feb 6 09:38:27 2019][1696407.847416] [] schedule+0x29/0x70 [Wed Feb 6 09:38:27 2019][1696407.852561] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:27 2019][1696407.858591] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.864610] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:27 2019][1696407.869669] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:27 2019][1696407.875954] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696407.882574] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696407.888861] ksoftirqd/26 S ffff913aa991c100 0 143 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696407.896139] Call Trace: [Wed Feb 6 09:38:28 2019][1696407.898786] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696407.903939] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:28 2019][1696407.909966] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696407.915984] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696407.921062] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696407.927336] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696407.933970] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696407.940239] kworker/26:0H S ffff913aa991a080 0 145 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696407.947516] Call Trace: [Wed Feb 6 09:38:28 2019][1696407.950149] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:28 2019][1696407.956353] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696407.961506] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:28 2019][1696407.967257] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:28 2019][1696407.973981] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696407.979046] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696407.985317] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696407.991955] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696407.998233] watchdog/27 S ffff914aa94b8000 0 146 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.005524] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.008156] [] ? touch_nmi_watchdog+0x30/0x30 [Wed Feb 6 09:38:28 2019][1696408.014358] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:28 2019][1696408.019863] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.025011] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:28 2019][1696408.031044] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.037061] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696408.042124] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.048411] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696408.055026] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.061304] migration/27 S ffff9159fc663200 0 147 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.068605] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.071239] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.076379] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:28 2019][1696408.082413] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.088429] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696408.093491] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.099780] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696408.106405] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.112693] ksoftirqd/27 S ffff914aa94ba080 0 148 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.119978] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.122609] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.127763] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:28 2019][1696408.133787] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.139808] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696408.144882] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.151169] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696408.157809] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.164083] kworker/27:0H S ffff914aa94bc100 0 150 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.171380] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.174015] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:28 2019][1696408.180203] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.185367] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:28 2019][1696408.191125] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:28 2019][1696408.197834] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696408.202909] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.209186] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696408.215817] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.222089] watchdog/28 S ffff911b296de180 0 151 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.229380] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.232014] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:28 2019][1696408.237510] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.242672] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:28 2019][1696408.248690] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.254717] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696408.259775] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.266045] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696408.272683] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.278952] migration/28 S ffff911b296dd140 0 152 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.286228] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.288861] [] ? cpu_stop_queue_work+0x70/0x70 [Wed Feb 6 09:38:28 2019][1696408.295150] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.300300] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:28 2019][1696408.306317] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.312343] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696408.317398] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.323688] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696408.330304] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.336579] ksoftirqd/28 S ffff911b296dc100 0 153 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.343891] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.346528] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.351680] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:28 2019][1696408.357706] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.363720] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696408.368787] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.375064] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696408.381682] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.387972] kworker/28:0 S ffff911b296db0c0 0 154 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.395253] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.397886] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:28 2019][1696408.404082] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.409225] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:28 2019][1696408.414980] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:28 2019][1696408.421700] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696408.426756] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.433043] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696408.439665] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.445939] kworker/28:0H S ffff911b296da080 0 155 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.453243] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.455877] [] ? start_worker+0x27/0x30 [Wed Feb 6 09:38:28 2019][1696408.461546] [] ? manage_workers.isra.25+0x189/0x2a0 [Wed Feb 6 09:38:28 2019][1696408.468270] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.473417] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:28 2019][1696408.479178] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:28 2019][1696408.485894] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696408.490950] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.497232] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696408.503864] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.510134] watchdog/29 S ffff912aa99d6180 0 156 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.517437] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.520071] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:28 2019][1696408.525559] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.530724] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:28 2019][1696408.536738] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.542757] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696408.547832] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.554112] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696408.560748] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.567020] migration/29 S ffff912aa99d5140 0 157 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.574297] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.576928] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.582090] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:28 2019][1696408.588099] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.594118] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696408.599194] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.605473] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696408.612105] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.618380] ksoftirqd/29 S ffff912aa9a28000 0 158 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.625649] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.628293] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.633436] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:28 2019][1696408.639463] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.645476] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696408.650536] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.656826] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696408.663453] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.669727] kworker/29:0H S ffff912aa9a2a080 0 160 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.677034] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.679668] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:28 2019][1696408.685858] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.691017] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:28 2019][1696408.696767] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:28 2019][1696408.703485] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696408.708543] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.714817] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696408.721457] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.727735] watchdog/30 S ffff913aa9960000 0 161 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.735025] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.737668] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:28 2019][1696408.743161] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.748318] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:28 2019][1696408.754329] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.760345] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696408.765421] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.771689] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696408.778317] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.784601] migration/30 S ffff913aa9961040 0 162 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.791879] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.794525] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.799666] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:28 2019][1696408.805698] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.811715] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:28 2019][1696408.816779] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.823065] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:28 2019][1696408.829683] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:28 2019][1696408.835973] ksoftirqd/30 S ffff913aa9962080 0 163 2 0x00000000 [Wed Feb 6 09:38:28 2019][1696408.843255] Call Trace: [Wed Feb 6 09:38:28 2019][1696408.845899] [] schedule+0x29/0x70 [Wed Feb 6 09:38:28 2019][1696408.851045] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:28 2019][1696408.857071] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696408.863085] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696408.868147] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696408.874435] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696408.881059] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696408.887351] kworker/30:0H S ffff913aa9964100 0 165 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696408.894642] Call Trace: [Wed Feb 6 09:38:29 2019][1696408.897275] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:29 2019][1696408.903465] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696408.908627] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:29 2019][1696408.914382] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:29 2019][1696408.921108] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696408.926171] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696408.932447] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696408.939082] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696408.945355] watchdog/31 S ffff914aa94be180 0 166 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696408.952642] Call Trace: [Wed Feb 6 09:38:29 2019][1696408.955285] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:29 2019][1696408.960775] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696408.965938] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:29 2019][1696408.971947] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696408.977964] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696408.983040] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696408.989318] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696408.995954] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.002230] migration/31 S ffff914aa94bd140 0 167 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.009513] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.012142] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.017297] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:29 2019][1696409.023309] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.029323] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696409.034400] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.040678] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696409.047300] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.053588] ksoftirqd/31 S ffff914aa94f0000 0 168 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.060867] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.063499] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.068642] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:29 2019][1696409.074674] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.080692] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696409.085755] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.092042] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696409.098666] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.104957] kworker/31:0H S ffff914aa94f2080 0 170 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.112242] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.114877] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:29 2019][1696409.121064] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.126226] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:29 2019][1696409.131983] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:29 2019][1696409.138693] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696409.143764] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.150034] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696409.156671] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.162940] watchdog/32 S ffff911b296d9040 0 171 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.170237] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.172888] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:29 2019][1696409.178385] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.183548] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:29 2019][1696409.189566] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.195581] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696409.200655] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.206922] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696409.213558] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.219828] migration/32 S ffff911b29748000 0 172 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.227116] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.229770] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.234920] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:29 2019][1696409.240949] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.246964] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696409.252038] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.258309] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696409.264930] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.271220] ksoftirqd/32 S ffff911b29749040 0 173 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.278508] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.281138] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.286289] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:29 2019][1696409.292317] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.298337] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696409.303397] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.309681] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696409.316298] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.322578] kworker/32:0H S ffff911b2974b0c0 0 175 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.329881] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.332518] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:29 2019][1696409.338707] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.343868] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:29 2019][1696409.349628] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:29 2019][1696409.356348] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696409.361411] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.367685] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696409.374321] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.380591] watchdog/33 S ffff912aa9a2e180 0 176 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.387878] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.390534] [] ? watchdog_cleanup+0x10/0x10 [Wed Feb 6 09:38:29 2019][1696409.396547] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:29 2019][1696409.402058] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.407211] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:29 2019][1696409.413223] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.419250] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696409.424310] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.430585] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696409.437225] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.443495] migration/33 S ffff912aa9a2d140 0 177 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.450797] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.453429] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.458574] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:29 2019][1696409.464609] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.470625] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696409.475691] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.481984] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696409.488607] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.494894] ksoftirqd/33 S ffff912aa9a2c100 0 178 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.502165] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.504797] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.509951] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:29 2019][1696409.515986] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.522006] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696409.527080] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.533353] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696409.539994] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.546268] kworker/33:0H S ffff912aa9a58000 0 180 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.553549] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.556183] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:29 2019][1696409.562390] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.567541] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:29 2019][1696409.573312] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:29 2019][1696409.580033] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696409.585099] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.591384] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696409.598000] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.604289] watchdog/34 S ffff913aa9966180 0 181 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.611558] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.614192] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:29 2019][1696409.619687] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.624851] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:29 2019][1696409.630868] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.636897] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696409.641963] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.648246] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696409.654865] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.661150] migration/34 S ffff913aa9965140 0 182 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.668433] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.671065] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.676228] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:29 2019][1696409.682245] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.688258] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696409.693332] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.699606] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696409.706243] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.712519] ksoftirqd/34 S ffff913aa9990000 0 183 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.719809] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.722439] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.727591] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:29 2019][1696409.733620] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.739638] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696409.744712] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.750992] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696409.757624] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.763892] kworker/34:0H S ffff913aa9992080 0 185 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.771200] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.773835] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:29 2019][1696409.780026] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.785191] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:29 2019][1696409.790948] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:29 2019][1696409.797671] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:29 2019][1696409.802733] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.809024] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:29 2019][1696409.815654] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.821941] watchdog/35 S ffff914aa94f6180 0 186 2 0x00000000 [Wed Feb 6 09:38:29 2019][1696409.829221] Call Trace: [Wed Feb 6 09:38:29 2019][1696409.831864] [] ? touch_nmi_watchdog+0x30/0x30 [Wed Feb 6 09:38:29 2019][1696409.838051] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:29 2019][1696409.843560] [] schedule+0x29/0x70 [Wed Feb 6 09:38:29 2019][1696409.848711] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:29 2019][1696409.854740] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:29 2019][1696409.860757] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696409.865835] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696409.872112] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696409.878748] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696409.885021] migration/35 S ffff914aa94f5140 0 187 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696409.892326] Call Trace: [Wed Feb 6 09:38:30 2019][1696409.894965] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696409.900113] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:30 2019][1696409.906138] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696409.912146] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696409.917213] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696409.923502] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696409.930128] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696409.936396] ksoftirqd/35 S ffff914aa94f4100 0 188 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696409.943682] Call Trace: [Wed Feb 6 09:38:30 2019][1696409.946316] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696409.951463] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:30 2019][1696409.957495] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696409.963513] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696409.968572] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696409.974860] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696409.981486] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696409.987772] kworker/35:0H S ffff914aa9520000 0 190 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696409.995046] Call Trace: [Wed Feb 6 09:38:30 2019][1696409.997682] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:30 2019][1696410.003874] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.009037] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:30 2019][1696410.014795] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:30 2019][1696410.021518] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.026585] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.032883] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.039503] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.045792] watchdog/36 S ffff911b2974e180 0 191 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.053077] Call Trace: [Wed Feb 6 09:38:30 2019][1696410.055710] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:30 2019][1696410.061207] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.066368] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:30 2019][1696410.072385] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.078409] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.083469] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.089742] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.096382] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.102657] migration/36 S ffff911b2974d140 0 192 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.109939] Call Trace: [Wed Feb 6 09:38:30 2019][1696410.112583] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.117725] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:30 2019][1696410.123756] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.129767] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.134832] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.141119] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.147745] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.154033] ksoftirqd/36 S ffff911b2974c100 0 193 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.161315] Call Trace: [Wed Feb 6 09:38:30 2019][1696410.163947] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.169111] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:30 2019][1696410.175128] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.181141] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.186222] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.192496] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.199122] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.205412] kworker/36:0H S ffff911b297c1040 0 195 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.212701] Call Trace: [Wed Feb 6 09:38:30 2019][1696410.215346] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:30 2019][1696410.221530] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.226693] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:30 2019][1696410.232443] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:30 2019][1696410.239150] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.244230] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.250502] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.257143] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.263420] watchdog/37 S ffff912aa9a5e180 0 196 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.270713] Call Trace: [Wed Feb 6 09:38:30 2019][1696410.273354] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:30 2019][1696410.278844] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.284005] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:30 2019][1696410.290014] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.296021] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.301101] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.307377] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.314000] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.320288] migration/37 S ffff912aa9a5d140 0 197 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.327566] Call Trace: [Wed Feb 6 09:38:30 2019][1696410.330198] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.335356] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:30 2019][1696410.341371] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.347382] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.352458] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.358727] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.365353] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.371639] ksoftirqd/37 S ffff912aa9a5c100 0 198 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.378916] Call Trace: [Wed Feb 6 09:38:30 2019][1696410.381560] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.386702] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:30 2019][1696410.392721] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.398735] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.403797] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.410081] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.416703] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.422992] kworker/37:0H S ffff912aa9a5a080 0 200 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.430284] Call Trace: [Wed Feb 6 09:38:30 2019][1696410.432930] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:30 2019][1696410.439114] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.444276] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:30 2019][1696410.450027] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:30 2019][1696410.456749] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.461814] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.468101] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.474719] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.481006] watchdog/38 S ffff913aa9996180 0 201 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.488288] Call Trace: [Wed Feb 6 09:38:30 2019][1696410.490937] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:30 2019][1696410.496425] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.501591] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:30 2019][1696410.507609] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.513625] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.518699] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.524975] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.531611] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.537879] migration/38 S ffff913aa9995140 0 202 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.545177] Call Trace: [Wed Feb 6 09:38:30 2019][1696410.547830] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.552983] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:30 2019][1696410.559005] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.565017] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.570093] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.576373] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.582993] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.589283] ksoftirqd/38 S ffff913aa9994100 0 203 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.596567] Call Trace: [Wed Feb 6 09:38:30 2019][1696410.599196] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.604353] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:30 2019][1696410.610374] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.616387] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.621460] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.627733] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.634366] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.640636] kworker/38:0H S ffff913aa99c8000 0 205 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.647941] Call Trace: [Wed Feb 6 09:38:30 2019][1696410.650575] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:30 2019][1696410.656764] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.661926] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:30 2019][1696410.667676] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:30 2019][1696410.674387] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.679460] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.685730] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.692367] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.698644] watchdog/39 S ffff914aa9526180 0 206 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.705933] Call Trace: [Wed Feb 6 09:38:30 2019][1696410.708565] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:30 2019][1696410.714059] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.719216] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:30 2019][1696410.725238] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.731251] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.736326] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.742600] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.749246] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.755515] migration/39 S ffff914aa9525140 0 207 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.762801] Call Trace: [Wed Feb 6 09:38:30 2019][1696410.765448] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.770599] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:30 2019][1696410.776626] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.782641] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.787719] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.793995] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.800619] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.806903] ksoftirqd/39 S ffff914aa9524100 0 208 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.814187] Call Trace: [Wed Feb 6 09:38:30 2019][1696410.816832] [] schedule+0x29/0x70 [Wed Feb 6 09:38:30 2019][1696410.821983] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:30 2019][1696410.828011] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.834016] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:30 2019][1696410.839083] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.845355] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:30 2019][1696410.851977] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:30 2019][1696410.858269] kworker/39:0H S ffff914aa9522080 0 210 2 0x00000000 [Wed Feb 6 09:38:30 2019][1696410.865558] Call Trace: [Wed Feb 6 09:38:31 2019][1696410.868192] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:31 2019][1696410.874384] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696410.879545] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:31 2019][1696410.885300] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:31 2019][1696410.892021] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696410.897081] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696410.903359] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696410.909990] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696410.916260] watchdog/40 S ffff911b297c6180 0 211 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696410.923578] Call Trace: [Wed Feb 6 09:38:31 2019][1696410.926216] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:31 2019][1696410.931714] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696410.936875] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:31 2019][1696410.942887] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696410.948914] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696410.953980] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696410.960266] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696410.966880] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696410.973162] migration/40 S ffff911b297c5140 0 212 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696410.980434] Call Trace: [Wed Feb 6 09:38:31 2019][1696410.983067] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696410.988214] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:31 2019][1696410.994248] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.000260] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.005336] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.011612] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.018251] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.024525] ksoftirqd/40 S ffff911b297c4100 0 213 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.031804] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.034449] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.039601] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:31 2019][1696411.045624] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.051638] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.056710] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.062981] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.069622] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.075898] kworker/40:0H S ffff911b297c2080 0 215 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.083183] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.085830] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:31 2019][1696411.092017] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.097181] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:31 2019][1696411.102939] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:31 2019][1696411.109661] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.114725] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.121014] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.127640] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.133926] watchdog/41 S ffff912aa9a59040 0 216 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.141203] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.143850] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:31 2019][1696411.149348] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.154510] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:31 2019][1696411.160527] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.166555] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.171609] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.177881] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.184514] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.190783] migration/41 S ffff912aa9aa0000 0 217 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.198088] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.200726] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.205870] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:31 2019][1696411.211889] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.217900] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.222962] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.229261] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.235880] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.242151] ksoftirqd/41 S ffff912aa9aa1040 0 218 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.249457] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.252094] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.257248] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:31 2019][1696411.263278] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.269286] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.274341] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.280632] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.287252] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.293531] kworker/41:0H S ffff912aa9aa30c0 0 220 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.300834] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.303469] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:31 2019][1696411.309657] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.314825] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:31 2019][1696411.320579] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:31 2019][1696411.327287] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.332363] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.338632] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.345273] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.351549] watchdog/42 S ffff913aa99ce180 0 221 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.358839] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.361475] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:31 2019][1696411.366965] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.372131] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:31 2019][1696411.378145] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.384177] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.389257] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.395542] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.402159] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.408433] migration/42 S ffff913aa99cd140 0 222 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.415728] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.418355] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.423501] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:31 2019][1696411.429531] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.435549] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.440624] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.446903] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.453537] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.459812] ksoftirqd/42 S ffff913aa99cc100 0 223 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.467103] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.469742] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.474895] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:31 2019][1696411.480924] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.486938] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.492014] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.498281] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.504899] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.511189] kworker/42:0H S ffff913aa99ca080 0 225 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.518481] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.521114] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:31 2019][1696411.527316] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.532464] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:31 2019][1696411.538221] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:31 2019][1696411.544947] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.550005] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.556278] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.562911] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.569190] watchdog/43 S ffff914aa9521040 0 226 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.576492] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.579117] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:31 2019][1696411.584605] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.589767] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:31 2019][1696411.595776] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.601795] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.606869] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.613141] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.619772] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.626040] migration/43 S ffff914aa9588000 0 227 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.633334] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.635969] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.641117] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:31 2019][1696411.647146] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.653162] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.658238] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.664516] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.671150] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.677418] ksoftirqd/43 S ffff914aa9589040 0 228 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.684711] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.687341] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.692486] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:31 2019][1696411.698514] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.704532] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.709608] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.715883] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.722518] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.728788] kworker/43:0H S ffff914aa958b0c0 0 230 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.736068] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.738717] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:31 2019][1696411.744904] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.750067] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:31 2019][1696411.755823] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:31 2019][1696411.762544] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.767604] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.773874] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.780510] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.786784] watchdog/44 S ffff911b29028000 0 231 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.794065] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.796720] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:31 2019][1696411.802214] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.807379] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:31 2019][1696411.813393] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.819410] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:31 2019][1696411.824487] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.830767] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:31 2019][1696411.837388] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:31 2019][1696411.843675] migration/44 S ffff911b29029040 0 232 2 0x00000000 [Wed Feb 6 09:38:31 2019][1696411.850952] Call Trace: [Wed Feb 6 09:38:31 2019][1696411.853599] [] schedule+0x29/0x70 [Wed Feb 6 09:38:31 2019][1696411.858753] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:31 2019][1696411.864783] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696411.870800] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696411.875873] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696411.882153] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696411.888790] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696411.895066] ksoftirqd/44 S ffff911b2902a080 0 233 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696411.902363] Call Trace: [Wed Feb 6 09:38:32 2019][1696411.904998] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696411.910150] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:32 2019][1696411.916184] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696411.922199] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696411.927284] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696411.933556] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696411.940177] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696411.946468] kworker/44:0H S ffff911b2902c100 0 235 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696411.953745] Call Trace: [Wed Feb 6 09:38:32 2019][1696411.956393] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:32 2019][1696411.962576] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696411.967740] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:32 2019][1696411.973494] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:32 2019][1696411.980200] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696411.985280] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696411.991556] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696411.998185] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.004460] watchdog/45 S ffff912aa9aa6180 0 236 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.011750] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.014380] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:32 2019][1696412.019874] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.025041] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:32 2019][1696412.031053] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.037068] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.042144] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.048420] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696412.055049] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.061336] migration/45 S ffff912aa9aa5140 0 237 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.068611] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.071243] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.076408] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:32 2019][1696412.082422] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.088440] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.093515] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.099788] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696412.106422] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.112700] ksoftirqd/45 S ffff912aa9aa4100 0 238 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.119998] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.122628] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.127778] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:32 2019][1696412.133811] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.139823] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.144885] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.151174] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696412.157796] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.164068] kworker/45:0H S ffff912aa9ad1040 0 240 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.171373] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.174005] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:32 2019][1696412.180195] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.185356] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:32 2019][1696412.191115] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:32 2019][1696412.197822] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.202902] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.209182] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696412.215811] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.222097] watchdog/46 S ffff913aa99c9040 0 241 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.229397] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.232033] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:32 2019][1696412.237540] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.242687] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:32 2019][1696412.248702] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.254730] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.259794] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.266083] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696412.272708] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.278995] migration/46 S ffff913aa9a30000 0 242 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.286263] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.288910] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.294063] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:32 2019][1696412.300091] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.306111] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.311181] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.317469] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696412.324095] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.330384] ksoftirqd/46 S ffff913aa9a31040 0 243 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.337668] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.340317] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.345465] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:32 2019][1696412.351497] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.357514] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.362576] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.368865] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696412.375488] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.381777] kworker/46:0H S ffff913aa9a330c0 0 245 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.389067] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.391714] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:32 2019][1696412.397904] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.403061] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:32 2019][1696412.408815] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:32 2019][1696412.415535] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.420597] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.426886] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696412.433503] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.439775] watchdog/47 S ffff914aa958e180 0 246 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.447082] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.449722] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:32 2019][1696412.455221] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.460385] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:32 2019][1696412.466401] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.472415] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.477492] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.483766] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696412.490394] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.496687] migration/47 S ffff914aa958d140 0 247 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.503968] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.506612] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.511754] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:32 2019][1696412.517787] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.523807] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.528861] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.535158] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696412.541782] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.548055] ksoftirqd/47 S ffff914aa958c100 0 248 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.555344] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.557973] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.563120] [] smpboot_thread_fn+0xe2/0x1a0 [Wed Feb 6 09:38:32 2019][1696412.569149] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.575165] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.580241] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.586521] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696412.593154] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.599432] kworker/47:0H S ffff914aa95b9040 0 250 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.606729] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.609361] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:32 2019][1696412.615549] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.620714] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:32 2019][1696412.626473] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:32 2019][1696412.633182] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.638257] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.644535] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696412.651172] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.657447] kdevtmpfs S ffff911b2911a080 0 255 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.664736] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.667374] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.672530] [] devtmpfsd+0x16d/0x180 [Wed Feb 6 09:38:32 2019][1696412.677956] [] ? handle_create.isra.2+0x220/0x220 [Wed Feb 6 09:38:32 2019][1696412.684487] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.689564] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.695845] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696412.702465] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.708748] netns S ffff911b2911b0c0 0 256 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.716046] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.718692] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.723845] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:32 2019][1696412.729701] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:32 2019][1696412.735893] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.740962] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.747237] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696412.753870] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.760142] khungtaskd S ffff911b2911c100 0 257 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.767440] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.770071] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.775218] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:38:32 2019][1696412.781251] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:38:32 2019][1696412.787791] [] schedule_timeout_interruptible+0x1e/0x20 [Wed Feb 6 09:38:32 2019][1696412.794847] [] watchdog+0x4c/0x2c0 [Wed Feb 6 09:38:32 2019][1696412.800096] [] ? reset_hung_task_detector+0x20/0x20 [Wed Feb 6 09:38:32 2019][1696412.806806] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.811881] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.818154] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:32 2019][1696412.824797] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.831073] writeback S ffff911b2911d140 0 258 2 0x00000000 [Wed Feb 6 09:38:32 2019][1696412.838363] Call Trace: [Wed Feb 6 09:38:32 2019][1696412.841012] [] schedule+0x29/0x70 [Wed Feb 6 09:38:32 2019][1696412.846163] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:32 2019][1696412.852018] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:32 2019][1696412.858209] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:32 2019][1696412.863280] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:32 2019][1696412.869557] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696412.876201] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696412.882475] kintegrityd S ffff911b2911e180 0 259 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696412.889779] Call Trace: [Wed Feb 6 09:38:33 2019][1696412.892414] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696412.897563] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:33 2019][1696412.903419] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:33 2019][1696412.909606] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696412.914666] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696412.920954] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696412.927579] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696412.933867] bioset S ffff911b29119040 0 260 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696412.941164] Call Trace: [Wed Feb 6 09:38:33 2019][1696412.943812] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696412.948956] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:33 2019][1696412.954812] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:33 2019][1696412.960997] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696412.966055] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696412.972350] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696412.978975] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696412.985256] bioset S ffff911b29118000 0 261 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696412.992558] Call Trace: [Wed Feb 6 09:38:33 2019][1696412.995194] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696413.000342] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:33 2019][1696413.006197] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:33 2019][1696413.012379] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696413.017446] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.023734] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696413.030360] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.036647] bioset S ffff9129ff078000 0 262 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696413.043936] Call Trace: [Wed Feb 6 09:38:33 2019][1696413.046586] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696413.051727] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:33 2019][1696413.057585] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:33 2019][1696413.063778] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696413.068836] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.075131] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696413.081754] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.088039] kblockd S ffff9129ff079040 0 263 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696413.095339] Call Trace: [Wed Feb 6 09:38:33 2019][1696413.097973] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696413.103122] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:33 2019][1696413.108983] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:33 2019][1696413.115172] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696413.120235] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.126525] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696413.133148] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.139436] md S ffff9129ff07a080 0 264 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696413.146731] Call Trace: [Wed Feb 6 09:38:33 2019][1696413.149379] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696413.154530] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:33 2019][1696413.160384] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:33 2019][1696413.166571] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696413.171649] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.177919] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696413.184543] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.190831] edac-poller S ffff9129ff07b0c0 0 265 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696413.198119] Call Trace: [Wed Feb 6 09:38:33 2019][1696413.200770] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696413.205911] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:33 2019][1696413.211770] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:33 2019][1696413.217960] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696413.223021] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.229314] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696413.235940] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.242212] watchdogd S ffff9129ff07c100 0 266 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696413.249520] Call Trace: [Wed Feb 6 09:38:33 2019][1696413.252149] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696413.257301] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:33 2019][1696413.263155] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:33 2019][1696413.269346] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696413.274406] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.280697] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696413.287324] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.293607] irq/26-AMD-Vi S ffff9129ff07e180 0 268 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696413.300906] Call Trace: [Wed Feb 6 09:38:33 2019][1696413.303536] [] ? irq_finalize_oneshot.part.31+0xf0/0xf0 [Wed Feb 6 09:38:33 2019][1696413.310593] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696413.315757] [] irq_thread+0xd1/0x180 [Wed Feb 6 09:38:33 2019][1696413.321168] [] ? irq_forced_thread_fn+0x50/0x50 [Wed Feb 6 09:38:33 2019][1696413.327544] [] ? irq_thread_check_affinity+0xf0/0xf0 [Wed Feb 6 09:38:33 2019][1696413.334343] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696413.339418] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.345697] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696413.352330] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.358607] irq/27-AMD-Vi S ffff9129fc018000 0 269 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696413.365908] Call Trace: [Wed Feb 6 09:38:33 2019][1696413.368541] [] ? irq_finalize_oneshot.part.31+0xf0/0xf0 [Wed Feb 6 09:38:33 2019][1696413.375596] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696413.380758] [] irq_thread+0xd1/0x180 [Wed Feb 6 09:38:33 2019][1696413.386171] [] ? irq_forced_thread_fn+0x50/0x50 [Wed Feb 6 09:38:33 2019][1696413.392546] [] ? irq_thread_check_affinity+0xf0/0xf0 [Wed Feb 6 09:38:33 2019][1696413.399344] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696413.404414] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.410688] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696413.417323] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.423601] irq/28-AMD-Vi S ffff9129fc019040 0 270 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696413.430899] Call Trace: [Wed Feb 6 09:38:33 2019][1696413.433533] [] ? irq_finalize_oneshot.part.31+0xf0/0xf0 [Wed Feb 6 09:38:33 2019][1696413.440589] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696413.445749] [] irq_thread+0xd1/0x180 [Wed Feb 6 09:38:33 2019][1696413.451163] [] ? irq_forced_thread_fn+0x50/0x50 [Wed Feb 6 09:38:33 2019][1696413.457537] [] ? irq_thread_check_affinity+0xf0/0xf0 [Wed Feb 6 09:38:33 2019][1696413.464338] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696413.469412] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.475690] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696413.482325] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.488601] irq/29-AMD-Vi S ffff9129fc01a080 0 271 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696413.495903] Call Trace: [Wed Feb 6 09:38:33 2019][1696413.498535] [] ? irq_finalize_oneshot.part.31+0xf0/0xf0 [Wed Feb 6 09:38:33 2019][1696413.505585] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696413.510750] [] irq_thread+0xd1/0x180 [Wed Feb 6 09:38:33 2019][1696413.516162] [] ? irq_forced_thread_fn+0x50/0x50 [Wed Feb 6 09:38:33 2019][1696413.522534] [] ? irq_thread_check_affinity+0xf0/0xf0 [Wed Feb 6 09:38:33 2019][1696413.529342] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696413.534406] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.540692] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696413.547310] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.553600] kswapd0 S ffff9129fc130000 0 276 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696413.560887] Call Trace: [Wed Feb 6 09:38:33 2019][1696413.563518] [] ? calculate_normal_threshold+0x27/0x50 [Wed Feb 6 09:38:33 2019][1696413.570401] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696413.575560] [] kswapd+0x38f/0x440 [Wed Feb 6 09:38:33 2019][1696413.580705] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:33 2019][1696413.586733] [] ? balance_pgdat+0x5e0/0x5e0 [Wed Feb 6 09:38:33 2019][1696413.592665] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696413.597739] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.604012] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696413.610641] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.616916] kswapd1 S ffff9129fc131040 0 277 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696413.624203] Call Trace: [Wed Feb 6 09:38:33 2019][1696413.626834] [] ? calculate_normal_threshold+0x27/0x50 [Wed Feb 6 09:38:33 2019][1696413.633715] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696413.638882] [] kswapd+0x38f/0x440 [Wed Feb 6 09:38:33 2019][1696413.644034] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:33 2019][1696413.650049] [] ? balance_pgdat+0x5e0/0x5e0 [Wed Feb 6 09:38:33 2019][1696413.655991] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696413.661052] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.667340] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696413.673956] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.680225] kswapd2 S ffff9129fc132080 0 278 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696413.687517] Call Trace: [Wed Feb 6 09:38:33 2019][1696413.690151] [] ? calculate_normal_threshold+0x27/0x50 [Wed Feb 6 09:38:33 2019][1696413.697031] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696413.702194] [] kswapd+0x38f/0x440 [Wed Feb 6 09:38:33 2019][1696413.707345] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:33 2019][1696413.713370] [] ? balance_pgdat+0x5e0/0x5e0 [Wed Feb 6 09:38:33 2019][1696413.719293] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696413.724368] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.730642] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696413.737261] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.743553] kswapd3 S ffff9129fc1330c0 0 279 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696413.750850] Call Trace: [Wed Feb 6 09:38:33 2019][1696413.753484] [] ? calculate_normal_threshold+0x27/0x50 [Wed Feb 6 09:38:33 2019][1696413.760365] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696413.765523] [] kswapd+0x38f/0x440 [Wed Feb 6 09:38:33 2019][1696413.770677] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:33 2019][1696413.776688] [] ? balance_pgdat+0x5e0/0x5e0 [Wed Feb 6 09:38:33 2019][1696413.782632] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696413.787696] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.793986] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696413.800611] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.806893] ksmd S ffff9129fc134100 0 280 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696413.814175] Call Trace: [Wed Feb 6 09:38:33 2019][1696413.816818] [] schedule+0x29/0x70 [Wed Feb 6 09:38:33 2019][1696413.821966] [] ksm_scan_thread+0x16a/0x240 [Wed Feb 6 09:38:33 2019][1696413.827907] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:33 2019][1696413.833925] [] ? ksm_do_scan+0x11b0/0x11b0 [Wed Feb 6 09:38:33 2019][1696413.839867] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:33 2019][1696413.844934] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.851221] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:33 2019][1696413.857846] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:33 2019][1696413.864132] khugepaged S ffff9129fc135140 0 281 2 0x00000000 [Wed Feb 6 09:38:33 2019][1696413.871411] Call Trace: [Wed Feb 6 09:38:33 2019][1696413.874057] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696413.879211] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:38:34 2019][1696413.885237] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:38:34 2019][1696413.891766] [] khugepaged+0x226/0x480 [Wed Feb 6 09:38:34 2019][1696413.897274] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:34 2019][1696413.903283] [] ? khugepaged_scan_mm_slot+0xf00/0xf00 [Wed Feb 6 09:38:34 2019][1696413.910084] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696413.915152] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696413.921426] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696413.928060] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696413.934329] crypto S ffff9129fc136180 0 282 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696413.941636] Call Trace: [Wed Feb 6 09:38:34 2019][1696413.944271] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696413.949416] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:34 2019][1696413.955275] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:34 2019][1696413.961466] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696413.966528] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696413.972816] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696413.979435] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696413.985722] kthrotld S ffff9129fc1a8000 0 290 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696413.993024] Call Trace: [Wed Feb 6 09:38:34 2019][1696413.995668] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.000811] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:34 2019][1696414.006672] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:34 2019][1696414.012862] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.017924] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.024214] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.030837] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.037125] kworker/3:1 S ffff9159fce68000 0 293 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.044406] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.047039] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:34 2019][1696414.053235] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.058381] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:34 2019][1696414.064133] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:34 2019][1696414.070857] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.075925] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.082197] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.088832] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.095106] kmpath_rdacd S ffff9129fc1ad140 0 295 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.102396] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.105028] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.110170] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:34 2019][1696414.116024] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:34 2019][1696414.122207] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.127266] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.133553] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.140179] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.146466] kaluad S ffff9129fc1ac100 0 296 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.153749] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.156394] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.161540] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:34 2019][1696414.167395] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:34 2019][1696414.173582] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.178643] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.184930] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.191555] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.197840] kpsmoused S ffff9129fc1aa080 0 299 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.205116] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.207765] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.212911] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:34 2019][1696414.218765] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:34 2019][1696414.224949] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.230026] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.236299] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.242924] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.249214] ipv6_addrconf S ffff911b2902e180 0 302 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.256494] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.259130] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.264292] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:34 2019][1696414.270128] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:34 2019][1696414.276324] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.281383] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.287650] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.294287] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.300558] deferwq S ffff9129fb858000 0 315 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.307847] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.310498] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.315650] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:34 2019][1696414.321504] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:34 2019][1696414.327694] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.332770] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.339049] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.345683] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.351953] kworker/24:1 S ffff9129fb85b0c0 0 318 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.359257] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.361889] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:34 2019][1696414.368074] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.373236] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:34 2019][1696414.378994] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:34 2019][1696414.385719] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.390781] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.397057] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.403691] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.409958] kworker/33:1 S ffff912aa9ad5140 0 321 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.417253] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.419885] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:34 2019][1696414.426070] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.431235] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:34 2019][1696414.436993] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:34 2019][1696414.443702] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.448777] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.455055] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.461687] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.467960] kauditd S ffff9129fb99a080 0 404 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.475254] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.477907] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.483060] [] kauditd_thread+0x184/0x1e0 [Wed Feb 6 09:38:34 2019][1696414.488910] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:34 2019][1696414.494665] [] ? audit_printk_skb+0x70/0x70 [Wed Feb 6 09:38:34 2019][1696414.500690] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.505744] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.512036] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.518663] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.524936] scsi_eh_0 S ffff9139f89430c0 0 2282 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.532240] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.534873] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.540024] [] scsi_error_handler+0xa3/0x8b0 [Wed Feb 6 09:38:34 2019][1696414.546136] [] ? scsi_eh_get_sense+0x250/0x250 [Wed Feb 6 09:38:34 2019][1696414.552409] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.557473] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.563759] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.570376] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.576666] scsi_tmf_0 S ffff9139f8944100 0 2291 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.583953] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.586593] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.591737] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:34 2019][1696414.597596] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:34 2019][1696414.603787] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.608846] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.615134] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.621754] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.628029] scsi_eh_1 S ffff9129fb160000 0 2380 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.635329] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.637960] [] ? finish_task_switch+0x54/0x1c0 [Wed Feb 6 09:38:34 2019][1696414.644235] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.649398] [] scsi_error_handler+0xa3/0x8b0 [Wed Feb 6 09:38:34 2019][1696414.655502] [] ? scsi_eh_get_sense+0x250/0x250 [Wed Feb 6 09:38:34 2019][1696414.661786] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.666847] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.673136] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.679761] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.686049] scsi_tmf_1 S ffff9129fbaf8000 0 2384 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.693337] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.695982] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.701133] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:34 2019][1696414.706987] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:34 2019][1696414.713169] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.718249] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.724524] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.731148] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.737435] ata_sff S ffff9149fa9a0000 0 4059 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.744724] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.747360] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.752508] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:34 2019][1696414.758367] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:34 2019][1696414.764557] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.769616] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.775906] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.782525] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.788799] poll_mpt3sas0_h S ffff9129f3d230c0 0 5684 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.796103] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.798737] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.803884] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:34 2019][1696414.809740] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:34 2019][1696414.815922] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:34 2019][1696414.820976] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.827266] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:34 2019][1696414.833882] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:34 2019][1696414.840157] kworker/5:1 S ffff9139f61f0000 0 7260 2 0x00000000 [Wed Feb 6 09:38:34 2019][1696414.847466] Call Trace: [Wed Feb 6 09:38:34 2019][1696414.850099] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:34 2019][1696414.856289] [] schedule+0x29/0x70 [Wed Feb 6 09:38:34 2019][1696414.861449] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:34 2019][1696414.867197] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:34 2019][1696414.873918] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696414.878974] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696414.885251] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696414.891890] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696414.898167] kworker/15:1 S ffff9159f6629040 0 7264 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696414.905455] Call Trace: [Wed Feb 6 09:38:35 2019][1696414.908091] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:35 2019][1696414.914294] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696414.919441] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:35 2019][1696414.925196] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:35 2019][1696414.932013] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696414.937104] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696414.943392] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696414.950012] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696414.956286] kworker/19:1 S ffff9159f662e180 0 7267 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696414.963590] Call Trace: [Wed Feb 6 09:38:35 2019][1696414.966225] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:35 2019][1696414.972412] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696414.977565] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:35 2019][1696414.983315] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:35 2019][1696414.990018] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696414.995095] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.001372] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.008006] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.014275] kworker/26:1 S ffff9149f3e9d140 0 7271 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696415.021563] Call Trace: [Wed Feb 6 09:38:35 2019][1696415.024197] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:35 2019][1696415.030398] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696415.035542] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:35 2019][1696415.041299] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:35 2019][1696415.048022] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696415.053078] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.059353] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.066031] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.072303] kworker/27:1 S ffff9159f662d140 0 7272 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696415.079608] Call Trace: [Wed Feb 6 09:38:35 2019][1696415.082242] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:35 2019][1696415.088430] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696415.093593] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:35 2019][1696415.099351] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:35 2019][1696415.106073] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696415.111137] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.117425] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.124043] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.130329] kworker/30:1 S ffff9149f3e9a080 0 7273 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696415.137606] Call Trace: [Wed Feb 6 09:38:35 2019][1696415.140238] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:35 2019][1696415.146441] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696415.151585] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:35 2019][1696415.157348] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:35 2019][1696415.164069] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696415.169130] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.175419] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.182039] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.188327] kworker/31:1 S ffff9159f662b0c0 0 7275 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696415.195633] Call Trace: [Wed Feb 6 09:38:35 2019][1696415.198305] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:35 2019][1696415.204601] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696415.209763] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:35 2019][1696415.215521] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:35 2019][1696415.222229] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696415.227306] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.233586] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.240220] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.246498] kworker/39:1 S ffff9159f662c100 0 7276 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696415.253805] Call Trace: [Wed Feb 6 09:38:35 2019][1696415.256437] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:35 2019][1696415.262623] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696415.267786] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:35 2019][1696415.273534] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:35 2019][1696415.280248] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696415.285322] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.291600] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.298236] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.304512] kworker/37:1 S ffff9139f61f4100 0 7277 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696415.311791] Call Trace: [Wed Feb 6 09:38:35 2019][1696415.314439] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:35 2019][1696415.320623] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696415.325789] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:35 2019][1696415.331547] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:35 2019][1696415.338256] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696415.343356] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.349630] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.356267] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.362534] kworker/43:1 S ffff9159f6648000 0 7280 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696415.369810] Call Trace: [Wed Feb 6 09:38:35 2019][1696415.372458] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:35 2019][1696415.378651] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696415.383812] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:35 2019][1696415.389568] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:35 2019][1696415.396293] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696415.401356] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.407629] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.414265] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.420536] kworker/46:1 S ffff9149f3e9b0c0 0 7282 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696415.427831] Call Trace: [Wed Feb 6 09:38:35 2019][1696415.430474] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:35 2019][1696415.436663] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696415.441830] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:35 2019][1696415.447587] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:35 2019][1696415.454291] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696415.459374] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.465649] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.472269] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.478602] kworker/7:1 S ffff9159f664b0c0 0 7307 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696415.485903] Call Trace: [Wed Feb 6 09:38:35 2019][1696415.488552] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:35 2019][1696415.494740] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696415.499901] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:35 2019][1696415.505651] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:35 2019][1696415.512363] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696415.517437] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.523708] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.530339] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.536608] mlx5_health0000 S ffff9159f664c100 0 7334 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696415.543901] Call Trace: [Wed Feb 6 09:38:35 2019][1696415.546533] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696415.551686] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:35 2019][1696415.557538] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:35 2019][1696415.563721] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696415.568800] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.575073] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.581710] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.587977] scsi_eh_2 S ffff9159f664d140 0 7338 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696415.595245] Call Trace: [Wed Feb 6 09:38:35 2019][1696415.597894] [] ? __scsi_iterate_devices+0x68/0x90 [Wed Feb 6 09:38:35 2019][1696415.604432] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696415.609595] [] scsi_error_handler+0xa3/0x8b0 [Wed Feb 6 09:38:35 2019][1696415.615707] [] ? scsi_eh_get_sense+0x250/0x250 [Wed Feb 6 09:38:35 2019][1696415.621992] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696415.627047] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.633327] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.639944] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.646212] scsi_tmf_2 S ffff9159f664a080 0 7340 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696415.653518] Call Trace: [Wed Feb 6 09:38:35 2019][1696415.656151] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696415.661300] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:35 2019][1696415.667154] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:35 2019][1696415.673338] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696415.678405] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.684692] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.691318] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.697606] mlx5_cmd_0000:8 S ffff911a1a508000 0 7346 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696415.704886] Call Trace: [Wed Feb 6 09:38:35 2019][1696415.707535] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696415.712678] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:35 2019][1696415.718538] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:35 2019][1696415.724725] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696415.729790] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.736080] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.742703] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.749010] mlx5_page_alloc S ffff911a1a50d140 0 7394 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696415.756301] Call Trace: [Wed Feb 6 09:38:35 2019][1696415.758949] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696415.764088] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:35 2019][1696415.769948] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:35 2019][1696415.776136] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696415.781199] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.787487] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.794101] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.800372] kworker/u261:0 S ffff9149fa32c100 0 7439 2 0x00000000 [Wed Feb 6 09:38:35 2019][1696415.807680] Call Trace: [Wed Feb 6 09:38:35 2019][1696415.810315] [] schedule+0x29/0x70 [Wed Feb 6 09:38:35 2019][1696415.815459] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:35 2019][1696415.821229] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:35 2019][1696415.827930] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:35 2019][1696415.832993] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.839280] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:35 2019][1696415.845896] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:35 2019][1696415.852174] kworker/u262:0 S ffff911a1a50b0c0 0 7440 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696415.859478] Call Trace: [Wed Feb 6 09:38:36 2019][1696415.862112] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696415.867259] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:36 2019][1696415.873038] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:36 2019][1696415.879758] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696415.884821] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696415.891112] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696415.897738] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696415.904011] kworker/u263:0 S ffff912aa9ad4100 0 7441 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696415.911307] Call Trace: [Wed Feb 6 09:38:36 2019][1696415.913944] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696415.919088] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:36 2019][1696415.924857] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:36 2019][1696415.931565] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696415.936621] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696415.942912] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696415.949530] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696415.955805] kworker/u265:0 S ffff914aa95be180 0 7443 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696415.963099] Call Trace: [Wed Feb 6 09:38:36 2019][1696415.965732] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696415.970872] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:36 2019][1696415.976642] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:36 2019][1696415.983348] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696415.988405] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696415.994700] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.001321] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.007633] mlx5_fc S ffff914aa95bd140 0 7445 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.014938] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.017570] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.022718] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:36 2019][1696416.028573] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:36 2019][1696416.034760] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696416.039820] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.046109] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.052735] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.059010] ib-comp-wq S ffff911a1a50a080 0 7466 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.066316] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.068947] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.074096] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:36 2019][1696416.079951] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:36 2019][1696416.086134] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696416.091197] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.097488] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.104113] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.110388] ib-comp-unb-wq S ffff911a1a509040 0 7467 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.117690] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.120324] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.125464] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:36 2019][1696416.131320] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:36 2019][1696416.137558] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696416.142619] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.148908] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.155532] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.161807] ib_mcast S ffff911a1a50c100 0 7468 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.169101] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.171744] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.176885] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:36 2019][1696416.182741] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:36 2019][1696416.188926] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696416.193986] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.200278] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.206901] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.213190] ib_nl_sa_wq S ffff9129f33b1040 0 7469 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.220481] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.223114] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.228277] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:36 2019][1696416.234117] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:36 2019][1696416.240303] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696416.245382] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.251654] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.258291] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.264557] mlx5_ib_sigerr_ S ffff9129f33b0000 0 7477 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.271878] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.274523] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.279669] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:36 2019][1696416.285525] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:36 2019][1696416.291716] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696416.296786] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.303062] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.309695] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.315970] ib_mad1 S ffff9129f33b2080 0 7482 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.323264] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.325898] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.331046] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:36 2019][1696416.336897] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:36 2019][1696416.343081] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696416.348160] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.354435] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.361060] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.367347] to_fifo S ffff9129f33b30c0 0 7483 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.374627] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.377253] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.382410] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:36 2019][1696416.388257] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:36 2019][1696416.394444] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696416.399516] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.405807] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.412443] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.418722] mkey_cache S ffff9129f33b4100 0 7486 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.425995] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.428635] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.433780] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:36 2019][1696416.439634] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:36 2019][1696416.445821] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696416.450883] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.457171] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.463798] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.470083] kworker/35:1H S ffff914aa95bb0c0 0 7526 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.477359] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.480005] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:36 2019][1696416.486189] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.491354] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:36 2019][1696416.497113] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:36 2019][1696416.503830] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696416.508901] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.515170] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.521810] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.528086] kworker/16:1H S ffff9129fc01b0c0 0 7528 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.535443] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.538111] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:36 2019][1696416.544335] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.549507] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:36 2019][1696416.555286] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:36 2019][1696416.562001] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696416.567147] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.573503] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.580122] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.586411] kworker/0:1H S ffff9129fc01d140 0 7529 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.593703] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.596334] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:36 2019][1696416.602535] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.607680] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:36 2019][1696416.613436] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:36 2019][1696416.620160] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696416.625226] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.631501] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.638137] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.644414] poll_mpt3sas0_s S ffff9129fc01e180 0 7530 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.651709] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.654343] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.659491] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:36 2019][1696416.665349] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:36 2019][1696416.671530] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696416.676594] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.682882] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.689500] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.695787] jbd2/sdk2-8 S ffff9129fb162080 0 7801 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.703151] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.705846] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.711092] [] kjournald2+0x203/0x260 [jbd2] [Wed Feb 6 09:38:36 2019][1696416.717256] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:36 2019][1696416.723279] [] ? commit_timeout+0x10/0x10 [jbd2] [Wed Feb 6 09:38:36 2019][1696416.729736] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696416.734792] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.741080] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.747697] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.753984] ext4-rsv-conver S ffff9129fb161040 0 7802 2 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.761266] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.763902] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.769055] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:36 2019][1696416.774910] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:36 2019][1696416.781099] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:36 2019][1696416.786175] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.792455] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:36 2019][1696416.799088] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:36 2019][1696416.805367] systemd-journal S ffff9159f6b92080 0 7874 1 0x00000000 [Wed Feb 6 09:38:36 2019][1696416.812658] Call Trace: [Wed Feb 6 09:38:36 2019][1696416.815291] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:38:36 2019][1696416.820959] [] schedule+0x29/0x70 [Wed Feb 6 09:38:36 2019][1696416.826122] [] ? filename_lookup+0x2b/0xc0 [Wed Feb 6 09:38:36 2019][1696416.832056] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Wed Feb 6 09:38:36 2019][1696416.839031] [] ? do_filp_open+0x4d/0xb0 [Wed Feb 6 09:38:36 2019][1696416.844730] [] ? schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:37 2019][1696416.851564] [] ? ep_poll+0x23e/0x360 [Wed Feb 6 09:38:37 2019][1696416.857036] [] ? do_truncate+0x88/0xc0 [Wed Feb 6 09:38:37 2019][1696416.862725] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:37 2019][1696416.868625] [] ? SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:38:37 2019][1696416.874562] [] ? system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:37 2019][1696416.880922] systemd-udevd S ffff9159f6b90000 0 7905 1 0x00000000 [Wed Feb 6 09:38:37 2019][1696416.888216] Call Trace: [Wed Feb 6 09:38:37 2019][1696416.890847] [] ? kmem_cache_alloc+0x35/0x1f0 [Wed Feb 6 09:38:37 2019][1696416.896947] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696416.902114] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:37 2019][1696416.909346] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Wed Feb 6 09:38:37 2019][1696416.916313] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:37 2019][1696416.922861] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:38:37 2019][1696416.928099] [] ? do_unlinkat+0xf1/0x2d0 [Wed Feb 6 09:38:37 2019][1696416.933768] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:37 2019][1696416.939532] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:38:37 2019][1696416.945287] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:37 2019][1696416.951487] ccp-1-q2 S ffff9149fab7a080 0 10459 2 0x00000000 [Wed Feb 6 09:38:37 2019][1696416.958766] Call Trace: [Wed Feb 6 09:38:37 2019][1696416.961407] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696416.966556] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:37 2019][1696416.973362] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:37 2019][1696416.980074] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:37 2019][1696416.986791] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:37 2019][1696416.991854] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696416.998138] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:37 2019][1696417.004886] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.011305] ccp-1-q3 S ffff9129fb09ddc0 0 10469 2 0x00000000 [Wed Feb 6 09:38:37 2019][1696417.018688] Call Trace: [Wed Feb 6 09:38:37 2019][1696417.021316] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:37 2019][1696417.026811] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696417.031979] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.038777] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.045484] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:37 2019][1696417.052204] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:37 2019][1696417.057261] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.063551] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:37 2019][1696417.070174] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.076464] ccp-1-q4 S ffff9149fab7c100 0 10487 2 0x00000000 [Wed Feb 6 09:38:37 2019][1696417.083740] Call Trace: [Wed Feb 6 09:38:37 2019][1696417.086372] [] ? kprobe_flush_task+0xd0/0x170 [Wed Feb 6 09:38:37 2019][1696417.092576] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696417.097730] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.104526] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.111246] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:37 2019][1696417.117950] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:37 2019][1696417.123018] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.129294] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:37 2019][1696417.135914] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.142200] ccp-2-q0 S ffff9149faf68c80 0 10521 2 0x00000000 [Wed Feb 6 09:38:37 2019][1696417.149477] Call Trace: [Wed Feb 6 09:38:37 2019][1696417.152120] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:37 2019][1696417.157612] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696417.162777] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.169575] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.176283] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:37 2019][1696417.183003] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:37 2019][1696417.188064] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.194339] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:37 2019][1696417.200971] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.207239] ccp-2-q1 S ffff9129fb9cea40 0 10530 2 0x00000000 [Wed Feb 6 09:38:37 2019][1696417.214542] Call Trace: [Wed Feb 6 09:38:37 2019][1696417.217170] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:37 2019][1696417.222658] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696417.227822] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.234621] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.241329] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:37 2019][1696417.248050] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:37 2019][1696417.253117] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.259407] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:37 2019][1696417.266025] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.272310] ccp-2-q2 S ffff9139fb67b200 0 10541 2 0x00000000 [Wed Feb 6 09:38:37 2019][1696417.279603] Call Trace: [Wed Feb 6 09:38:37 2019][1696417.282232] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:37 2019][1696417.287723] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696417.292886] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.299685] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.306408] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:37 2019][1696417.313115] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:37 2019][1696417.318189] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.324459] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:37 2019][1696417.331085] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.337372] ccp-2-q3 S ffff9149fa3d9f40 0 10550 2 0x00000000 [Wed Feb 6 09:38:37 2019][1696417.344665] Call Trace: [Wed Feb 6 09:38:37 2019][1696417.347295] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:37 2019][1696417.352810] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696417.357963] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.364763] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.371486] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:37 2019][1696417.378197] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:37 2019][1696417.383272] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.389550] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:37 2019][1696417.396179] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.402452] ccp-2-q4 S ffff9139f339b0c0 0 10557 2 0x00000000 [Wed Feb 6 09:38:37 2019][1696417.409745] Call Trace: [Wed Feb 6 09:38:37 2019][1696417.412374] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:37 2019][1696417.417864] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696417.423030] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.429826] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.436547] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:37 2019][1696417.443259] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:37 2019][1696417.448332] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.454603] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:37 2019][1696417.461236] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.467504] ccp-3-q2 S ffff9139f339e180 0 10585 2 0x00000000 [Wed Feb 6 09:38:37 2019][1696417.474792] Call Trace: [Wed Feb 6 09:38:37 2019][1696417.477423] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696417.482574] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.489379] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.496087] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:37 2019][1696417.502803] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:37 2019][1696417.507859] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.514149] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:37 2019][1696417.520769] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.527041] ccp-3-q3 S ffff9149fa3d8000 0 10588 2 0x00000000 [Wed Feb 6 09:38:37 2019][1696417.534332] Call Trace: [Wed Feb 6 09:38:37 2019][1696417.536962] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:37 2019][1696417.542458] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696417.547621] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.554411] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.561123] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:37 2019][1696417.567841] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:37 2019][1696417.572898] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.579186] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:37 2019][1696417.585811] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.592099] ccp-3-q4 S ffff9139f3398000 0 10636 2 0x00000000 [Wed Feb 6 09:38:37 2019][1696417.599375] Call Trace: [Wed Feb 6 09:38:37 2019][1696417.602007] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:37 2019][1696417.607504] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696417.612668] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.619467] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.626183] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:37 2019][1696417.632905] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:37 2019][1696417.637969] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.644254] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:37 2019][1696417.650876] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.657161] ccp-4-q0 S ffff9149fbfdf080 0 10650 2 0x00000000 [Wed Feb 6 09:38:37 2019][1696417.664453] Call Trace: [Wed Feb 6 09:38:37 2019][1696417.667106] [] ? kprobe_flush_task+0xd0/0x170 [Wed Feb 6 09:38:37 2019][1696417.673297] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696417.678460] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.685257] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.691980] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:37 2019][1696417.698689] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:37 2019][1696417.703766] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.710044] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:37 2019][1696417.716664] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.722952] ccp-4-q1 S ffff9159fc531040 0 10661 2 0x00000000 [Wed Feb 6 09:38:37 2019][1696417.730232] Call Trace: [Wed Feb 6 09:38:37 2019][1696417.732871] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:37 2019][1696417.738368] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696417.743526] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.750318] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.757036] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:37 2019][1696417.763743] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:37 2019][1696417.768816] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.775085] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:37 2019][1696417.781725] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.788001] ccp-4-q2 S ffff9159fc532080 0 10666 2 0x00000000 [Wed Feb 6 09:38:37 2019][1696417.795292] Call Trace: [Wed Feb 6 09:38:37 2019][1696417.797935] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:37 2019][1696417.803427] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696417.808591] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.815387] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:37 2019][1696417.822096] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:37 2019][1696417.828820] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:37 2019][1696417.833884] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.840155] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:37 2019][1696417.846793] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:37 2019][1696417.853064] ccp-4-q3 S ffff9139fb512bc0 0 10667 2 0x00000000 [Wed Feb 6 09:38:37 2019][1696417.860346] Call Trace: [Wed Feb 6 09:38:37 2019][1696417.862989] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:37 2019][1696417.868478] [] schedule+0x29/0x70 [Wed Feb 6 09:38:37 2019][1696417.873643] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696417.880431] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696417.887143] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696417.893858] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696417.898921] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696417.905207] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696417.911825] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696417.918112] ccp-4-q4 S ffff9149f7588000 0 10689 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696417.925381] Call Trace: [Wed Feb 6 09:38:38 2019][1696417.928025] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:38 2019][1696417.933521] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696417.938682] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696417.945479] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696417.952201] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696417.958911] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696417.963986] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696417.970255] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696417.976876] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696417.983166] ccp-5-q2 S ffff9149fbfdbe80 0 10700 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696417.990435] Call Trace: [Wed Feb 6 09:38:38 2019][1696417.993064] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696417.998214] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.005044] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.011759] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696418.018471] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696418.023544] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.029812] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696418.036447] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.042718] ccp-5-q3 S ffff9159fb6a8000 0 10714 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696418.050007] Call Trace: [Wed Feb 6 09:38:38 2019][1696418.052657] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696418.057804] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.064609] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.071313] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696418.078031] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696418.083088] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.089366] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696418.096002] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.102281] ccp-5-q4 S ffff9149f6e8b200 0 10755 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696418.109581] Call Trace: [Wed Feb 6 09:38:38 2019][1696418.112208] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696418.117353] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.124158] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.130862] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696418.137570] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696418.142645] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.148915] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696418.155550] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.161816] ccp-6-q0 S ffff9159fb6ac100 0 10781 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696418.169102] Call Trace: [Wed Feb 6 09:38:38 2019][1696418.171737] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:38 2019][1696418.177233] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696418.182392] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.189182] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.195904] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696418.202604] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696418.207671] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.213957] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696418.220574] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.226858] ccp-6-q1 S ffff9159fb6ad140 0 10816 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696418.234131] Call Trace: [Wed Feb 6 09:38:38 2019][1696418.236774] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:38 2019][1696418.242264] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696418.247438] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.254234] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.260946] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696418.267664] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696418.272721] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.279013] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696418.285631] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.291904] ccp-6-q2 S ffff9159fb6ae180 0 10819 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696418.299204] Call Trace: [Wed Feb 6 09:38:38 2019][1696418.301837] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:38 2019][1696418.307332] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696418.312496] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.319293] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.326014] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696418.332723] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696418.337798] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.344068] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696418.350704] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.356984] ccp-6-q3 S ffff9159ec778000 0 10855 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696418.364281] Call Trace: [Wed Feb 6 09:38:38 2019][1696418.366914] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:38 2019][1696418.372411] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696418.377574] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.384370] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.391091] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696418.397801] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696418.402876] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.409145] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696418.415768] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.422058] ccp-6-q4 S ffff9149f6e8ddc0 0 10861 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696418.429338] Call Trace: [Wed Feb 6 09:38:38 2019][1696418.431974] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:38 2019][1696418.437470] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696418.442634] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.449433] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.456138] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696418.462864] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696418.467929] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.474201] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696418.480837] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.487116] ccp-7-q2 S ffff9129fba62bc0 0 10865 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696418.494405] Call Trace: [Wed Feb 6 09:38:38 2019][1696418.497034] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:38 2019][1696418.502522] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696418.507687] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.514488] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.521203] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696418.527924] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696418.532980] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.539258] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696418.545891] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.552166] ccp-7-q3 S ffff9159ec77b0c0 0 10876 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696418.559468] Call Trace: [Wed Feb 6 09:38:38 2019][1696418.562100] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696418.567254] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.574066] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.580780] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696418.587497] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696418.592553] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.598843] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696418.605458] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.611729] ccp-7-q4 S ffff9159ec77c100 0 10878 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696418.619029] Call Trace: [Wed Feb 6 09:38:38 2019][1696418.621656] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696418.626799] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.633607] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.640309] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696418.647015] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696418.652084] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.658355] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696418.664978] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.671265] ccp-8-q0 S ffff9159ec77d140 0 10883 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696418.678542] Call Trace: [Wed Feb 6 09:38:38 2019][1696418.681186] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696418.686332] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.693146] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.699852] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696418.706557] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696418.711624] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.717893] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696418.724506] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.730787] ccp-8-q1 S ffff9159ec77e180 0 10884 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696418.738070] Call Trace: [Wed Feb 6 09:38:38 2019][1696418.740706] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696418.745858] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.752668] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.759379] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696418.766098] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696418.771152] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.777448] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696418.784065] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.790331] ccp-8-q2 S ffff9159fc6d8000 0 10885 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696418.797627] Call Trace: [Wed Feb 6 09:38:38 2019][1696418.800255] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696418.805401] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.812208] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.818920] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:38 2019][1696418.825642] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:38 2019][1696418.830705] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.836995] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:38 2019][1696418.843621] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:38 2019][1696418.849905] ccp-8-q3 S ffff9159fc6d9040 0 10886 2 0x00000000 [Wed Feb 6 09:38:38 2019][1696418.857176] Call Trace: [Wed Feb 6 09:38:38 2019][1696418.859817] [] schedule+0x29/0x70 [Wed Feb 6 09:38:38 2019][1696418.864964] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:38 2019][1696418.871774] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:39 2019][1696418.878484] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:39 2019][1696418.885192] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:39 2019][1696418.890267] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:39 2019][1696418.896535] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:39 2019][1696418.903163] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:39 2019][1696418.909452] ccp-8-q4 S ffff9159fc6da080 0 10887 2 0x00000000 [Wed Feb 6 09:38:39 2019][1696418.916736] Call Trace: [Wed Feb 6 09:38:39 2019][1696418.919369] [] schedule+0x29/0x70 [Wed Feb 6 09:38:39 2019][1696418.924534] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Wed Feb 6 09:38:39 2019][1696418.931332] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Wed Feb 6 09:38:39 2019][1696418.938039] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Wed Feb 6 09:38:39 2019][1696418.944740] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:39 2019][1696418.949797] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:39 2019][1696418.956083] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:39 2019][1696418.962703] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:39 2019][1696418.968989] kmpathd S ffff9159f7749040 0 11776 2 0x00000000 [Wed Feb 6 09:38:39 2019][1696418.976286] Call Trace: [Wed Feb 6 09:38:39 2019][1696418.978927] [] schedule+0x29/0x70 [Wed Feb 6 09:38:39 2019][1696418.984072] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:39 2019][1696418.989926] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:39 2019][1696418.996110] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:39 2019][1696419.001171] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:39 2019][1696419.007464] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:39 2019][1696419.014088] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:39 2019][1696419.020361] kmpath_handlerd S ffff9149f6e8be80 0 11777 2 0x00000000 [Wed Feb 6 09:38:39 2019][1696419.027656] Call Trace: [Wed Feb 6 09:38:39 2019][1696419.030283] [] schedule+0x29/0x70 [Wed Feb 6 09:38:39 2019][1696419.035428] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:39 2019][1696419.041282] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:39 2019][1696419.047473] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:39 2019][1696419.052546] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:39 2019][1696419.058815] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:39 2019][1696419.065456] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:39 2019][1696419.071734] multipathd S ffff9159f7d16180 0 13033 1 0x00000000 [Wed Feb 6 09:38:39 2019][1696419.079004] Call Trace: [Wed Feb 6 09:38:39 2019][1696419.081649] [] schedule+0x29/0x70 [Wed Feb 6 09:38:39 2019][1696419.086795] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:39 2019][1696419.092994] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:39 2019][1696419.098487] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:39 2019][1696419.103811] [] ? mntput+0x24/0x40 [Wed Feb 6 09:38:39 2019][1696419.108970] [] ? __fput+0x186/0x260 [Wed Feb 6 09:38:39 2019][1696419.114295] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:39 2019][1696419.119625] [] ? task_work_run+0xc0/0xe0 [Wed Feb 6 09:38:39 2019][1696419.125380] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:39 2019][1696419.131582] multipathd S ffff9159f6bab0c0 0 13038 1 0x00000000 [Wed Feb 6 09:38:39 2019][1696419.138851] Call Trace: [Wed Feb 6 09:38:39 2019][1696419.141493] [] schedule+0x29/0x70 [Wed Feb 6 09:38:39 2019][1696419.146638] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:39 2019][1696419.152839] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:39 2019][1696419.158333] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:39 2019][1696419.163655] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:39 2019][1696419.168988] [] ? fd_install+0x25/0x30 [Wed Feb 6 09:38:39 2019][1696419.174488] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:39 2019][1696419.180687] multipathd S ffff9159fc5ba580 0 13048 1 0x00000000 [Wed Feb 6 09:38:39 2019][1696419.187966] Call Trace: [Wed Feb 6 09:38:39 2019][1696419.190609] [] schedule+0x29/0x70 [Wed Feb 6 09:38:39 2019][1696419.195752] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:39 2019][1696419.202999] [] ? __slab_free+0x81/0x2f0 [Wed Feb 6 09:38:39 2019][1696419.208669] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:38:39 2019][1696419.214513] [] ? __pollwait+0x7f/0xf0 [Wed Feb 6 09:38:39 2019][1696419.220017] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:39 2019][1696419.226548] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:39 2019][1696419.232836] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:38:39 2019][1696419.238414] [] ? netlink_recvmsg+0x212/0x490 [Wed Feb 6 09:38:39 2019][1696419.244527] [] ? plist_add+0x91/0xd0 [Wed Feb 6 09:38:39 2019][1696419.249929] [] ? tracing_is_on+0x15/0x30 [Wed Feb 6 09:38:39 2019][1696419.255682] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:39 2019][1696419.261192] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:38:39 2019][1696419.267202] [] ? wake_up_q+0x5b/0x80 [Wed Feb 6 09:38:39 2019][1696419.272615] [] ? futex_wake_op+0x4a5/0x610 [Wed Feb 6 09:38:39 2019][1696419.278556] [] ? do_futex+0x17a/0x5a0 [Wed Feb 6 09:38:39 2019][1696419.284053] [] ? _raw_spin_unlock_bh+0x1e/0x20 [Wed Feb 6 09:38:39 2019][1696419.290344] [] ? release_sock+0x120/0x170 [Wed Feb 6 09:38:39 2019][1696419.296186] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:38:39 2019][1696419.301432] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:39 2019][1696419.307615] multipathd S ffff9159f6bae180 0 13050 1 0x00000000 [Wed Feb 6 09:38:39 2019][1696419.314904] Call Trace: [Wed Feb 6 09:38:39 2019][1696419.317543] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:39 2019][1696419.324250] [] schedule+0x29/0x70 [Wed Feb 6 09:38:39 2019][1696419.329413] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:38:39 2019][1696419.336559] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:39 2019][1696419.342499] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:38:39 2019][1696419.349816] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:39 2019][1696419.356365] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:39 2019][1696419.362636] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:38:39 2019][1696419.368233] [] ? get_page_from_freelist+0x412/0xa70 [Wed Feb 6 09:38:39 2019][1696419.374944] [] ? load_balance+0x1be/0x9a0 [Wed Feb 6 09:38:39 2019][1696419.380794] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:39 2019][1696419.386287] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:38:39 2019][1696419.393267] [] ? add_page_to_unevictable_list+0xc8/0x100 [Wed Feb 6 09:38:39 2019][1696419.400404] [] ? page_add_new_anon_rmap+0x12d/0x170 [Wed Feb 6 09:38:39 2019][1696419.407115] [] ? handle_pte_fault+0x5b8/0xd10 [Wed Feb 6 09:38:39 2019][1696419.413317] [] ? __kmalloc+0x55/0x230 [Wed Feb 6 09:38:39 2019][1696419.418818] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:39 2019][1696419.424931] [] ? __set_task_blocked+0x41/0xa0 [Wed Feb 6 09:38:39 2019][1696419.431123] [] SyS_ppoll+0xee/0x1f0 [Wed Feb 6 09:38:39 2019][1696419.436462] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:39 2019][1696419.442648] multipathd S ffff9159ff3a30c0 0 13612 1 0x00000000 [Wed Feb 6 09:38:39 2019][1696419.449939] Call Trace: [Wed Feb 6 09:38:39 2019][1696419.452569] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:39 2019][1696419.459277] [] schedule+0x29/0x70 [Wed Feb 6 09:38:39 2019][1696419.464437] [] do_nanosleep+0x86/0xf0 [Wed Feb 6 09:38:39 2019][1696419.469928] [] hrtimer_nanosleep+0xbb/0x180 [Wed Feb 6 09:38:39 2019][1696419.475944] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:39 2019][1696419.481888] [] ? do_nanosleep+0x72/0xf0 [Wed Feb 6 09:38:39 2019][1696419.487551] [] SyS_nanosleep+0x96/0xb0 [Wed Feb 6 09:38:39 2019][1696419.493134] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:39 2019][1696419.499335] multipathd S ffff9159ff3a1040 0 13613 1 0x00000000 [Wed Feb 6 09:38:39 2019][1696419.506612] Call Trace: [Wed Feb 6 09:38:39 2019][1696419.509240] [] schedule+0x29/0x70 [Wed Feb 6 09:38:39 2019][1696419.514397] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:39 2019][1696419.520580] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:39 2019][1696419.526072] [] ? mntput+0x24/0x40 [Wed Feb 6 09:38:39 2019][1696419.531234] [] ? path_openat+0x172/0x640 [Wed Feb 6 09:38:39 2019][1696419.536983] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:39 2019][1696419.542306] [] ? lockref_put_or_lock+0x48/0x80 [Wed Feb 6 09:38:39 2019][1696419.548600] [] ? mntput+0x24/0x40 [Wed Feb 6 09:38:39 2019][1696419.553749] [] ? __fput+0x186/0x260 [Wed Feb 6 09:38:39 2019][1696419.559070] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:39 2019][1696419.564407] [] ? task_work_run+0xc0/0xe0 [Wed Feb 6 09:38:39 2019][1696419.570166] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:39 2019][1696419.576362] multipathd S ffff9159f6e9e180 0 17011 1 0x00000000 [Wed Feb 6 09:38:39 2019][1696419.583643] Call Trace: [Wed Feb 6 09:38:39 2019][1696419.586273] [] schedule+0x29/0x70 [Wed Feb 6 09:38:39 2019][1696419.591441] [] dm_wait_event+0xa1/0xf0 [dm_mod] [Wed Feb 6 09:38:39 2019][1696419.597801] [] ? strlcpy+0x42/0x60 [Wed Feb 6 09:38:39 2019][1696419.603034] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:39 2019][1696419.609060] [] dev_wait+0x50/0xd0 [dm_mod] [Wed Feb 6 09:38:39 2019][1696419.614995] [] ctl_ioctl+0x24e/0x550 [dm_mod] [Wed Feb 6 09:38:39 2019][1696419.621188] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:38:39 2019][1696419.628171] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [Wed Feb 6 09:38:39 2019][1696419.634625] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [Wed Feb 6 09:38:39 2019][1696419.640830] [] do_vfs_ioctl+0x3a0/0x5a0 [Wed Feb 6 09:38:39 2019][1696419.646499] [] ? __set_task_blocked+0x41/0xa0 [Wed Feb 6 09:38:39 2019][1696419.652701] [] ? __set_current_blocked+0x42/0x70 [Wed Feb 6 09:38:39 2019][1696419.659144] [] SyS_ioctl+0xa1/0xc0 [Wed Feb 6 09:38:39 2019][1696419.664379] [] ? SyS_rt_sigprocmask+0xc4/0x100 [Wed Feb 6 09:38:39 2019][1696419.670669] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:39 2019][1696419.676853] multipathd S ffff9159f6e9c100 0 17151 1 0x00000000 [Wed Feb 6 09:38:39 2019][1696419.684126] Call Trace: [Wed Feb 6 09:38:39 2019][1696419.686774] [] schedule+0x29/0x70 [Wed Feb 6 09:38:39 2019][1696419.691930] [] dm_wait_event+0xa1/0xf0 [dm_mod] [Wed Feb 6 09:38:39 2019][1696419.698300] [] ? strlcpy+0x42/0x60 [Wed Feb 6 09:38:39 2019][1696419.703530] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:39 2019][1696419.709557] [] dev_wait+0x50/0xd0 [dm_mod] [Wed Feb 6 09:38:39 2019][1696419.715481] [] ctl_ioctl+0x24e/0x550 [dm_mod] [Wed Feb 6 09:38:39 2019][1696419.721681] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:38:39 2019][1696419.728649] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [Wed Feb 6 09:38:39 2019][1696419.735097] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [Wed Feb 6 09:38:39 2019][1696419.741299] [] do_vfs_ioctl+0x3a0/0x5a0 [Wed Feb 6 09:38:39 2019][1696419.746971] [] ? __set_task_blocked+0x41/0xa0 [Wed Feb 6 09:38:39 2019][1696419.753158] [] ? __set_current_blocked+0x42/0x70 [Wed Feb 6 09:38:39 2019][1696419.759614] [] SyS_ioctl+0xa1/0xc0 [Wed Feb 6 09:38:39 2019][1696419.764845] [] ? SyS_rt_sigprocmask+0xc4/0x100 [Wed Feb 6 09:38:39 2019][1696419.771120] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:39 2019][1696419.777307] multipathd S ffff9159f6e98000 0 17266 1 0x00000000 [Wed Feb 6 09:38:39 2019][1696419.784597] Call Trace: [Wed Feb 6 09:38:39 2019][1696419.787235] [] schedule+0x29/0x70 [Wed Feb 6 09:38:39 2019][1696419.792386] [] dm_wait_event+0xa1/0xf0 [dm_mod] [Wed Feb 6 09:38:39 2019][1696419.798757] [] ? strlcpy+0x42/0x60 [Wed Feb 6 09:38:39 2019][1696419.803984] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:39 2019][1696419.810016] [] dev_wait+0x50/0xd0 [dm_mod] [Wed Feb 6 09:38:39 2019][1696419.815939] [] ctl_ioctl+0x24e/0x550 [dm_mod] [Wed Feb 6 09:38:39 2019][1696419.822126] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:38:39 2019][1696419.829116] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [Wed Feb 6 09:38:39 2019][1696419.835566] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [Wed Feb 6 09:38:39 2019][1696419.841751] [] do_vfs_ioctl+0x3a0/0x5a0 [Wed Feb 6 09:38:39 2019][1696419.847428] [] ? __set_task_blocked+0x41/0xa0 [Wed Feb 6 09:38:39 2019][1696419.853613] [] ? __set_current_blocked+0x42/0x70 [Wed Feb 6 09:38:39 2019][1696419.860076] [] SyS_ioctl+0xa1/0xc0 [Wed Feb 6 09:38:39 2019][1696419.865302] [] ? SyS_rt_sigprocmask+0xc4/0x100 [Wed Feb 6 09:38:39 2019][1696419.871572] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:39 2019][1696419.877777] multipathd S ffff9149f6e630c0 0 18450 1 0x00000080 [Wed Feb 6 09:38:40 2019][1696419.885072] Call Trace: [Wed Feb 6 09:38:40 2019][1696419.887704] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696419.892852] [] dm_wait_event+0xa1/0xf0 [dm_mod] [Wed Feb 6 09:38:40 2019][1696419.899222] [] ? strlcpy+0x42/0x60 [Wed Feb 6 09:38:40 2019][1696419.904451] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:40 2019][1696419.910471] [] dev_wait+0x50/0xd0 [dm_mod] [Wed Feb 6 09:38:40 2019][1696419.916412] [] ctl_ioctl+0x24e/0x550 [dm_mod] [Wed Feb 6 09:38:40 2019][1696419.922603] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:38:40 2019][1696419.929573] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [Wed Feb 6 09:38:40 2019][1696419.936038] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [Wed Feb 6 09:38:40 2019][1696419.942226] [] do_vfs_ioctl+0x3a0/0x5a0 [Wed Feb 6 09:38:40 2019][1696419.947903] [] ? __set_task_blocked+0x41/0xa0 [Wed Feb 6 09:38:40 2019][1696419.954086] [] ? __set_current_blocked+0x42/0x70 [Wed Feb 6 09:38:40 2019][1696419.960549] [] SyS_ioctl+0xa1/0xc0 [Wed Feb 6 09:38:40 2019][1696419.965780] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:40 2019][1696419.971972] multipathd S ffff9149f6e61040 0 18506 1 0x00000080 [Wed Feb 6 09:38:40 2019][1696419.979273] Call Trace: [Wed Feb 6 09:38:40 2019][1696419.981910] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696419.987063] [] dm_wait_event+0xa1/0xf0 [dm_mod] [Wed Feb 6 09:38:40 2019][1696419.993430] [] ? strlcpy+0x42/0x60 [Wed Feb 6 09:38:40 2019][1696419.998661] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:40 2019][1696420.004697] [] dev_wait+0x50/0xd0 [dm_mod] [Wed Feb 6 09:38:40 2019][1696420.010623] [] ctl_ioctl+0x24e/0x550 [dm_mod] [Wed Feb 6 09:38:40 2019][1696420.016810] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:38:40 2019][1696420.023799] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [Wed Feb 6 09:38:40 2019][1696420.030250] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [Wed Feb 6 09:38:40 2019][1696420.036444] [] do_vfs_ioctl+0x3a0/0x5a0 [Wed Feb 6 09:38:40 2019][1696420.042122] [] ? __set_task_blocked+0x41/0xa0 [Wed Feb 6 09:38:40 2019][1696420.048303] [] ? __set_current_blocked+0x42/0x70 [Wed Feb 6 09:38:40 2019][1696420.054754] [] SyS_ioctl+0xa1/0xc0 [Wed Feb 6 09:38:40 2019][1696420.059992] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:40 2019][1696420.066190] kvm-irqfd-clean S ffff9149fc73d780 0 13585 2 0x00000000 [Wed Feb 6 09:38:40 2019][1696420.073479] Call Trace: [Wed Feb 6 09:38:40 2019][1696420.076123] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696420.081266] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:40 2019][1696420.087123] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:40 2019][1696420.093315] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:40 2019][1696420.098372] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.104654] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:40 2019][1696420.111271] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.117545] kworker/8:1H S ffff9129ec626180 0 16770 2 0x00000000 [Wed Feb 6 09:38:40 2019][1696420.124854] Call Trace: [Wed Feb 6 09:38:40 2019][1696420.127492] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:40 2019][1696420.133675] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696420.138838] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:40 2019][1696420.144590] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:40 2019][1696420.151296] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:40 2019][1696420.156372] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.162651] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:40 2019][1696420.169285] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.175565] kworker/3:1H S ffff9159f774b0c0 0 16771 2 0x00000000 [Wed Feb 6 09:38:40 2019][1696420.182871] Call Trace: [Wed Feb 6 09:38:40 2019][1696420.185504] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:40 2019][1696420.191692] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696420.196855] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:40 2019][1696420.202612] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:40 2019][1696420.209334] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:40 2019][1696420.214391] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.220679] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:40 2019][1696420.227293] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.233585] kworker/40:1H S ffff9129ec620000 0 16783 2 0x00000000 [Wed Feb 6 09:38:40 2019][1696420.240884] Call Trace: [Wed Feb 6 09:38:40 2019][1696420.243518] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:40 2019][1696420.249707] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696420.254867] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:40 2019][1696420.260622] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:40 2019][1696420.267329] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:40 2019][1696420.272404] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.278682] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:40 2019][1696420.285318] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.291596] kworker/25:1H S ffff9139f6cec100 0 16785 2 0x00000000 [Wed Feb 6 09:38:40 2019][1696420.298882] Call Trace: [Wed Feb 6 09:38:40 2019][1696420.301529] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:40 2019][1696420.307713] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696420.312876] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:40 2019][1696420.318624] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:40 2019][1696420.325333] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:40 2019][1696420.330409] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.336682] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:40 2019][1696420.343302] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.349590] kworker/31:1H S ffff9159f7748000 0 16786 2 0x00000000 [Wed Feb 6 09:38:40 2019][1696420.356872] Call Trace: [Wed Feb 6 09:38:40 2019][1696420.359519] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:40 2019][1696420.365707] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696420.370868] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:40 2019][1696420.376624] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:40 2019][1696420.383350] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:40 2019][1696420.388414] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.394691] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:40 2019][1696420.401324] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.407599] kworker/22:1H S ffff9149fab7d140 0 16787 2 0x00000000 [Wed Feb 6 09:38:40 2019][1696420.414908] Call Trace: [Wed Feb 6 09:38:40 2019][1696420.417544] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:40 2019][1696420.423729] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696420.428890] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:40 2019][1696420.434643] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:40 2019][1696420.441352] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:40 2019][1696420.446426] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.452692] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:40 2019][1696420.459330] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.465600] kworker/21:1H S ffff9139f6cee180 0 16788 2 0x00000000 [Wed Feb 6 09:38:40 2019][1696420.472880] Call Trace: [Wed Feb 6 09:38:40 2019][1696420.475527] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:40 2019][1696420.481713] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696420.486876] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:40 2019][1696420.492634] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:40 2019][1696420.499353] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:40 2019][1696420.504412] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.510697] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:40 2019][1696420.517312] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.523602] kworker/2:1H S ffff9149fab7b0c0 0 16789 2 0x00000000 [Wed Feb 6 09:38:40 2019][1696420.530889] Call Trace: [Wed Feb 6 09:38:40 2019][1696420.533524] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:40 2019][1696420.539712] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696420.544874] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:40 2019][1696420.550628] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:40 2019][1696420.557353] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:40 2019][1696420.562407] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.568687] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:40 2019][1696420.575319] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.581590] kdmflush S ffff9139f6ced140 0 16996 2 0x00000000 [Wed Feb 6 09:38:40 2019][1696420.588892] Call Trace: [Wed Feb 6 09:38:40 2019][1696420.591520] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696420.596666] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:40 2019][1696420.602528] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:40 2019][1696420.608714] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:40 2019][1696420.613777] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.620068] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:40 2019][1696420.626694] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.632969] bioset S ffff9139f6ce8000 0 17009 2 0x00000000 [Wed Feb 6 09:38:40 2019][1696420.640270] Call Trace: [Wed Feb 6 09:38:40 2019][1696420.642904] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696420.648056] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:40 2019][1696420.653913] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:40 2019][1696420.660103] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:40 2019][1696420.665176] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.671448] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:40 2019][1696420.678083] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.684363] kdmwork-253:0 S ffff9159f7d11040 0 17010 2 0x00000000 [Wed Feb 6 09:38:40 2019][1696420.691653] Call Trace: [Wed Feb 6 09:38:40 2019][1696420.694286] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696420.699435] [] kthread_worker_fn+0xf7/0x180 [Wed Feb 6 09:38:40 2019][1696420.705458] [] ? kthread_stop+0xf0/0xf0 [Wed Feb 6 09:38:40 2019][1696420.711122] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:40 2019][1696420.716200] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.722482] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:40 2019][1696420.729101] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.735387] kworker/15:1H S ffff9159fa26d140 0 17116 2 0x00000000 [Wed Feb 6 09:38:40 2019][1696420.742669] Call Trace: [Wed Feb 6 09:38:40 2019][1696420.745309] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:40 2019][1696420.751507] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696420.756650] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:40 2019][1696420.762407] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:40 2019][1696420.769128] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:40 2019][1696420.774186] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.780475] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:40 2019][1696420.787101] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.793388] kworker/32:1H S ffff9129ec622080 0 17123 2 0x00000000 [Wed Feb 6 09:38:40 2019][1696420.800668] Call Trace: [Wed Feb 6 09:38:40 2019][1696420.803295] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:40 2019][1696420.809501] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696420.814647] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:40 2019][1696420.820404] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:40 2019][1696420.827128] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:40 2019][1696420.832194] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.838467] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:40 2019][1696420.845098] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:40 2019][1696420.851371] kdmflush S ffff9129ec6230c0 0 17146 2 0x00000000 [Wed Feb 6 09:38:40 2019][1696420.858668] Call Trace: [Wed Feb 6 09:38:40 2019][1696420.861302] [] schedule+0x29/0x70 [Wed Feb 6 09:38:40 2019][1696420.866455] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:40 2019][1696420.872308] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:40 2019][1696420.878499] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:41 2019][1696420.883575] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696420.889849] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:41 2019][1696420.896496] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696420.902768] bioset S ffff9129ec624100 0 17148 2 0x00000000 [Wed Feb 6 09:38:41 2019][1696420.910054] Call Trace: [Wed Feb 6 09:38:41 2019][1696420.912701] [] schedule+0x29/0x70 [Wed Feb 6 09:38:41 2019][1696420.917850] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:41 2019][1696420.923703] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:41 2019][1696420.929895] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:41 2019][1696420.934956] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696420.941228] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:41 2019][1696420.947865] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696420.954136] kdmwork-253:1 S ffff9129ec625140 0 17150 2 0x00000000 [Wed Feb 6 09:38:41 2019][1696420.961408] Call Trace: [Wed Feb 6 09:38:41 2019][1696420.964052] [] schedule+0x29/0x70 [Wed Feb 6 09:38:41 2019][1696420.969201] [] kthread_worker_fn+0xf7/0x180 [Wed Feb 6 09:38:41 2019][1696420.975226] [] ? kthread_stop+0xf0/0xf0 [Wed Feb 6 09:38:41 2019][1696420.980889] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:41 2019][1696420.985965] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696420.992234] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:41 2019][1696420.998870] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696421.005140] kdmflush S ffff9159fce6c100 0 17261 2 0x00000000 [Wed Feb 6 09:38:41 2019][1696421.012427] Call Trace: [Wed Feb 6 09:38:41 2019][1696421.015078] [] schedule+0x29/0x70 [Wed Feb 6 09:38:41 2019][1696421.020230] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:41 2019][1696421.026087] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:41 2019][1696421.032274] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:41 2019][1696421.037349] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696421.043618] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:41 2019][1696421.050246] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696421.056534] bioset S ffff9159fce6a080 0 17264 2 0x00000000 [Wed Feb 6 09:38:41 2019][1696421.063832] Call Trace: [Wed Feb 6 09:38:41 2019][1696421.066464] [] schedule+0x29/0x70 [Wed Feb 6 09:38:41 2019][1696421.071629] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:41 2019][1696421.077475] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:41 2019][1696421.083662] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:41 2019][1696421.088739] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696421.095013] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:41 2019][1696421.101651] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696421.107929] kdmwork-253:2 S ffff9159f7d15140 0 17265 2 0x00000000 [Wed Feb 6 09:38:41 2019][1696421.115212] Call Trace: [Wed Feb 6 09:38:41 2019][1696421.117856] [] schedule+0x29/0x70 [Wed Feb 6 09:38:41 2019][1696421.122998] [] kthread_worker_fn+0xf7/0x180 [Wed Feb 6 09:38:41 2019][1696421.129028] [] ? kthread_stop+0xf0/0xf0 [Wed Feb 6 09:38:41 2019][1696421.134689] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:41 2019][1696421.139753] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696421.146045] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:41 2019][1696421.152669] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696421.158944] kworker/25:2 S ffff9139f8b4e180 0 17391 2 0x00000000 [Wed Feb 6 09:38:41 2019][1696421.166251] Call Trace: [Wed Feb 6 09:38:41 2019][1696421.168882] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:41 2019][1696421.175070] [] schedule+0x29/0x70 [Wed Feb 6 09:38:41 2019][1696421.180219] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:41 2019][1696421.185972] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:41 2019][1696421.192694] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:41 2019][1696421.197755] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696421.204028] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:41 2019][1696421.210664] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696421.216939] auditd S ffff9159f6ba8000 0 17473 1 0x00000000 [Wed Feb 6 09:38:41 2019][1696421.224233] Call Trace: [Wed Feb 6 09:38:41 2019][1696421.226865] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:41 2019][1696421.233570] [] schedule+0x29/0x70 [Wed Feb 6 09:38:41 2019][1696421.238729] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:38:41 2019][1696421.245874] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:41 2019][1696421.251800] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:38:41 2019][1696421.259129] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:41 2019][1696421.265666] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:38:41 2019][1696421.270914] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:41 2019][1696421.276662] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:38:41 2019][1696421.282417] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:41 2019][1696421.288625] auditd S ffff9139f65e5140 0 17474 1 0x00000000 [Wed Feb 6 09:38:41 2019][1696421.295916] Call Trace: [Wed Feb 6 09:38:41 2019][1696421.298550] [] schedule+0x29/0x70 [Wed Feb 6 09:38:41 2019][1696421.303698] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:41 2019][1696421.309900] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:41 2019][1696421.315396] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:38:41 2019][1696421.321077] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:41 2019][1696421.326411] [] ? jbd2_complete_transaction+0x52/0xa0 [jbd2] [Wed Feb 6 09:38:41 2019][1696421.333816] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:41 2019][1696421.339151] [] ? do_fsync+0x78/0xb0 [Wed Feb 6 09:38:41 2019][1696421.344475] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:41 2019][1696421.350678] rpciod S ffff9159fce6b0c0 0 17477 2 0x00000080 [Wed Feb 6 09:38:41 2019][1696421.357960] Call Trace: [Wed Feb 6 09:38:41 2019][1696421.360606] [] schedule+0x29/0x70 [Wed Feb 6 09:38:41 2019][1696421.365750] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:41 2019][1696421.371610] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:41 2019][1696421.377801] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:41 2019][1696421.382862] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696421.389150] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:41 2019][1696421.395776] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696421.402062] xprtiod S ffff9159fce6e180 0 17478 2 0x00000080 [Wed Feb 6 09:38:41 2019][1696421.409346] Call Trace: [Wed Feb 6 09:38:41 2019][1696421.411991] [] schedule+0x29/0x70 [Wed Feb 6 09:38:41 2019][1696421.417141] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:41 2019][1696421.422993] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:41 2019][1696421.429177] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:41 2019][1696421.434253] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696421.440532] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:41 2019][1696421.447163] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:41 2019][1696421.453431] polkitd S ffff9149fec2b0c0 0 17508 1 0x00000080 [Wed Feb 6 09:38:41 2019][1696421.460717] Call Trace: [Wed Feb 6 09:38:41 2019][1696421.463349] [] schedule+0x29/0x70 [Wed Feb 6 09:38:41 2019][1696421.468504] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:41 2019][1696421.475750] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:38:41 2019][1696421.482721] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:38:41 2019][1696421.488562] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:41 2019][1696421.495098] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:41 2019][1696421.501383] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:38:41 2019][1696421.506960] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:41 2019][1696421.513253] [] ? __switch_to+0x151/0x580 [Wed Feb 6 09:38:41 2019][1696421.519010] [] ? update_curr+0x14c/0x1e0 [Wed Feb 6 09:38:41 2019][1696421.524764] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:41 2019][1696421.530273] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:41 2019][1696421.535769] [] ? ttwu_do_activate+0x6f/0x80 [Wed Feb 6 09:38:41 2019][1696421.541796] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:38:41 2019][1696421.547807] [] ? inotify_free_event+0xe/0x10 [Wed Feb 6 09:38:41 2019][1696421.553910] [] ? kfree+0x106/0x140 [Wed Feb 6 09:38:41 2019][1696421.559160] [] ? list_del+0xd/0x30 [Wed Feb 6 09:38:41 2019][1696421.564398] [] ? inotify_read+0x2ef/0x420 [Wed Feb 6 09:38:41 2019][1696421.570242] [] ? eventfd_ctx_read+0x67/0x1e0 [Wed Feb 6 09:38:41 2019][1696421.576356] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:38:41 2019][1696421.581590] [] ? __audit_syscall_exit+0x1e6/0x280 [Wed Feb 6 09:38:41 2019][1696421.588139] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:41 2019][1696421.594325] gmain S ffff9129f3156180 0 17524 1 0x00000080 [Wed Feb 6 09:38:41 2019][1696421.601597] Call Trace: [Wed Feb 6 09:38:41 2019][1696421.604247] [] ? __rmqueue+0x8a/0x460 [Wed Feb 6 09:38:41 2019][1696421.609745] [] schedule+0x29/0x70 [Wed Feb 6 09:38:41 2019][1696421.614903] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:41 2019][1696421.622125] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:38:41 2019][1696421.627979] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:41 2019][1696421.634511] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:41 2019][1696421.640783] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:38:41 2019][1696421.646384] [] ? radix_tree_lookup_slot+0x22/0x50 [Wed Feb 6 09:38:41 2019][1696421.652925] [] ? __find_get_page+0x1e/0xa0 [Wed Feb 6 09:38:41 2019][1696421.658851] [] ? filemap_fault+0x17e/0x490 [Wed Feb 6 09:38:41 2019][1696421.664791] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:41 2019][1696421.670286] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:41 2019][1696421.675795] [] ? do_read_fault.isra.61+0x139/0x1b0 [Wed Feb 6 09:38:41 2019][1696421.682417] [] ? handle_pte_fault+0x2f4/0xd10 [Wed Feb 6 09:38:41 2019][1696421.688599] [] ? eventfd_ctx_read+0x67/0x1e0 [Wed Feb 6 09:38:41 2019][1696421.694719] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:38:41 2019][1696421.699959] [] ? __audit_syscall_exit+0x1e6/0x280 [Wed Feb 6 09:38:41 2019][1696421.706492] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:41 2019][1696421.712691] gdbus S ffff9149f6954100 0 17529 1 0x00000080 [Wed Feb 6 09:38:41 2019][1696421.719971] Call Trace: [Wed Feb 6 09:38:41 2019][1696421.722613] [] schedule+0x29/0x70 [Wed Feb 6 09:38:41 2019][1696421.727757] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:41 2019][1696421.734986] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:38:41 2019][1696421.740829] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:38:41 2019][1696421.746684] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:41 2019][1696421.753225] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:41 2019][1696421.759515] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:38:41 2019][1696421.765100] [] ? sock_sendmsg+0xb6/0xf0 [Wed Feb 6 09:38:41 2019][1696421.770779] [] ? sock_recvmsg+0xc5/0x100 [Wed Feb 6 09:38:41 2019][1696421.776535] [] ? update_curr+0x14c/0x1e0 [Wed Feb 6 09:38:41 2019][1696421.782307] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:41 2019][1696421.787804] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:41 2019][1696421.793301] [] ? ___sys_sendmsg+0x3e9/0x400 [Wed Feb 6 09:38:41 2019][1696421.799328] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:38:41 2019][1696421.805780] [] ? pollwake+0x73/0x90 [Wed Feb 6 09:38:41 2019][1696421.811110] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:41 2019][1696421.816860] [] ? eventfd_ctx_read+0x67/0x1e0 [Wed Feb 6 09:38:41 2019][1696421.822977] [] ? __sys_sendmsg+0x85/0x90 [Wed Feb 6 09:38:41 2019][1696421.828728] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:38:41 2019][1696421.833961] [] ? __audit_syscall_exit+0x1e6/0x280 [Wed Feb 6 09:38:41 2019][1696421.840520] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:41 2019][1696421.846712] polkitd S ffff9139f6cea080 0 17530 1 0x00000080 [Wed Feb 6 09:38:41 2019][1696421.853994] Call Trace: [Wed Feb 6 09:38:41 2019][1696421.856639] [] ? get_page_from_freelist+0x412/0xa70 [Wed Feb 6 09:38:42 2019][1696421.863341] [] schedule+0x29/0x70 [Wed Feb 6 09:38:42 2019][1696421.868504] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:42 2019][1696421.875734] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:38:42 2019][1696421.881593] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:42 2019][1696421.888133] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:42 2019][1696421.894421] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:38:42 2019][1696421.900008] [] ? __enqueue_entity+0x78/0x80 [Wed Feb 6 09:38:42 2019][1696421.906033] [] ? __alloc_pages_nodemask+0x176/0x420 [Wed Feb 6 09:38:42 2019][1696421.912738] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696421.918245] [] ? physflat_send_IPI_mask+0xe/0x10 [Wed Feb 6 09:38:42 2019][1696421.924686] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:38:42 2019][1696421.930718] [] ? wake_up_q+0x5b/0x80 [Wed Feb 6 09:38:42 2019][1696421.936129] [] ? futex_wake+0x90/0x180 [Wed Feb 6 09:38:42 2019][1696421.941711] [] ? eventfd_ctx_read+0x67/0x1e0 [Wed Feb 6 09:38:42 2019][1696421.947825] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:38:42 2019][1696421.953060] [] ? __audit_syscall_exit+0x1e6/0x280 [Wed Feb 6 09:38:42 2019][1696421.959615] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:42 2019][1696421.965805] JS GC Helper S ffff9139f6ceb0c0 0 17531 1 0x00000080 [Wed Feb 6 09:38:42 2019][1696421.973086] Call Trace: [Wed Feb 6 09:38:42 2019][1696421.975732] [] schedule+0x29/0x70 [Wed Feb 6 09:38:42 2019][1696421.980877] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:42 2019][1696421.987078] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:42 2019][1696421.992576] [] ? handle_pte_fault+0x2f4/0xd10 [Wed Feb 6 09:38:42 2019][1696421.998774] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:42 2019][1696422.004093] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:42 2019][1696422.009428] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:42 2019][1696422.015614] JS Sour~ Thread S ffff9139f6ce9040 0 17533 1 0x00000080 [Wed Feb 6 09:38:42 2019][1696422.022898] Call Trace: [Wed Feb 6 09:38:42 2019][1696422.025548] [] schedule+0x29/0x70 [Wed Feb 6 09:38:42 2019][1696422.030695] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:42 2019][1696422.036895] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:42 2019][1696422.042395] [] ? lru_cache_add+0xe/0x10 [Wed Feb 6 09:38:42 2019][1696422.048075] [] ? page_add_new_anon_rmap+0xb8/0x170 [Wed Feb 6 09:38:42 2019][1696422.054693] [] ? get_futex_key+0x1c8/0x2c0 [Wed Feb 6 09:38:42 2019][1696422.060635] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:42 2019][1696422.065961] [] ? yama_task_prctl+0x24/0x110 [Wed Feb 6 09:38:42 2019][1696422.071989] [] ? strlcpy+0x42/0x60 [Wed Feb 6 09:38:42 2019][1696422.077227] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:42 2019][1696422.082563] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:42 2019][1696422.088755] polkitd S ffff9139f83ab0c0 0 17539 1 0x00000080 [Wed Feb 6 09:38:42 2019][1696422.096053] Call Trace: [Wed Feb 6 09:38:42 2019][1696422.098681] [] schedule+0x29/0x70 [Wed Feb 6 09:38:42 2019][1696422.103822] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:42 2019][1696422.111065] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:42 2019][1696422.116987] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:38:42 2019][1696422.122827] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:42 2019][1696422.129380] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:42 2019][1696422.135658] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:38:42 2019][1696422.141241] [] ? __alloc_pages_nodemask+0x176/0x420 [Wed Feb 6 09:38:42 2019][1696422.147962] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.153461] [] ? physflat_send_IPI_mask+0xe/0x10 [Wed Feb 6 09:38:42 2019][1696422.159922] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:38:42 2019][1696422.165940] [] ? wake_up_q+0x5b/0x80 [Wed Feb 6 09:38:42 2019][1696422.171362] [] ? futex_wake+0x90/0x180 [Wed Feb 6 09:38:42 2019][1696422.176948] [] ? eventfd_ctx_read+0x67/0x1e0 [Wed Feb 6 09:38:42 2019][1696422.183064] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:38:42 2019][1696422.188301] [] ? __audit_syscall_exit+0x1e6/0x280 [Wed Feb 6 09:38:42 2019][1696422.194835] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:42 2019][1696422.201025] smartd S ffff9149d5fca080 0 17510 1 0x00000080 [Wed Feb 6 09:38:42 2019][1696422.208326] Call Trace: [Wed Feb 6 09:38:42 2019][1696422.210965] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:42 2019][1696422.217675] [] schedule+0x29/0x70 [Wed Feb 6 09:38:42 2019][1696422.222836] [] do_nanosleep+0x86/0xf0 [Wed Feb 6 09:38:42 2019][1696422.228335] [] hrtimer_nanosleep+0xbb/0x180 [Wed Feb 6 09:38:42 2019][1696422.234348] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:42 2019][1696422.240277] [] ? do_nanosleep+0x72/0xf0 [Wed Feb 6 09:38:42 2019][1696422.245959] [] SyS_nanosleep+0x96/0xb0 [Wed Feb 6 09:38:42 2019][1696422.251543] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:42 2019][1696422.257744] dbus-daemon S ffff9149d5fcd140 0 17512 1 0x00000080 [Wed Feb 6 09:38:42 2019][1696422.265033] Call Trace: [Wed Feb 6 09:38:42 2019][1696422.267687] [] schedule+0x29/0x70 [Wed Feb 6 09:38:42 2019][1696422.272837] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:42 2019][1696422.280081] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Wed Feb 6 09:38:42 2019][1696422.287053] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:42 2019][1696422.293600] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:38:42 2019][1696422.298838] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:42 2019][1696422.304608] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:38:42 2019][1696422.310364] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:42 2019][1696422.316568] rpcbind S ffff9149f7215140 0 17518 1 0x00000080 [Wed Feb 6 09:38:42 2019][1696422.323848] Call Trace: [Wed Feb 6 09:38:42 2019][1696422.326487] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:42 2019][1696422.333193] [] schedule+0x29/0x70 [Wed Feb 6 09:38:42 2019][1696422.338358] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:38:42 2019][1696422.345501] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:42 2019][1696422.351445] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:38:42 2019][1696422.358760] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:42 2019][1696422.365306] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:42 2019][1696422.371579] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:38:42 2019][1696422.377174] [] ? __mem_cgroup_uncharge_common+0xcd/0x2f0 [Wed Feb 6 09:38:42 2019][1696422.384316] [] ? lru_cache_add+0xe/0x10 [Wed Feb 6 09:38:42 2019][1696422.390004] [] ? putback_lru_page+0x52/0xb0 [Wed Feb 6 09:38:42 2019][1696422.396021] [] ? migrate_pages+0x2d7/0x790 [Wed Feb 6 09:38:42 2019][1696422.401946] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.407459] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.412955] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.418447] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.423955] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.429449] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.434943] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.440453] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:38:42 2019][1696422.446297] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:38:42 2019][1696422.451547] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:42 2019][1696422.457735] sssd S ffff9129f6526180 0 17535 1 0x00000080 [Wed Feb 6 09:38:42 2019][1696422.465027] Call Trace: [Wed Feb 6 09:38:42 2019][1696422.467658] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:42 2019][1696422.474367] [] schedule+0x29/0x70 [Wed Feb 6 09:38:42 2019][1696422.479539] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:38:42 2019][1696422.486686] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:42 2019][1696422.492612] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:38:42 2019][1696422.499938] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:42 2019][1696422.506467] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:42 2019][1696422.512757] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:38:42 2019][1696422.518334] [] ? sock_sendmsg+0xb6/0xf0 [Wed Feb 6 09:38:42 2019][1696422.524003] [] ? sock_recvmsg+0xc5/0x100 [Wed Feb 6 09:38:42 2019][1696422.529773] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.535271] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.540776] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.546268] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.551775] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.557262] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.562754] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.568259] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:42 2019][1696422.573753] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:38:42 2019][1696422.579606] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:38:42 2019][1696422.584838] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:42 2019][1696422.591034] irqbalance S ffff9129f6521040 0 17549 1 0x00000080 [Wed Feb 6 09:38:42 2019][1696422.598317] Call Trace: [Wed Feb 6 09:38:42 2019][1696422.600961] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:42 2019][1696422.607663] [] schedule+0x29/0x70 [Wed Feb 6 09:38:42 2019][1696422.612824] [] do_nanosleep+0x86/0xf0 [Wed Feb 6 09:38:42 2019][1696422.618317] [] hrtimer_nanosleep+0xbb/0x180 [Wed Feb 6 09:38:42 2019][1696422.624331] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:42 2019][1696422.630274] [] ? do_nanosleep+0x72/0xf0 [Wed Feb 6 09:38:42 2019][1696422.635943] [] SyS_nanosleep+0x96/0xb0 [Wed Feb 6 09:38:42 2019][1696422.641543] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:42 2019][1696422.647728] mdadm S ffff9139fbad6180 0 17554 1 0x00000080 [Wed Feb 6 09:38:42 2019][1696422.654997] Call Trace: [Wed Feb 6 09:38:42 2019][1696422.657653] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:42 2019][1696422.664366] [] schedule+0x29/0x70 [Wed Feb 6 09:38:42 2019][1696422.669532] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:38:42 2019][1696422.676667] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:42 2019][1696422.682612] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:38:42 2019][1696422.689918] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:42 2019][1696422.696448] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:42 2019][1696422.702737] [] do_select+0x6d1/0x7c0 [Wed Feb 6 09:38:42 2019][1696422.708143] [] ? __alloc_pages_nodemask+0x176/0x420 [Wed Feb 6 09:38:42 2019][1696422.714863] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:38:42 2019][1696422.721834] [] ? lru_cache_add+0xe/0x10 [Wed Feb 6 09:38:42 2019][1696422.727515] [] ? page_add_new_anon_rmap+0xb8/0x170 [Wed Feb 6 09:38:42 2019][1696422.734134] [] ? handle_pte_fault+0x5b8/0xd10 [Wed Feb 6 09:38:42 2019][1696422.740335] [] ? __mem_cgroup_uncharge_common+0xcd/0x2f0 [Wed Feb 6 09:38:42 2019][1696422.747478] [] ? free_hot_cold_page_list+0x46/0xa0 [Wed Feb 6 09:38:42 2019][1696422.754115] [] ? release_pages+0x24e/0x430 [Wed Feb 6 09:38:42 2019][1696422.760046] [] core_sys_select+0x1ff/0x340 [Wed Feb 6 09:38:42 2019][1696422.765975] [] ? evict+0x115/0x180 [Wed Feb 6 09:38:42 2019][1696422.771221] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:38:42 2019][1696422.777056] [] ? d_free+0x4f/0x70 [Wed Feb 6 09:38:42 2019][1696422.782218] [] ? __dentry_kill+0x128/0x180 [Wed Feb 6 09:38:42 2019][1696422.788140] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:38:42 2019][1696422.793974] [] SyS_select+0xca/0x120 [Wed Feb 6 09:38:42 2019][1696422.799400] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:42 2019][1696422.805589] gssproxy S ffff9129f3151040 0 17563 1 0x00000080 [Wed Feb 6 09:38:42 2019][1696422.812884] Call Trace: [Wed Feb 6 09:38:42 2019][1696422.815511] [] schedule+0x29/0x70 [Wed Feb 6 09:38:42 2019][1696422.820674] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:42 2019][1696422.827899] [] ? __enqueue_entity+0x78/0x80 [Wed Feb 6 09:38:42 2019][1696422.833912] [] ? enqueue_entity+0x2ef/0xbe0 [Wed Feb 6 09:38:42 2019][1696422.839945] [] ? cpumask_next_and+0x35/0x50 [Wed Feb 6 09:38:42 2019][1696422.845961] [] ? security_inode_alloc+0x24/0x30 [Wed Feb 6 09:38:42 2019][1696422.852322] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:42 2019][1696422.858872] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:38:43 2019][1696422.864109] [] ? SyS_epoll_ctl+0x130/0xc80 [Wed Feb 6 09:38:43 2019][1696422.870051] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:43 2019][1696422.875806] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:38:43 2019][1696422.881565] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:43 2019][1696422.887767] gssproxy S ffff9159f7a85140 0 17572 1 0x00000080 [Wed Feb 6 09:38:43 2019][1696422.895060] Call Trace: [Wed Feb 6 09:38:43 2019][1696422.897703] [] schedule+0x29/0x70 [Wed Feb 6 09:38:43 2019][1696422.902853] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:43 2019][1696422.909053] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:43 2019][1696422.914553] [] ? physflat_send_IPI_mask+0xe/0x10 [Wed Feb 6 09:38:43 2019][1696422.921014] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:38:43 2019][1696422.927031] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:43 2019][1696422.932367] [] ? call_rwsem_wake+0x1b/0x30 [Wed Feb 6 09:38:43 2019][1696422.938299] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:43 2019][1696422.943631] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:43 2019][1696422.949813] gssproxy S ffff9159f7a82080 0 17573 1 0x00000080 [Wed Feb 6 09:38:43 2019][1696422.957097] Call Trace: [Wed Feb 6 09:38:43 2019][1696422.959729] [] schedule+0x29/0x70 [Wed Feb 6 09:38:43 2019][1696422.964878] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:43 2019][1696422.971085] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:43 2019][1696422.976582] [] ? handle_pte_fault+0x2f4/0xd10 [Wed Feb 6 09:38:43 2019][1696422.982767] [] ? sched_clock_cpu+0x85/0xc0 [Wed Feb 6 09:38:43 2019][1696422.988705] [] ? idle_balance+0x7f/0x250 [Wed Feb 6 09:38:43 2019][1696422.994456] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:43 2019][1696422.999793] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:43 2019][1696423.005117] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:43 2019][1696423.011314] gssproxy S ffff9159fb6a9040 0 17574 1 0x00000080 [Wed Feb 6 09:38:43 2019][1696423.018611] Call Trace: [Wed Feb 6 09:38:43 2019][1696423.021250] [] schedule+0x29/0x70 [Wed Feb 6 09:38:43 2019][1696423.026393] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:43 2019][1696423.032599] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:43 2019][1696423.038093] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:43 2019][1696423.043408] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:43 2019][1696423.048747] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:43 2019][1696423.054935] gssproxy S ffff9159fb6aa080 0 17575 1 0x00000080 [Wed Feb 6 09:38:43 2019][1696423.062223] Call Trace: [Wed Feb 6 09:38:43 2019][1696423.064873] [] schedule+0x29/0x70 [Wed Feb 6 09:38:43 2019][1696423.070014] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:43 2019][1696423.076219] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:43 2019][1696423.081709] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:43 2019][1696423.087032] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:43 2019][1696423.092364] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:43 2019][1696423.098551] gssproxy S ffff9159fa269040 0 17576 1 0x00000080 [Wed Feb 6 09:38:43 2019][1696423.105850] Call Trace: [Wed Feb 6 09:38:43 2019][1696423.108480] [] schedule+0x29/0x70 [Wed Feb 6 09:38:43 2019][1696423.113629] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:43 2019][1696423.119828] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:43 2019][1696423.125316] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:43 2019][1696423.130631] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:43 2019][1696423.135971] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:43 2019][1696423.142158] dsm_sa_eventmgr S ffff9139ee339040 0 17568 1 0x00000080 [Wed Feb 6 09:38:43 2019][1696423.149441] Call Trace: [Wed Feb 6 09:38:43 2019][1696423.152087] [] schedule+0x29/0x70 [Wed Feb 6 09:38:43 2019][1696423.157230] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:43 2019][1696423.164475] [] ? tlb_finish_mmu+0x55/0x60 [Wed Feb 6 09:38:43 2019][1696423.170315] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:43 2019][1696423.176420] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:43 2019][1696423.182971] [] do_sigtimedwait+0x196/0x240 [Wed Feb 6 09:38:43 2019][1696423.188903] [] SYSC_rt_sigtimedwait+0xb4/0x120 [Wed Feb 6 09:38:43 2019][1696423.195191] [] ? __do_page_fault+0x228/0x500 [Wed Feb 6 09:38:43 2019][1696423.201293] [] SyS_rt_sigtimedwait+0xe/0x10 [Wed Feb 6 09:38:43 2019][1696423.207323] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:43 2019][1696423.213505] dsm_sa_eventmgr S ffff9149f7211040 0 17587 1 0x00000080 [Wed Feb 6 09:38:43 2019][1696423.220778] Call Trace: [Wed Feb 6 09:38:43 2019][1696423.223413] [] ? security_ipc_permission+0x1f/0x30 [Wed Feb 6 09:38:43 2019][1696423.230047] [] schedule+0x29/0x70 [Wed Feb 6 09:38:43 2019][1696423.235193] [] SYSC_semtimedop+0x89a/0xc90 [Wed Feb 6 09:38:43 2019][1696423.241121] [] ? futex_wake+0x90/0x180 [Wed Feb 6 09:38:43 2019][1696423.246715] [] ? do_smart_update+0x11d/0x130 [Wed Feb 6 09:38:43 2019][1696423.252818] [] SyS_semtimedop+0xe/0x10 [Wed Feb 6 09:38:43 2019][1696423.258411] [] SyS_semop+0x10/0x20 [Wed Feb 6 09:38:43 2019][1696423.263648] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:43 2019][1696423.269830] dsm_sa_eventmgr S ffff9159fc6dd140 0 18952 1 0x00000080 [Wed Feb 6 09:38:43 2019][1696423.277126] Call Trace: [Wed Feb 6 09:38:43 2019][1696423.279757] [] schedule+0x29/0x70 [Wed Feb 6 09:38:43 2019][1696423.284902] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:43 2019][1696423.292147] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:38:43 2019][1696423.299115] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:38:43 2019][1696423.304957] [] ? __pollwait+0x7f/0xf0 [Wed Feb 6 09:38:43 2019][1696423.310465] [] ? list_del+0xd/0x30 [Wed Feb 6 09:38:43 2019][1696423.315697] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:43 2019][1696423.322233] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:43 2019][1696423.328521] [] do_select+0x6d1/0x7c0 [Wed Feb 6 09:38:43 2019][1696423.333934] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:43 2019][1696423.339440] [] ? tracing_is_on+0x15/0x30 [Wed Feb 6 09:38:43 2019][1696423.345190] [] ? tracing_record_cmdline+0x1d/0x120 [Wed Feb 6 09:38:43 2019][1696423.351822] [] ? probe_sched_wakeup+0x2b/0xa0 [Wed Feb 6 09:38:43 2019][1696423.358006] [] ? ttwu_do_wakeup+0xb5/0xe0 [Wed Feb 6 09:38:43 2019][1696423.363854] [] ? do_smart_update+0x8f/0x130 [Wed Feb 6 09:38:43 2019][1696423.369878] [] ? SYSC_semtimedop+0x254/0xc90 [Wed Feb 6 09:38:43 2019][1696423.375975] [] core_sys_select+0x1ff/0x340 [Wed Feb 6 09:38:43 2019][1696423.381921] [] ? do_futex+0x12a/0x5a0 [Wed Feb 6 09:38:43 2019][1696423.387416] [] SyS_select+0xca/0x120 [Wed Feb 6 09:38:43 2019][1696423.392835] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:43 2019][1696423.399017] chronyd S ffff9149f83fa080 0 17580 1 0x00000080 [Wed Feb 6 09:38:43 2019][1696423.406315] Call Trace: [Wed Feb 6 09:38:43 2019][1696423.408953] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:43 2019][1696423.415663] [] schedule+0x29/0x70 [Wed Feb 6 09:38:43 2019][1696423.420822] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:38:43 2019][1696423.427957] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:43 2019][1696423.433884] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:38:43 2019][1696423.441215] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:43 2019][1696423.447743] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:43 2019][1696423.454019] [] do_select+0x6d1/0x7c0 [Wed Feb 6 09:38:43 2019][1696423.459445] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:43 2019][1696423.464941] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:43 2019][1696423.470437] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:43 2019][1696423.475941] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:43 2019][1696423.481431] [] ? sock_sendmsg+0xb6/0xf0 [Wed Feb 6 09:38:43 2019][1696423.487113] [] ? ___sys_recvmsg+0x133/0x2d0 [Wed Feb 6 09:38:43 2019][1696423.493123] [] ? __check_object_size+0x1ca/0x250 [Wed Feb 6 09:38:43 2019][1696423.499572] [] core_sys_select+0x1ff/0x340 [Wed Feb 6 09:38:43 2019][1696423.505517] [] ? __d_free+0x35/0x40 [Wed Feb 6 09:38:43 2019][1696423.510841] [] ? d_free+0x60/0x70 [Wed Feb 6 09:38:43 2019][1696423.515991] [] ? __dentry_kill+0x128/0x180 [Wed Feb 6 09:38:43 2019][1696423.521930] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:38:43 2019][1696423.527765] [] SyS_select+0xca/0x120 [Wed Feb 6 09:38:43 2019][1696423.533187] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:43 2019][1696423.539369] sssd_nss S ffff9129f80a8000 0 17601 17535 0x00000080 [Wed Feb 6 09:38:43 2019][1696423.546636] Call Trace: [Wed Feb 6 09:38:43 2019][1696423.549271] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:43 2019][1696423.555993] [] schedule+0x29/0x70 [Wed Feb 6 09:38:43 2019][1696423.561145] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:38:43 2019][1696423.568285] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:43 2019][1696423.574224] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:38:43 2019][1696423.581537] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:43 2019][1696423.588087] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:38:43 2019][1696423.593321] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:43 2019][1696423.599441] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:43 2019][1696423.605200] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:38:43 2019][1696423.610954] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:43 2019][1696423.617153] sssd_pam S ffff9129f80ae180 0 17602 17535 0x00000080 [Wed Feb 6 09:38:43 2019][1696423.624434] Call Trace: [Wed Feb 6 09:38:43 2019][1696423.627074] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:43 2019][1696423.633778] [] schedule+0x29/0x70 [Wed Feb 6 09:38:43 2019][1696423.638942] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:38:43 2019][1696423.646088] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:43 2019][1696423.652013] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:38:43 2019][1696423.659345] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:43 2019][1696423.665883] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:38:43 2019][1696423.671117] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:43 2019][1696423.677230] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:43 2019][1696423.682981] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:38:43 2019][1696423.688749] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:43 2019][1696423.694932] mlx4 S ffff914aa95ba080 0 17701 2 0x00000080 [Wed Feb 6 09:38:43 2019][1696423.702222] Call Trace: [Wed Feb 6 09:38:43 2019][1696423.704873] [] schedule+0x29/0x70 [Wed Feb 6 09:38:43 2019][1696423.710025] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:43 2019][1696423.715880] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:43 2019][1696423.722071] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:43 2019][1696423.727141] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:43 2019][1696423.733412] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:43 2019][1696423.740047] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:43 2019][1696423.746318] mlx4_ib S ffff9159f65ce180 0 17716 2 0x00000080 [Wed Feb 6 09:38:43 2019][1696423.753618] Call Trace: [Wed Feb 6 09:38:43 2019][1696423.756263] [] schedule+0x29/0x70 [Wed Feb 6 09:38:43 2019][1696423.761406] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:43 2019][1696423.767262] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:43 2019][1696423.773445] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:43 2019][1696423.778508] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:43 2019][1696423.784799] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:43 2019][1696423.791424] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:43 2019][1696423.797700] mlx4_ib_mcg S ffff9159f65c9040 0 17717 2 0x00000080 [Wed Feb 6 09:38:43 2019][1696423.805004] Call Trace: [Wed Feb 6 09:38:43 2019][1696423.807634] [] schedule+0x29/0x70 [Wed Feb 6 09:38:43 2019][1696423.812777] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:43 2019][1696423.818632] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:43 2019][1696423.824815] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:43 2019][1696423.829879] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:43 2019][1696423.836167] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:43 2019][1696423.842794] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:43 2019][1696423.849081] ipoib_flush S ffff9159f65cd140 0 17808 2 0x00000080 [Wed Feb 6 09:38:43 2019][1696423.856370] Call Trace: [Wed Feb 6 09:38:43 2019][1696423.859017] [] schedule+0x29/0x70 [Wed Feb 6 09:38:43 2019][1696423.864166] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:44 2019][1696423.870018] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:44 2019][1696423.876198] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:44 2019][1696423.881278] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:44 2019][1696423.887565] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:44 2019][1696423.894188] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:44 2019][1696423.900473] mlx5i S ffff9159f65cc100 0 17809 2 0x00000080 [Wed Feb 6 09:38:44 2019][1696423.907747] Call Trace: [Wed Feb 6 09:38:44 2019][1696423.910381] [] schedule+0x29/0x70 [Wed Feb 6 09:38:44 2019][1696423.915536] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:44 2019][1696423.921371] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:44 2019][1696423.927569] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:44 2019][1696423.932648] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:44 2019][1696423.938925] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:44 2019][1696423.945558] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:44 2019][1696423.951838] ipoib_wq S ffff9159f65ca080 0 17813 2 0x00000080 [Wed Feb 6 09:38:44 2019][1696423.959158] Call Trace: [Wed Feb 6 09:38:44 2019][1696423.961783] [] schedule+0x29/0x70 [Wed Feb 6 09:38:44 2019][1696423.966927] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:44 2019][1696423.972783] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:44 2019][1696423.978974] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:44 2019][1696423.984050] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:44 2019][1696423.990325] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:44 2019][1696423.996962] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:44 2019][1696424.003233] rdma_cm S ffff9129f5cc0000 0 18185 2 0x00000080 [Wed Feb 6 09:38:44 2019][1696424.010514] Call Trace: [Wed Feb 6 09:38:44 2019][1696424.013161] [] schedule+0x29/0x70 [Wed Feb 6 09:38:44 2019][1696424.018305] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:44 2019][1696424.024160] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:44 2019][1696424.030349] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:44 2019][1696424.035425] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:44 2019][1696424.041697] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:44 2019][1696424.048319] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:44 2019][1696424.054603] kworker/38:1H S ffff9149f6e60000 0 18229 2 0x00000080 [Wed Feb 6 09:38:44 2019][1696424.061882] Call Trace: [Wed Feb 6 09:38:44 2019][1696424.064516] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:44 2019][1696424.070720] [] schedule+0x29/0x70 [Wed Feb 6 09:38:44 2019][1696424.075872] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:44 2019][1696424.081622] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:44 2019][1696424.088348] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:44 2019][1696424.093411] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:44 2019][1696424.099687] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:44 2019][1696424.106322] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:44 2019][1696424.112599] dsm_sa_datamgrd S ffff9129f3c00000 0 18343 1 0x00000080 [Wed Feb 6 09:38:44 2019][1696424.119892] Call Trace: [Wed Feb 6 09:38:44 2019][1696424.122521] [] schedule+0x29/0x70 [Wed Feb 6 09:38:44 2019][1696424.127670] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:44 2019][1696424.134912] [] ? __slab_free+0x81/0x2f0 [Wed Feb 6 09:38:44 2019][1696424.140575] [] ? unmap_region+0xf4/0x140 [Wed Feb 6 09:38:44 2019][1696424.146328] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:44 2019][1696424.152439] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:44 2019][1696424.158976] [] do_sigtimedwait+0x196/0x240 [Wed Feb 6 09:38:44 2019][1696424.164906] [] SYSC_rt_sigtimedwait+0xb4/0x120 [Wed Feb 6 09:38:44 2019][1696424.171196] [] ? do_munmap+0x327/0x480 [Wed Feb 6 09:38:44 2019][1696424.176781] [] SyS_rt_sigtimedwait+0xe/0x10 [Wed Feb 6 09:38:44 2019][1696424.182792] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:44 2019][1696424.188992] dsm_sa_datamgrd S ffff9149f6951040 0 18866 1 0x00000080 [Wed Feb 6 09:38:44 2019][1696424.196263] Call Trace: [Wed Feb 6 09:38:44 2019][1696424.198897] [] ? security_ipc_permission+0x1f/0x30 [Wed Feb 6 09:38:44 2019][1696424.205517] [] schedule+0x29/0x70 [Wed Feb 6 09:38:44 2019][1696424.210680] [] SYSC_semtimedop+0x89a/0xc90 [Wed Feb 6 09:38:44 2019][1696424.216612] [] ? tracing_is_on+0x15/0x30 [Wed Feb 6 09:38:44 2019][1696424.222381] [] ? tracing_record_cmdline+0x1d/0x120 [Wed Feb 6 09:38:44 2019][1696424.229007] [] ? probe_sched_switch+0x2d/0xa0 [Wed Feb 6 09:38:44 2019][1696424.235207] [] ? handle_mm_fault+0x39d/0x9b0 [Wed Feb 6 09:38:44 2019][1696424.241312] [] ? __do_page_fault+0x228/0x500 [Wed Feb 6 09:38:44 2019][1696424.247427] [] SyS_semtimedop+0xe/0x10 [Wed Feb 6 09:38:44 2019][1696424.253003] [] SyS_semop+0x10/0x20 [Wed Feb 6 09:38:44 2019][1696424.258251] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:44 2019][1696424.264435] dsm_sa_datamgrd S ffff9149f6950000 0 18867 1 0x00000080 [Wed Feb 6 09:38:44 2019][1696424.271726] Call Trace: [Wed Feb 6 09:38:44 2019][1696424.274356] [] schedule+0x29/0x70 [Wed Feb 6 09:38:44 2019][1696424.279505] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:44 2019][1696424.286750] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:38:44 2019][1696424.292596] [] ? __pollwait+0x7f/0xf0 [Wed Feb 6 09:38:44 2019][1696424.298090] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:44 2019][1696424.304638] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:44 2019][1696424.310912] [] do_select+0x6d1/0x7c0 [Wed Feb 6 09:38:44 2019][1696424.316327] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:44 2019][1696424.321814] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:44 2019][1696424.327301] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:44 2019][1696424.332815] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:44 2019][1696424.338313] [] ? list_del+0xd/0x30 [Wed Feb 6 09:38:44 2019][1696424.343546] [] ? __rmqueue+0x8a/0x460 [Wed Feb 6 09:38:44 2019][1696424.349056] [] ? zone_statistics+0x88/0xa0 [Wed Feb 6 09:38:44 2019][1696424.354984] [] core_sys_select+0x1ff/0x340 [Wed Feb 6 09:38:44 2019][1696424.360928] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:44 2019][1696424.366415] [] ? physflat_send_IPI_mask+0xe/0x10 [Wed Feb 6 09:38:44 2019][1696424.372865] [] ? native_smp_send_reschedule+0x52/0x70 [Wed Feb 6 09:38:44 2019][1696424.379762] [] ? resched_curr+0xae/0xc0 [Wed Feb 6 09:38:44 2019][1696424.385436] [] ? check_preempt_curr+0x80/0xa0 [Wed Feb 6 09:38:44 2019][1696424.391625] [] ? wake_up_new_task+0x131/0x1a0 [Wed Feb 6 09:38:44 2019][1696424.397824] [] SyS_select+0xca/0x120 [Wed Feb 6 09:38:44 2019][1696424.403232] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:44 2019][1696424.409436] dsm_sa_datamgrd S ffff9149f69530c0 0 18868 1 0x00000080 [Wed Feb 6 09:38:44 2019][1696424.416705] Call Trace: [Wed Feb 6 09:38:44 2019][1696424.419341] [] ? security_ipc_permission+0x1f/0x30 [Wed Feb 6 09:38:44 2019][1696424.425960] [] schedule+0x29/0x70 [Wed Feb 6 09:38:44 2019][1696424.431121] [] SYSC_semtimedop+0x89a/0xc90 [Wed Feb 6 09:38:44 2019][1696424.437046] [] ? SYSC_sendto+0x17e/0x1c0 [Wed Feb 6 09:38:44 2019][1696424.442811] [] SyS_semtimedop+0xe/0x10 [Wed Feb 6 09:38:44 2019][1696424.448387] [] SyS_semop+0x10/0x20 [Wed Feb 6 09:38:44 2019][1696424.453637] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:44 2019][1696424.459824] dsm_sa_datamgrd S ffff9149f6955140 0 18869 1 0x00000080 [Wed Feb 6 09:38:44 2019][1696424.467095] Call Trace: [Wed Feb 6 09:38:44 2019][1696424.469745] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:44 2019][1696424.476453] [] schedule+0x29/0x70 [Wed Feb 6 09:38:44 2019][1696424.481615] [] do_nanosleep+0x86/0xf0 [Wed Feb 6 09:38:44 2019][1696424.487113] [] hrtimer_nanosleep+0xbb/0x180 [Wed Feb 6 09:38:44 2019][1696424.493139] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:44 2019][1696424.499062] [] ? do_nanosleep+0x72/0xf0 [Wed Feb 6 09:38:44 2019][1696424.504745] [] SyS_nanosleep+0x96/0xb0 [Wed Feb 6 09:38:44 2019][1696424.510321] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:44 2019][1696424.516514] dsm_sa_datamgrd S ffff9159fce69040 0 18870 1 0x00000080 [Wed Feb 6 09:38:44 2019][1696424.523814] Call Trace: [Wed Feb 6 09:38:44 2019][1696424.526444] [] ? security_ipc_permission+0x1f/0x30 [Wed Feb 6 09:38:44 2019][1696424.533063] [] schedule+0x29/0x70 [Wed Feb 6 09:38:44 2019][1696424.538224] [] SYSC_semtimedop+0x89a/0xc90 [Wed Feb 6 09:38:44 2019][1696424.544150] [] SyS_semtimedop+0xe/0x10 [Wed Feb 6 09:38:44 2019][1696424.549732] [] SyS_semop+0x10/0x20 [Wed Feb 6 09:38:44 2019][1696424.554979] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:44 2019][1696424.561166] dsm_sa_datamgrd S ffff9159fce6d140 0 18871 1 0x00000080 [Wed Feb 6 09:38:44 2019][1696424.568462] Call Trace: [Wed Feb 6 09:38:44 2019][1696424.571092] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:44 2019][1696424.577796] [] schedule+0x29/0x70 [Wed Feb 6 09:38:44 2019][1696424.582956] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:38:44 2019][1696424.590093] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:44 2019][1696424.596037] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:38:44 2019][1696424.603355] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:44 2019][1696424.609889] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:44 2019][1696424.616176] [] do_select+0x6d1/0x7c0 [Wed Feb 6 09:38:44 2019][1696424.621587] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:44 2019][1696424.627093] [] ? security_inode_permission+0x22/0x30 [Wed Feb 6 09:38:44 2019][1696424.633890] [] ? __inode_permission+0x52/0xd0 [Wed Feb 6 09:38:44 2019][1696424.640087] [] ? inode_permission+0x18/0x50 [Wed Feb 6 09:38:44 2019][1696424.646100] [] ? link_path_walk+0x27e/0x8b0 [Wed Feb 6 09:38:44 2019][1696424.652128] [] ? __d_lookup+0x7a/0x160 [Wed Feb 6 09:38:44 2019][1696424.657714] [] ? security_inode_permission+0x22/0x30 [Wed Feb 6 09:38:44 2019][1696424.664520] [] ? __inode_permission+0x52/0xd0 [Wed Feb 6 09:38:44 2019][1696424.670705] [] ? inode_permission+0x18/0x50 [Wed Feb 6 09:38:44 2019][1696424.676733] [] ? link_path_walk+0x27e/0x8b0 [Wed Feb 6 09:38:44 2019][1696424.682747] [] ? __d_lookup+0x7a/0x160 [Wed Feb 6 09:38:44 2019][1696424.688333] [] ? lookup_fast+0x12d/0x230 [Wed Feb 6 09:38:44 2019][1696424.694087] [] ? mntput+0x24/0x40 [Wed Feb 6 09:38:44 2019][1696424.699238] [] ? path_lookupat+0x122/0x8b0 [Wed Feb 6 09:38:44 2019][1696424.705179] [] ? kmem_cache_alloc+0x35/0x1f0 [Wed Feb 6 09:38:44 2019][1696424.711283] [] ? getname_flags+0x4f/0x1a0 [Wed Feb 6 09:38:44 2019][1696424.717138] [] core_sys_select+0x1ff/0x340 [Wed Feb 6 09:38:44 2019][1696424.723072] [] ? cp_new_stat+0x165/0x1a0 [Wed Feb 6 09:38:44 2019][1696424.728835] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:38:44 2019][1696424.734676] [] SyS_select+0xca/0x120 [Wed Feb 6 09:38:44 2019][1696424.740096] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:44 2019][1696424.746289] dsm_sa_datamgrd S ffff9129f6522080 0 18872 1 0x00000080 [Wed Feb 6 09:38:44 2019][1696424.753590] Call Trace: [Wed Feb 6 09:38:44 2019][1696424.756221] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:44 2019][1696424.762925] [] schedule+0x29/0x70 [Wed Feb 6 09:38:44 2019][1696424.768081] [] do_nanosleep+0x86/0xf0 [Wed Feb 6 09:38:44 2019][1696424.773572] [] hrtimer_nanosleep+0xbb/0x180 [Wed Feb 6 09:38:44 2019][1696424.779589] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:44 2019][1696424.785530] [] ? do_nanosleep+0x72/0xf0 [Wed Feb 6 09:38:44 2019][1696424.791200] [] SyS_nanosleep+0x96/0xb0 [Wed Feb 6 09:38:44 2019][1696424.796794] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:44 2019][1696424.802978] dsm_sa_datamgrd S ffff9149f8659040 0 18951 1 0x00000080 [Wed Feb 6 09:38:44 2019][1696424.810269] Call Trace: [Wed Feb 6 09:38:44 2019][1696424.812900] [] ? hrtimer_try_to_cancel.part.26+0x58/0x100 [Wed Feb 6 09:38:44 2019][1696424.820129] [] schedule+0x29/0x70 [Wed Feb 6 09:38:44 2019][1696424.825293] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:44 2019][1696424.832524] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:38:44 2019][1696424.838381] [] ? __pollwait+0x7f/0xf0 [Wed Feb 6 09:38:44 2019][1696424.843879] [] ? list_del+0xd/0x30 [Wed Feb 6 09:38:44 2019][1696424.849124] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:44 2019][1696424.855654] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:44 2019][1696424.861946] [] do_select+0x6d1/0x7c0 [Wed Feb 6 09:38:45 2019][1696424.867359] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:45 2019][1696424.872851] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:38:45 2019][1696424.879315] [] ? pollwake+0x73/0x90 [Wed Feb 6 09:38:45 2019][1696424.884639] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:45 2019][1696424.890408] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:38:45 2019][1696424.896425] [] ? __wake_up_sync_key+0x4f/0x60 [Wed Feb 6 09:38:45 2019][1696424.902628] [] ? sock_def_readable+0x3a/0x70 [Wed Feb 6 09:38:45 2019][1696424.908729] [] ? unix_stream_sendmsg+0x3fd/0x430 [Wed Feb 6 09:38:45 2019][1696424.915194] [] ? sock_sendmsg+0xb6/0xf0 [Wed Feb 6 09:38:45 2019][1696424.920867] [] core_sys_select+0x1ff/0x340 [Wed Feb 6 09:38:45 2019][1696424.926795] [] ? SYSC_sendto+0x17e/0x1c0 [Wed Feb 6 09:38:45 2019][1696424.932563] [] SyS_select+0xca/0x120 [Wed Feb 6 09:38:45 2019][1696424.937970] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:45 2019][1696424.944171] kworker/23:1H S ffff9159f6c930c0 0 18429 2 0x00000080 [Wed Feb 6 09:38:45 2019][1696424.951448] Call Trace: [Wed Feb 6 09:38:45 2019][1696424.954084] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:45 2019][1696424.960274] [] schedule+0x29/0x70 [Wed Feb 6 09:38:45 2019][1696424.965436] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:45 2019][1696424.971193] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:45 2019][1696424.977904] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:45 2019][1696424.982976] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696424.989246] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:45 2019][1696424.995882] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.002154] kworker/19:1H S ffff9159f6c95140 0 18431 2 0x00000080 [Wed Feb 6 09:38:45 2019][1696425.009447] Call Trace: [Wed Feb 6 09:38:45 2019][1696425.012098] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:45 2019][1696425.018292] [] schedule+0x29/0x70 [Wed Feb 6 09:38:45 2019][1696425.023451] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:45 2019][1696425.029198] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:45 2019][1696425.035922] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:45 2019][1696425.040980] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.047253] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:45 2019][1696425.053891] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.060169] kdmflush S ffff9159f6c91040 0 18445 2 0x00000080 [Wed Feb 6 09:38:45 2019][1696425.067455] Call Trace: [Wed Feb 6 09:38:45 2019][1696425.070105] [] schedule+0x29/0x70 [Wed Feb 6 09:38:45 2019][1696425.075245] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:45 2019][1696425.081102] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:45 2019][1696425.087283] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:45 2019][1696425.092344] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.098638] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:45 2019][1696425.105262] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.111537] bioset S ffff9159f6c96180 0 18447 2 0x00000080 [Wed Feb 6 09:38:45 2019][1696425.118832] Call Trace: [Wed Feb 6 09:38:45 2019][1696425.121469] [] schedule+0x29/0x70 [Wed Feb 6 09:38:45 2019][1696425.126613] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:45 2019][1696425.132464] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:45 2019][1696425.138655] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:45 2019][1696425.143718] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.150004] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:45 2019][1696425.156623] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.162909] kdmwork-253:3 S ffff9159f6c94100 0 18449 2 0x00000080 [Wed Feb 6 09:38:45 2019][1696425.170187] Call Trace: [Wed Feb 6 09:38:45 2019][1696425.172834] [] schedule+0x29/0x70 [Wed Feb 6 09:38:45 2019][1696425.177973] [] kthread_worker_fn+0xf7/0x180 [Wed Feb 6 09:38:45 2019][1696425.183994] [] ? kthread_stop+0xf0/0xf0 [Wed Feb 6 09:38:45 2019][1696425.189657] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:45 2019][1696425.194722] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.201010] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:45 2019][1696425.207628] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.213912] kdmflush S ffff9159f6c90000 0 18501 2 0x00000080 [Wed Feb 6 09:38:45 2019][1696425.221186] Call Trace: [Wed Feb 6 09:38:45 2019][1696425.223826] [] schedule+0x29/0x70 [Wed Feb 6 09:38:45 2019][1696425.228972] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:45 2019][1696425.234825] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:45 2019][1696425.241011] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:45 2019][1696425.246074] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.252361] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:45 2019][1696425.258975] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.265267] bioset S ffff9129ea33c100 0 18504 2 0x00000080 [Wed Feb 6 09:38:45 2019][1696425.272555] Call Trace: [Wed Feb 6 09:38:45 2019][1696425.275181] [] schedule+0x29/0x70 [Wed Feb 6 09:38:45 2019][1696425.280322] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:45 2019][1696425.286182] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:45 2019][1696425.292371] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:45 2019][1696425.297434] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.303719] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:45 2019][1696425.310335] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.316622] kdmwork-253:4 S ffff9139fe8e8000 0 18505 2 0x00000080 [Wed Feb 6 09:38:45 2019][1696425.323903] Call Trace: [Wed Feb 6 09:38:45 2019][1696425.326533] [] schedule+0x29/0x70 [Wed Feb 6 09:38:45 2019][1696425.331687] [] kthread_worker_fn+0xf7/0x180 [Wed Feb 6 09:38:45 2019][1696425.337712] [] ? kthread_stop+0xf0/0xf0 [Wed Feb 6 09:38:45 2019][1696425.343378] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:45 2019][1696425.348452] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.354731] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:45 2019][1696425.361366] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.367644] kworker/10:1H S ffff9149d62d1040 0 18727 2 0x00000080 [Wed Feb 6 09:38:45 2019][1696425.374939] Call Trace: [Wed Feb 6 09:38:45 2019][1696425.377571] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:45 2019][1696425.383751] [] schedule+0x29/0x70 [Wed Feb 6 09:38:45 2019][1696425.388905] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:45 2019][1696425.394656] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:45 2019][1696425.401363] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:45 2019][1696425.406439] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.412713] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:45 2019][1696425.419334] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.425624] kworker/37:1H S ffff9139f615e180 0 18736 2 0x00000080 [Wed Feb 6 09:38:45 2019][1696425.432921] Call Trace: [Wed Feb 6 09:38:45 2019][1696425.435558] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:45 2019][1696425.441760] [] schedule+0x29/0x70 [Wed Feb 6 09:38:45 2019][1696425.446907] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:45 2019][1696425.452653] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:45 2019][1696425.459379] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:45 2019][1696425.464443] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.470720] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:45 2019][1696425.477356] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.483633] kworker/24:1H S ffff9129ee3c8000 0 18903 2 0x00000080 [Wed Feb 6 09:38:45 2019][1696425.490928] Call Trace: [Wed Feb 6 09:38:45 2019][1696425.493561] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:45 2019][1696425.499748] [] schedule+0x29/0x70 [Wed Feb 6 09:38:45 2019][1696425.504902] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:45 2019][1696425.510651] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:45 2019][1696425.517356] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:45 2019][1696425.522431] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.528710] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:45 2019][1696425.535342] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.541612] kworker/1:1H S ffff9139f6159040 0 18991 2 0x00000080 [Wed Feb 6 09:38:45 2019][1696425.548906] Call Trace: [Wed Feb 6 09:38:45 2019][1696425.551534] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:45 2019][1696425.557718] [] schedule+0x29/0x70 [Wed Feb 6 09:38:45 2019][1696425.562880] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:45 2019][1696425.568638] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:45 2019][1696425.575359] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:45 2019][1696425.580424] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.586711] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:45 2019][1696425.593329] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:45 2019][1696425.599622] sshd S ffff9139f7009040 0 19077 1 0x00000080 [Wed Feb 6 09:38:45 2019][1696425.606904] Call Trace: [Wed Feb 6 09:38:45 2019][1696425.609537] [] schedule+0x29/0x70 [Wed Feb 6 09:38:45 2019][1696425.614694] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:45 2019][1696425.621921] [] ? dequeue_entity+0x11c/0x5e0 [Wed Feb 6 09:38:45 2019][1696425.627937] [] ? add_wait_queue+0x3c/0x50 [Wed Feb 6 09:38:45 2019][1696425.633790] [] ? __pollwait+0x7f/0xf0 [Wed Feb 6 09:38:45 2019][1696425.639281] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:45 2019][1696425.645826] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:45 2019][1696425.652099] [] do_select+0x6d1/0x7c0 [Wed Feb 6 09:38:45 2019][1696425.657538] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:45 2019][1696425.663025] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:45 2019][1696425.668525] [] ? __alloc_skb+0xa1/0x2d0 [Wed Feb 6 09:38:45 2019][1696425.674206] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Wed Feb 6 09:38:45 2019][1696425.681178] [] ? cpumask_any_but+0x39/0x50 [Wed Feb 6 09:38:45 2019][1696425.687120] [] ? flush_tlb_page+0x38/0xa0 [Wed Feb 6 09:38:45 2019][1696425.692964] [] ? lru_cache_add+0xe/0x10 [Wed Feb 6 09:38:45 2019][1696425.698643] [] ? page_add_new_anon_rmap+0xb8/0x170 [Wed Feb 6 09:38:45 2019][1696425.705259] [] ? wp_page_copy.isra.71+0x3ab/0x560 [Wed Feb 6 09:38:45 2019][1696425.711809] [] ? unlock_page+0x2b/0x30 [Wed Feb 6 09:38:45 2019][1696425.717388] [] ? do_wp_page+0x19e/0x720 [Wed Feb 6 09:38:45 2019][1696425.723057] [] ? __check_object_size+0x1ca/0x250 [Wed Feb 6 09:38:45 2019][1696425.729516] [] core_sys_select+0x1ff/0x340 [Wed Feb 6 09:38:45 2019][1696425.735443] [] ? handle_mm_fault+0x39d/0x9b0 [Wed Feb 6 09:38:45 2019][1696425.741554] [] SyS_select+0xca/0x120 [Wed Feb 6 09:38:45 2019][1696425.746956] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:45 2019][1696425.753156] rsyslogd S ffff9139f7008000 0 19079 1 0x00000080 [Wed Feb 6 09:38:45 2019][1696425.760449] Call Trace: [Wed Feb 6 09:38:45 2019][1696425.763081] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:45 2019][1696425.769785] [] schedule+0x29/0x70 [Wed Feb 6 09:38:45 2019][1696425.774951] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:38:45 2019][1696425.782092] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:45 2019][1696425.788021] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:38:45 2019][1696425.795351] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:45 2019][1696425.801890] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:45 2019][1696425.808165] [] do_select+0x6d1/0x7c0 [Wed Feb 6 09:38:45 2019][1696425.813584] [] ? _raw_read_unlock_bh+0x1b/0x20 [Wed Feb 6 09:38:45 2019][1696425.819852] [] ? __skb_try_recv_datagram+0x172/0x300 [Wed Feb 6 09:38:45 2019][1696425.826665] [] ? task_scan_min+0x3e/0x60 [Wed Feb 6 09:38:45 2019][1696425.832423] [] ? task_scan_max+0x27/0x40 [Wed Feb 6 09:38:45 2019][1696425.838179] [] ? task_numa_fault+0x8d0/0xbb0 [Wed Feb 6 09:38:45 2019][1696425.844294] [] ? do_numa_page+0x1be/0x250 [Wed Feb 6 09:38:45 2019][1696425.850138] [] ? handle_pte_fault+0x316/0xd10 [Wed Feb 6 09:38:45 2019][1696425.856339] [] ? __brelse+0x3d/0x50 [Wed Feb 6 09:38:45 2019][1696425.861653] [] ? bh_lru_install+0x18a/0x1e0 [Wed Feb 6 09:38:45 2019][1696425.867685] [] ? should_numa_migrate_memory+0x5b/0x150 [Wed Feb 6 09:38:46 2019][1696425.874652] [] ? mpol_misplaced+0x18c/0x1e0 [Wed Feb 6 09:38:46 2019][1696425.880670] [] core_sys_select+0x1ff/0x340 [Wed Feb 6 09:38:46 2019][1696425.886619] [] ? handle_mm_fault+0x39d/0x9b0 [Wed Feb 6 09:38:46 2019][1696425.892719] [] ? __check_object_size+0x1ca/0x250 [Wed Feb 6 09:38:46 2019][1696425.899168] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:38:46 2019][1696425.905017] [] SyS_select+0xca/0x120 [Wed Feb 6 09:38:46 2019][1696425.910425] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:46 2019][1696425.916625] in:imjournal S ffff9149fbc06180 0 19089 1 0x00000080 [Wed Feb 6 09:38:46 2019][1696425.923903] Call Trace: [Wed Feb 6 09:38:46 2019][1696425.926535] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:46 2019][1696425.933258] [] schedule+0x29/0x70 [Wed Feb 6 09:38:46 2019][1696425.938410] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:38:46 2019][1696425.945551] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:46 2019][1696425.951491] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:38:46 2019][1696425.958801] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:46 2019][1696425.965349] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:38:46 2019][1696425.971625] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:38:46 2019][1696425.977219] [] ? jbd2_journal_dirty_metadata+0xd1/0x260 [jbd2] [Wed Feb 6 09:38:46 2019][1696425.984898] [] ? jbd2_journal_release_jbd_inode+0xf1/0x130 [jbd2] [Wed Feb 6 09:38:46 2019][1696425.992819] [] ? security_inode_permission+0x22/0x30 [Wed Feb 6 09:38:46 2019][1696425.999633] [] ? __enqueue_entity+0x78/0x80 [Wed Feb 6 09:38:46 2019][1696426.005645] [] ? link_path_walk+0x81/0x8b0 [Wed Feb 6 09:38:46 2019][1696426.011577] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:38:46 2019][1696426.017083] [] ? complete_walk+0x50/0xe0 [Wed Feb 6 09:38:46 2019][1696426.022834] [] ? path_lookupat+0x92/0x8b0 [Wed Feb 6 09:38:46 2019][1696426.028691] [] ? ttwu_do_activate+0x6f/0x80 [Wed Feb 6 09:38:46 2019][1696426.034707] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:38:46 2019][1696426.040721] [] ? kmem_cache_alloc+0x35/0x1f0 [Wed Feb 6 09:38:46 2019][1696426.046834] [] ? wake_up_q+0x5b/0x80 [Wed Feb 6 09:38:46 2019][1696426.052237] [] ? futex_wake_op+0x4a5/0x610 [Wed Feb 6 09:38:46 2019][1696426.058165] [] ? putname+0x3d/0x60 [Wed Feb 6 09:38:46 2019][1696426.063420] [] ? do_futex+0x17a/0x5a0 [Wed Feb 6 09:38:46 2019][1696426.068912] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:38:46 2019][1696426.074754] [] SyS_ppoll+0x1d3/0x1f0 [Wed Feb 6 09:38:46 2019][1696426.080179] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:46 2019][1696426.086361] rs:main Q:Reg S ffff9139f763a080 0 19094 1 0x00000080 [Wed Feb 6 09:38:46 2019][1696426.093644] Call Trace: [Wed Feb 6 09:38:46 2019][1696426.096287] [] schedule+0x29/0x70 [Wed Feb 6 09:38:46 2019][1696426.101431] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:46 2019][1696426.107643] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:46 2019][1696426.113132] [] ? generic_file_aio_write+0x77/0xa0 [Wed Feb 6 09:38:46 2019][1696426.119676] [] ? ext4_file_write+0xd2/0x1e0 [ext4] [Wed Feb 6 09:38:46 2019][1696426.126316] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:46 2019][1696426.131641] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:46 2019][1696426.136968] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:46 2019][1696426.143171] collectd S ffff9139f700d140 0 19081 1 0x00000080 [Wed Feb 6 09:38:46 2019][1696426.150457] Call Trace: [Wed Feb 6 09:38:46 2019][1696426.153103] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:46 2019][1696426.159810] [] schedule+0x29/0x70 [Wed Feb 6 09:38:46 2019][1696426.164964] [] do_nanosleep+0x86/0xf0 [Wed Feb 6 09:38:46 2019][1696426.170454] [] hrtimer_nanosleep+0xbb/0x180 [Wed Feb 6 09:38:46 2019][1696426.176470] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:46 2019][1696426.182412] [] ? do_nanosleep+0x72/0xf0 [Wed Feb 6 09:38:46 2019][1696426.188077] [] SyS_nanosleep+0x96/0xb0 [Wed Feb 6 09:38:46 2019][1696426.193659] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:46 2019][1696426.199860] writer#0 S ffff9149fbc01040 0 19092 1 0x00000080 [Wed Feb 6 09:38:46 2019][1696426.207129] Call Trace: [Wed Feb 6 09:38:46 2019][1696426.209774] [] schedule+0x29/0x70 [Wed Feb 6 09:38:46 2019][1696426.214915] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:46 2019][1696426.221121] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:46 2019][1696426.226616] [] ? ttwu_do_wakeup+0xb5/0xe0 [Wed Feb 6 09:38:46 2019][1696426.232459] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:38:46 2019][1696426.238488] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:46 2019][1696426.243807] [] ? mntput+0x24/0x40 [Wed Feb 6 09:38:46 2019][1696426.248964] [] ? __fput+0x186/0x260 [Wed Feb 6 09:38:46 2019][1696426.254280] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:46 2019][1696426.259624] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:46 2019][1696426.265809] writer#1 S ffff9149fbc02080 0 19093 1 0x00000080 [Wed Feb 6 09:38:46 2019][1696426.273100] Call Trace: [Wed Feb 6 09:38:46 2019][1696426.275733] [] schedule+0x29/0x70 [Wed Feb 6 09:38:46 2019][1696426.280882] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:46 2019][1696426.287083] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:46 2019][1696426.292574] [] ? fsnotify_grab_connector+0x43/0x70 [Wed Feb 6 09:38:46 2019][1696426.299192] [] ? fsnotify_destroy_marks+0x13/0xc0 [Wed Feb 6 09:38:46 2019][1696426.305739] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:46 2019][1696426.311058] [] ? mntput+0x24/0x40 [Wed Feb 6 09:38:46 2019][1696426.316210] [] ? __fput+0x186/0x260 [Wed Feb 6 09:38:46 2019][1696426.321542] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:46 2019][1696426.326863] [] ? task_work_run+0xc0/0xe0 [Wed Feb 6 09:38:46 2019][1696426.332634] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:46 2019][1696426.338815] writer#2 S ffff9159f6b94100 0 19095 1 0x00000080 [Wed Feb 6 09:38:46 2019][1696426.346102] Call Trace: [Wed Feb 6 09:38:46 2019][1696426.348730] [] schedule+0x29/0x70 [Wed Feb 6 09:38:46 2019][1696426.353876] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:46 2019][1696426.360072] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:46 2019][1696426.365564] [] ? ttwu_do_wakeup+0xb5/0xe0 [Wed Feb 6 09:38:46 2019][1696426.371421] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:38:46 2019][1696426.377441] [] ? get_futex_key+0x1c8/0x2c0 [Wed Feb 6 09:38:46 2019][1696426.383369] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:46 2019][1696426.388703] [] ? mntput+0x24/0x40 [Wed Feb 6 09:38:46 2019][1696426.393847] [] ? __fput+0x186/0x260 [Wed Feb 6 09:38:46 2019][1696426.399181] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:46 2019][1696426.404495] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:46 2019][1696426.410695] writer#3 S ffff9159f6b930c0 0 19096 1 0x00000080 [Wed Feb 6 09:38:46 2019][1696426.417981] Call Trace: [Wed Feb 6 09:38:46 2019][1696426.420615] [] schedule+0x29/0x70 [Wed Feb 6 09:38:46 2019][1696426.425764] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:46 2019][1696426.431965] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:46 2019][1696426.437454] [] ? ttwu_do_wakeup+0xb5/0xe0 [Wed Feb 6 09:38:46 2019][1696426.443296] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:38:46 2019][1696426.449325] [] ? get_futex_key+0x1c8/0x2c0 [Wed Feb 6 09:38:46 2019][1696426.455252] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:46 2019][1696426.460575] [] ? mntput+0x24/0x40 [Wed Feb 6 09:38:46 2019][1696426.465734] [] ? __fput+0x186/0x260 [Wed Feb 6 09:38:46 2019][1696426.471052] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:46 2019][1696426.476381] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:46 2019][1696426.482569] writer#4 S ffff9159f6b95140 0 19097 1 0x00000080 [Wed Feb 6 09:38:46 2019][1696426.489861] Call Trace: [Wed Feb 6 09:38:46 2019][1696426.492492] [] schedule+0x29/0x70 [Wed Feb 6 09:38:46 2019][1696426.497640] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:46 2019][1696426.503843] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:46 2019][1696426.509339] [] ? ttwu_do_wakeup+0xb5/0xe0 [Wed Feb 6 09:38:46 2019][1696426.515193] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:38:46 2019][1696426.521204] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:46 2019][1696426.526525] [] ? mntput+0x24/0x40 [Wed Feb 6 09:38:46 2019][1696426.531691] [] ? __fput+0x186/0x260 [Wed Feb 6 09:38:46 2019][1696426.537013] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:46 2019][1696426.542337] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:46 2019][1696426.548538] reader#0 S ffff9159f6b96180 0 19098 1 0x00000080 [Wed Feb 6 09:38:46 2019][1696426.555816] Call Trace: [Wed Feb 6 09:38:46 2019][1696426.558445] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:46 2019][1696426.565170] [] schedule+0x29/0x70 [Wed Feb 6 09:38:46 2019][1696426.570317] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:46 2019][1696426.576501] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:46 2019][1696426.582011] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:46 2019][1696426.587940] [] ? futex_wait_queue_me+0xa4/0x130 [Wed Feb 6 09:38:46 2019][1696426.594317] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:46 2019][1696426.599641] [] ? do_munmap+0x327/0x480 [Wed Feb 6 09:38:46 2019][1696426.605232] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:46 2019][1696426.610554] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:46 2019][1696426.616751] reader#1 S ffff9149f62b4100 0 19099 1 0x00000080 [Wed Feb 6 09:38:46 2019][1696426.624022] Call Trace: [Wed Feb 6 09:38:46 2019][1696426.626669] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:46 2019][1696426.633372] [] schedule+0x29/0x70 [Wed Feb 6 09:38:46 2019][1696426.638534] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:46 2019][1696426.644717] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:46 2019][1696426.650223] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:46 2019][1696426.656148] [] ? futex_wait_queue_me+0xa4/0x130 [Wed Feb 6 09:38:46 2019][1696426.662539] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:46 2019][1696426.667854] [] ? mntput+0x24/0x40 [Wed Feb 6 09:38:46 2019][1696426.673011] [] ? __fput+0x186/0x260 [Wed Feb 6 09:38:46 2019][1696426.678341] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:46 2019][1696426.683660] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:46 2019][1696426.689856] reader#2 S ffff9149d5f6b0c0 0 19100 1 0x00000080 [Wed Feb 6 09:38:46 2019][1696426.697128] Call Trace: [Wed Feb 6 09:38:46 2019][1696426.699771] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:46 2019][1696426.706475] [] schedule+0x29/0x70 [Wed Feb 6 09:38:46 2019][1696426.711644] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:46 2019][1696426.717834] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:46 2019][1696426.723326] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:46 2019][1696426.729269] [] ? futex_wait_queue_me+0xa4/0x130 [Wed Feb 6 09:38:46 2019][1696426.735639] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:46 2019][1696426.740974] [] ? do_munmap+0x327/0x480 [Wed Feb 6 09:38:46 2019][1696426.746555] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:46 2019][1696426.751886] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:46 2019][1696426.758073] reader#3 S ffff9149d5f6d140 0 19101 1 0x00000080 [Wed Feb 6 09:38:46 2019][1696426.765375] Call Trace: [Wed Feb 6 09:38:46 2019][1696426.768014] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:46 2019][1696426.774720] [] schedule+0x29/0x70 [Wed Feb 6 09:38:46 2019][1696426.779882] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:46 2019][1696426.786067] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:46 2019][1696426.791562] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:46 2019][1696426.797502] [] ? futex_wait_queue_me+0xa4/0x130 [Wed Feb 6 09:38:46 2019][1696426.803860] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:46 2019][1696426.809197] [] ? do_munmap+0x327/0x480 [Wed Feb 6 09:38:46 2019][1696426.814780] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:46 2019][1696426.820116] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:46 2019][1696426.826306] reader#4 S ffff9149d5f68000 0 19102 1 0x00000080 [Wed Feb 6 09:38:46 2019][1696426.833605] Call Trace: [Wed Feb 6 09:38:46 2019][1696426.836238] [] schedule+0x29/0x70 [Wed Feb 6 09:38:46 2019][1696426.841389] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:46 2019][1696426.847590] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:46 2019][1696426.853088] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:46 2019][1696426.858418] [] ? __free_pages+0x1d/0x30 [Wed Feb 6 09:38:46 2019][1696426.864085] [] ? free_pages.part.82+0x3f/0x50 [Wed Feb 6 09:38:46 2019][1696426.870286] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:47 2019][1696426.875604] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:47 2019][1696426.881805] nfsiod S ffff9139f615d140 0 19163 2 0x00000080 [Wed Feb 6 09:38:47 2019][1696426.889097] Call Trace: [Wed Feb 6 09:38:47 2019][1696426.891743] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696426.896887] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:47 2019][1696426.902740] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:47 2019][1696426.908931] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:47 2019][1696426.913986] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:47 2019][1696426.920275] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:47 2019][1696426.926894] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:47 2019][1696426.933170] kworker/18:1H S ffff9149f8e95140 0 19228 2 0x00000080 [Wed Feb 6 09:38:47 2019][1696426.940472] Call Trace: [Wed Feb 6 09:38:47 2019][1696426.943107] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:47 2019][1696426.949295] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696426.954457] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:47 2019][1696426.960207] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:47 2019][1696426.966915] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:47 2019][1696426.971994] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:47 2019][1696426.978273] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:47 2019][1696426.984895] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:47 2019][1696426.991180] nfsv4.1-svc S ffff9149f3d96180 0 19229 2 0x00000080 [Wed Feb 6 09:38:47 2019][1696426.998458] Call Trace: [Wed Feb 6 09:38:47 2019][1696427.001107] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696427.006263] [] nfs41_callback_svc+0x15d/0x1c0 [nfsv4] [Wed Feb 6 09:38:47 2019][1696427.013163] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:47 2019][1696427.019185] [] ? nfs4_callback_svc+0x60/0x60 [nfsv4] [Wed Feb 6 09:38:47 2019][1696427.025983] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:47 2019][1696427.031060] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:47 2019][1696427.037340] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:47 2019][1696427.043968] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:47 2019][1696427.050256] crond S ffff9139ef7d9040 0 19272 1 0x00000080 [Wed Feb 6 09:38:47 2019][1696427.057544] Call Trace: [Wed Feb 6 09:38:47 2019][1696427.060185] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:47 2019][1696427.066889] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696427.072052] [] do_nanosleep+0x86/0xf0 [Wed Feb 6 09:38:47 2019][1696427.077547] [] hrtimer_nanosleep+0xbb/0x180 [Wed Feb 6 09:38:47 2019][1696427.083580] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:47 2019][1696427.089510] [] ? do_nanosleep+0x72/0xf0 [Wed Feb 6 09:38:47 2019][1696427.095181] [] SyS_nanosleep+0x96/0xb0 [Wed Feb 6 09:38:47 2019][1696427.100776] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:47 2019][1696427.106967] agetty S ffff9139ef7dd140 0 19332 1 0x00000080 [Wed Feb 6 09:38:47 2019][1696427.114265] Call Trace: [Wed Feb 6 09:38:47 2019][1696427.116897] [] ? bit_cursor+0x663/0x6a0 [Wed Feb 6 09:38:47 2019][1696427.122563] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696427.127717] [] schedule_timeout+0x221/0x2d0 [Wed Feb 6 09:38:47 2019][1696427.133730] [] ? get_work_pool+0x3e/0x70 [Wed Feb 6 09:38:47 2019][1696427.139484] [] ? flush_work+0x13b/0x190 [Wed Feb 6 09:38:47 2019][1696427.145168] [] ? set_room+0x68/0xc0 [Wed Feb 6 09:38:47 2019][1696427.150489] [] n_tty_read+0x27c/0x930 [Wed Feb 6 09:38:47 2019][1696427.156002] [] ? ldsem_down_read+0x3b/0x220 [Wed Feb 6 09:38:47 2019][1696427.162014] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:47 2019][1696427.167766] [] tty_read+0x90/0x100 [Wed Feb 6 09:38:47 2019][1696427.173012] [] vfs_read+0x9f/0x170 [Wed Feb 6 09:38:47 2019][1696427.178244] [] SyS_read+0x7f/0xf0 [Wed Feb 6 09:38:47 2019][1696427.183406] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:47 2019][1696427.189597] agetty S ffff9139f700c100 0 19334 1 0x00000080 [Wed Feb 6 09:38:47 2019][1696427.196896] Call Trace: [Wed Feb 6 09:38:47 2019][1696427.199525] [] ? __do_fault.isra.59+0x8a/0x100 [Wed Feb 6 09:38:47 2019][1696427.205804] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696427.210961] [] schedule_timeout+0x221/0x2d0 [Wed Feb 6 09:38:47 2019][1696427.216974] [] ? get_work_pool+0x3e/0x70 [Wed Feb 6 09:38:47 2019][1696427.222740] [] ? flush_work+0x13b/0x190 [Wed Feb 6 09:38:47 2019][1696427.228407] [] ? serial8250_start_tx+0xa5/0xe0 [Wed Feb 6 09:38:47 2019][1696427.234682] [] ? set_room+0x68/0xc0 [Wed Feb 6 09:38:47 2019][1696427.240014] [] n_tty_read+0x27c/0x930 [Wed Feb 6 09:38:47 2019][1696427.245507] [] ? ldsem_down_read+0x3b/0x220 [Wed Feb 6 09:38:47 2019][1696427.251532] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:47 2019][1696427.257283] [] tty_read+0x90/0x100 [Wed Feb 6 09:38:47 2019][1696427.262535] [] vfs_read+0x9f/0x170 [Wed Feb 6 09:38:47 2019][1696427.267770] [] SyS_read+0x7f/0xf0 [Wed Feb 6 09:38:47 2019][1696427.272921] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:47 2019][1696427.279123] kworker/44:1H S ffff9129f3759040 0 19463 2 0x00000080 [Wed Feb 6 09:38:47 2019][1696427.286421] Call Trace: [Wed Feb 6 09:38:47 2019][1696427.289073] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:47 2019][1696427.295259] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696427.300424] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:47 2019][1696427.306175] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:47 2019][1696427.312880] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:47 2019][1696427.317959] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:47 2019][1696427.324237] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:47 2019][1696427.330858] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:47 2019][1696427.337132] master S ffff9139f1ff0000 0 19477 1 0x00000080 [Wed Feb 6 09:38:47 2019][1696427.344414] Call Trace: [Wed Feb 6 09:38:47 2019][1696427.347057] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:47 2019][1696427.353762] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696427.358928] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:38:47 2019][1696427.366072] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:47 2019][1696427.372000] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:38:47 2019][1696427.379329] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:47 2019][1696427.385863] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:38:47 2019][1696427.391112] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:47 2019][1696427.396862] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:38:47 2019][1696427.402617] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:47 2019][1696427.408822] slapd S ffff9149d5fc8000 0 19480 1 0x00000080 [Wed Feb 6 09:38:47 2019][1696427.416105] Call Trace: [Wed Feb 6 09:38:47 2019][1696427.418735] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696427.423877] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:47 2019][1696427.430082] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:47 2019][1696427.435581] [] ? sched_clock+0x9/0x10 [Wed Feb 6 09:38:47 2019][1696427.441072] [] ? sched_slice.isra.58+0x66/0xc0 [Wed Feb 6 09:38:47 2019][1696427.447366] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:47 2019][1696427.452689] [] ? resched_curr+0xae/0xc0 [Wed Feb 6 09:38:47 2019][1696427.458357] [] ? check_preempt_curr+0x80/0xa0 [Wed Feb 6 09:38:47 2019][1696427.464559] [] ? wake_up_new_task+0x131/0x1a0 [Wed Feb 6 09:38:47 2019][1696427.470751] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:47 2019][1696427.476080] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:47 2019][1696427.482264] slapd S ffff9139f1ff1040 0 19485 1 0x00000080 [Wed Feb 6 09:38:47 2019][1696427.489542] Call Trace: [Wed Feb 6 09:38:47 2019][1696427.492178] [] ? pipe_read+0x2e5/0x400 [Wed Feb 6 09:38:47 2019][1696427.497765] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696427.502925] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:38:47 2019][1696427.510150] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Wed Feb 6 09:38:47 2019][1696427.517130] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:47 2019][1696427.523665] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:38:47 2019][1696427.528910] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:47 2019][1696427.534665] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:38:47 2019][1696427.540438] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:47 2019][1696427.546629] slapd S ffff9139f8b4d140 0 19486 1 0x00000080 [Wed Feb 6 09:38:47 2019][1696427.553914] Call Trace: [Wed Feb 6 09:38:47 2019][1696427.556551] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696427.561714] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:47 2019][1696427.567904] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:47 2019][1696427.573398] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:47 2019][1696427.578733] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:47 2019][1696427.584051] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:47 2019][1696427.590242] slapd S ffff9139f83ae180 0 84636 1 0x00000080 [Wed Feb 6 09:38:47 2019][1696427.597540] Call Trace: [Wed Feb 6 09:38:47 2019][1696427.600169] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696427.605318] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:47 2019][1696427.611525] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:47 2019][1696427.617019] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:47 2019][1696427.622339] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:47 2019][1696427.627682] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:47 2019][1696427.633870] slapd S ffff9139fe8e9040 0 29295 1 0x00000080 [Wed Feb 6 09:38:47 2019][1696427.641160] Call Trace: [Wed Feb 6 09:38:47 2019][1696427.643804] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696427.648948] [] futex_wait_queue_me+0xc6/0x130 [Wed Feb 6 09:38:47 2019][1696427.655151] [] futex_wait+0x17b/0x280 [Wed Feb 6 09:38:47 2019][1696427.660638] [] do_futex+0x106/0x5a0 [Wed Feb 6 09:38:47 2019][1696427.665961] [] SyS_futex+0x80/0x190 [Wed Feb 6 09:38:47 2019][1696427.671300] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:47 2019][1696427.677489] qmgr S ffff9139f27b9040 0 19491 19477 0x00000080 [Wed Feb 6 09:38:47 2019][1696427.684771] Call Trace: [Wed Feb 6 09:38:47 2019][1696427.687403] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:38:47 2019][1696427.694124] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696427.699267] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:38:47 2019][1696427.706410] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:38:47 2019][1696427.712350] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:38:47 2019][1696427.719667] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:38:47 2019][1696427.726200] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:38:47 2019][1696427.731447] [] ? mntput+0x24/0x40 [Wed Feb 6 09:38:47 2019][1696427.736598] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:47 2019][1696427.742367] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:38:47 2019][1696427.748126] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:38:47 2019][1696427.754327] kworker/35:2 S ffff9159f7a80000 0 19805 2 0x00000080 [Wed Feb 6 09:38:47 2019][1696427.761599] Call Trace: [Wed Feb 6 09:38:47 2019][1696427.764245] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:47 2019][1696427.770436] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696427.775595] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:47 2019][1696427.781347] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:47 2019][1696427.788064] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:47 2019][1696427.793120] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:47 2019][1696427.799409] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:47 2019][1696427.806030] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:47 2019][1696427.812304] kworker/36:1H S ffff9129f6524100 0 20169 2 0x00000080 [Wed Feb 6 09:38:47 2019][1696427.819610] Call Trace: [Wed Feb 6 09:38:47 2019][1696427.822244] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:47 2019][1696427.828429] [] schedule+0x29/0x70 [Wed Feb 6 09:38:47 2019][1696427.833595] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:47 2019][1696427.839349] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:47 2019][1696427.846058] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:47 2019][1696427.851137] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:47 2019][1696427.857415] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:47 2019][1696427.864037] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:47 2019][1696427.870321] kworker/7:1H S ffff9159f7a830c0 0 20226 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696427.877604] Call Trace: [Wed Feb 6 09:38:48 2019][1696427.880245] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696427.886427] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696427.891595] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696427.897352] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696427.904059] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696427.909130] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696427.915403] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696427.922037] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696427.928308] kworker/12:1H S ffff9129f65230c0 0 20590 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696427.935590] Call Trace: [Wed Feb 6 09:38:48 2019][1696427.938234] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696427.944423] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696427.949585] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696427.955344] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696427.962060] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696427.967117] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696427.973405] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696427.980022] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696427.986300] kworker/33:1H S ffff9139f6d60000 0 27717 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696427.993608] Call Trace: [Wed Feb 6 09:38:48 2019][1696427.996237] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.002421] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696428.007578] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696428.013329] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696428.020062] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696428.025127] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.031402] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696428.038037] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.044308] kworker/47:1H S ffff9159f65c8000 0 27748 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696428.051603] Call Trace: [Wed Feb 6 09:38:48 2019][1696428.054236] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.060423] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696428.065579] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696428.071332] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696428.078040] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696428.083115] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.089392] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696428.096026] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.102296] kworker/34:1H S ffff9149f40630c0 0 27791 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696428.109602] Call Trace: [Wed Feb 6 09:38:48 2019][1696428.112236] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.118427] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696428.123588] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696428.129339] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696428.136061] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696428.141122] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.147401] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696428.154032] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.160304] kworker/20:1H S ffff9129f3c030c0 0 27801 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696428.167610] Call Trace: [Wed Feb 6 09:38:48 2019][1696428.170244] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.176431] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696428.181595] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696428.187346] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696428.194051] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696428.199130] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.205409] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696428.212031] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.218314] kworker/27:1H S ffff9159f65cb0c0 0 27809 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696428.225598] Call Trace: [Wed Feb 6 09:38:48 2019][1696428.228247] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.234430] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696428.239595] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696428.245355] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696428.252062] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696428.257135] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.263403] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696428.270040] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.276311] kworker/6:1H S ffff9149f4061040 0 27841 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696428.283597] Call Trace: [Wed Feb 6 09:38:48 2019][1696428.286249] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.292441] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696428.297601] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696428.303351] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696428.310070] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696428.315131] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.321407] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696428.328040] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.334321] kworker/14:1H S ffff9149f4065140 0 27914 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696428.341626] Call Trace: [Wed Feb 6 09:38:48 2019][1696428.344257] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.350447] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696428.355607] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696428.361356] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696428.368078] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696428.373135] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.379414] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696428.386047] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.392325] kworker/4:1H S ffff9129f3c05140 0 28018 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696428.399631] Call Trace: [Wed Feb 6 09:38:48 2019][1696428.402270] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.408452] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696428.413615] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696428.419364] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696428.426088] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696428.431146] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.437422] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696428.444055] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.450323] kworker/17:1H S ffff9139f700e180 0 29158 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696428.457629] Call Trace: [Wed Feb 6 09:38:48 2019][1696428.460264] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.466470] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696428.471620] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696428.477375] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696428.484096] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696428.489158] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.495448] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696428.502061] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.508328] kworker/39:1H S ffff9159fab0d140 0 29212 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696428.515622] Call Trace: [Wed Feb 6 09:38:48 2019][1696428.518254] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.524443] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696428.529605] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696428.535352] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696428.542075] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696428.547134] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.553405] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696428.560041] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.566312] kworker/9:1H S ffff9139ef7de180 0 29218 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696428.573600] Call Trace: [Wed Feb 6 09:38:48 2019][1696428.576250] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.582432] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696428.587592] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696428.593343] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696428.600055] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696428.605128] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.611398] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696428.618035] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.624311] kworker/28:1H S ffff9129e9aa0000 0 29233 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696428.631600] Call Trace: [Wed Feb 6 09:38:48 2019][1696428.634246] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.640433] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696428.645595] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696428.651353] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696428.658075] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696428.663132] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.669420] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696428.676037] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.682327] kworker/11:1H S ffff9159f6faa080 0 29734 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696428.689618] Call Trace: [Wed Feb 6 09:38:48 2019][1696428.692264] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.698450] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696428.703609] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696428.709361] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696428.716082] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696428.721148] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.727436] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696428.734063] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.740350] kworker/5:1H S ffff9139f8945140 0 30257 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696428.747639] Call Trace: [Wed Feb 6 09:38:48 2019][1696428.750273] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.756453] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696428.761615] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696428.767366] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696428.774073] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696428.779154] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.785430] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696428.792051] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.798340] kworker/41:1H S ffff9139f83ac100 0 30572 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696428.805611] Call Trace: [Wed Feb 6 09:38:48 2019][1696428.808250] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.814439] [] schedule+0x29/0x70 [Wed Feb 6 09:38:48 2019][1696428.819602] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:48 2019][1696428.825360] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:48 2019][1696428.832079] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:48 2019][1696428.837137] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.843425] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:48 2019][1696428.850041] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:48 2019][1696428.856331] kworker/26:1H S ffff9149f6e64100 0 30703 2 0x00000080 [Wed Feb 6 09:38:48 2019][1696428.863612] Call Trace: [Wed Feb 6 09:38:48 2019][1696428.866260] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:48 2019][1696428.872445] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696428.877609] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:49 2019][1696428.883367] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:49 2019][1696428.890073] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696428.895148] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696428.901420] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696428.908051] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696428.914323] kworker/42:1H S ffff9149f6e65140 0 30717 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696428.921617] Call Trace: [Wed Feb 6 09:38:49 2019][1696428.924247] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:49 2019][1696428.930435] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696428.935601] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:49 2019][1696428.941356] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:49 2019][1696428.948076] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696428.953130] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696428.959411] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696428.966033] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696428.972305] kworker/43:1H S ffff9159eb756180 0 49676 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696428.979609] Call Trace: [Wed Feb 6 09:38:49 2019][1696428.982244] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:49 2019][1696428.988432] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696428.993596] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:49 2019][1696428.999343] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:49 2019][1696429.006052] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696429.011131] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.017408] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696429.024030] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.030315] kworker/30:1H S ffff9149ca746180 0 52904 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696429.037590] Call Trace: [Wed Feb 6 09:38:49 2019][1696429.040239] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:49 2019][1696429.046421] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696429.051574] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:49 2019][1696429.057329] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:49 2019][1696429.064038] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696429.069111] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.075390] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696429.082024] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.088303] kworker/13:1H S ffff9139ef7d8000 0 54683 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696429.095597] Call Trace: [Wed Feb 6 09:38:49 2019][1696429.098231] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:49 2019][1696429.104417] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696429.109578] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:49 2019][1696429.115335] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:49 2019][1696429.122041] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696429.127115] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.133386] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696429.140006] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.146298] kworker/29:1H S ffff9139ef7dc100 0 54700 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696429.153583] Call Trace: [Wed Feb 6 09:38:49 2019][1696429.156217] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:49 2019][1696429.162402] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696429.167563] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:49 2019][1696429.173314] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:49 2019][1696429.180025] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696429.185101] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.191378] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696429.198014] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.204288] kworker/45:1H S ffff9139ef7da080 0 54720 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696429.211576] Call Trace: [Wed Feb 6 09:38:49 2019][1696429.214203] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:49 2019][1696429.220392] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696429.225556] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:49 2019][1696429.231308] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:49 2019][1696429.238015] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696429.243091] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.249366] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696429.256005] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.262278] kworker/46:1H S ffff9149f865e180 0 57542 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696429.269551] Call Trace: [Wed Feb 6 09:38:49 2019][1696429.272201] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:49 2019][1696429.278386] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696429.283547] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:49 2019][1696429.289306] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:49 2019][1696429.296022] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696429.301080] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.307370] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696429.313988] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.320277] kworker/16:1 S ffff9129fb854100 0 59050 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696429.327562] Call Trace: [Wed Feb 6 09:38:49 2019][1696429.330194] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:49 2019][1696429.336384] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696429.341547] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:49 2019][1696429.347304] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:49 2019][1696429.354028] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696429.359088] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.365378] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696429.371994] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.378271] kworker/34:2 S ffff9149ca740000 0 95825 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696429.385575] Call Trace: [Wed Feb 6 09:38:49 2019][1696429.388207] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:49 2019][1696429.394387] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696429.399551] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:49 2019][1696429.405304] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:49 2019][1696429.412015] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696429.417087] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.423384] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696429.430018] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.436296] kworker/9:0 S ffff9139f8b49040 0 126392 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696429.443686] Call Trace: [Wed Feb 6 09:38:49 2019][1696429.446323] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:49 2019][1696429.452513] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696429.457672] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:49 2019][1696429.463425] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:49 2019][1696429.470144] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696429.475198] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.481488] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696429.488102] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.494368] kworker/18:1 S ffff9149ba352080 0 129420 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696429.501747] Call Trace: [Wed Feb 6 09:38:49 2019][1696429.504382] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:38:49 2019][1696429.510573] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696429.515733] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:38:49 2019][1696429.521493] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:38:49 2019][1696429.528212] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696429.533265] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.539546] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696429.546164] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.552442] cfs_rh_00 S ffff913aa98430c0 0 5813 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696429.559740] Call Trace: [Wed Feb 6 09:38:49 2019][1696429.562375] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696429.567530] [] cfs_wi_scheduler+0x30c/0x460 [libcfs] [Wed Feb 6 09:38:49 2019][1696429.574339] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:49 2019][1696429.580360] [] ? cfs_wi_sched_create+0x680/0x680 [libcfs] [Wed Feb 6 09:38:49 2019][1696429.587596] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696429.592655] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.598929] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696429.605567] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.611841] cfs_rh_01 S ffff913aa9841040 0 5814 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696429.619137] Call Trace: [Wed Feb 6 09:38:49 2019][1696429.621766] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696429.626917] [] cfs_wi_scheduler+0x30c/0x460 [libcfs] [Wed Feb 6 09:38:49 2019][1696429.633725] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:49 2019][1696429.639749] [] ? cfs_wi_sched_create+0x680/0x680 [libcfs] [Wed Feb 6 09:38:49 2019][1696429.646982] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696429.652054] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.658341] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696429.664977] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.671254] cfs_rh_02 S ffff9159a73e1040 0 5815 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696429.678553] Call Trace: [Wed Feb 6 09:38:49 2019][1696429.681183] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696429.686330] [] cfs_wi_scheduler+0x30c/0x460 [libcfs] [Wed Feb 6 09:38:49 2019][1696429.693145] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:49 2019][1696429.699169] [] ? cfs_wi_sched_create+0x680/0x680 [libcfs] [Wed Feb 6 09:38:49 2019][1696429.706399] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696429.711470] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.717741] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696429.724378] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.730646] cfs_rh_03 S ffff9159a73e0000 0 5816 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696429.737922] Call Trace: [Wed Feb 6 09:38:49 2019][1696429.740555] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696429.745732] [] cfs_wi_scheduler+0x30c/0x460 [libcfs] [Wed Feb 6 09:38:49 2019][1696429.752524] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:49 2019][1696429.758541] [] ? cfs_wi_sched_create+0x680/0x680 [libcfs] [Wed Feb 6 09:38:49 2019][1696429.765776] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696429.770835] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.777111] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696429.783749] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.790024] clio S ffff9159a73e4100 0 5829 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696429.797312] Call Trace: [Wed Feb 6 09:38:49 2019][1696429.799965] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696429.805113] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:49 2019][1696429.810972] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:49 2019][1696429.817158] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:49 2019][1696429.822216] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.828505] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:49 2019][1696429.835129] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:49 2019][1696429.841417] ptlrpc_hr00_000 S ffff9159a73e30c0 0 5834 2 0x00000080 [Wed Feb 6 09:38:49 2019][1696429.848686] Call Trace: [Wed Feb 6 09:38:49 2019][1696429.851331] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:49 2019][1696429.857425] [] schedule+0x29/0x70 [Wed Feb 6 09:38:49 2019][1696429.862640] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:49 2019][1696429.869260] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:49 2019][1696429.875047] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696429.882030] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696429.887096] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696429.893380] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696429.899996] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696429.906277] ptlrpc_hr00_001 S ffff9159a73e2080 0 5835 2 0x00000080 [Wed Feb 6 09:38:50 2019][1696429.913556] Call Trace: [Wed Feb 6 09:38:50 2019][1696429.916189] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:50 2019][1696429.922290] [] schedule+0x29/0x70 [Wed Feb 6 09:38:50 2019][1696429.927489] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696429.934111] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:50 2019][1696429.939913] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696429.946880] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696429.951942] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696429.958231] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696429.964855] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696429.971144] ptlrpc_hr00_002 S ffff9159a73e5140 0 5836 2 0x00000080 [Wed Feb 6 09:38:50 2019][1696429.978427] Call Trace: [Wed Feb 6 09:38:50 2019][1696429.981072] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:50 2019][1696429.987170] [] schedule+0x29/0x70 [Wed Feb 6 09:38:50 2019][1696429.992371] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696429.998989] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:50 2019][1696430.004770] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.011747] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696430.016803] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.023089] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696430.029706] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.036000] ptlrpc_hr00_003 S ffff9159a73e6180 0 5837 2 0x00000080 [Wed Feb 6 09:38:50 2019][1696430.043275] Call Trace: [Wed Feb 6 09:38:50 2019][1696430.045907] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:50 2019][1696430.052011] [] schedule+0x29/0x70 [Wed Feb 6 09:38:50 2019][1696430.057204] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.063823] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:50 2019][1696430.069619] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.076585] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696430.081662] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.087933] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696430.094554] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.100844] ptlrpc_hr00_004 S ffff91599eaf1040 0 5838 2 0x00000080 [Wed Feb 6 09:38:50 2019][1696430.108140] Call Trace: [Wed Feb 6 09:38:50 2019][1696430.110789] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:50 2019][1696430.116888] [] schedule+0x29/0x70 [Wed Feb 6 09:38:50 2019][1696430.122084] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.128698] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:50 2019][1696430.134478] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.141459] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696430.146522] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.152810] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696430.159435] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.165725] ptlrpc_hr00_005 S ffff91599eaf2080 0 5839 2 0x00000080 [Wed Feb 6 09:38:50 2019][1696430.173013] Call Trace: [Wed Feb 6 09:38:50 2019][1696430.175651] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:50 2019][1696430.181764] [] schedule+0x29/0x70 [Wed Feb 6 09:38:50 2019][1696430.186938] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.193553] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:50 2019][1696430.199357] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.206321] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696430.211386] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.217674] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696430.224298] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.230582] ptlrpc_hr01_000 S ffff91599eaf6180 0 5841 2 0x00000080 [Wed Feb 6 09:38:50 2019][1696430.237863] Call Trace: [Wed Feb 6 09:38:50 2019][1696430.240491] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:50 2019][1696430.246600] [] schedule+0x29/0x70 [Wed Feb 6 09:38:50 2019][1696430.251777] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.258415] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:50 2019][1696430.264194] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.271169] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696430.276244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.282523] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696430.289157] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.295427] ptlrpc_hr01_001 S ffff91599eaf5140 0 5842 2 0x00000080 [Wed Feb 6 09:38:50 2019][1696430.302719] Call Trace: [Wed Feb 6 09:38:50 2019][1696430.305347] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:50 2019][1696430.311450] [] schedule+0x29/0x70 [Wed Feb 6 09:38:50 2019][1696430.316641] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.323261] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:50 2019][1696430.329038] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.336021] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696430.341086] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.347359] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696430.353995] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.360268] ptlrpc_hr01_002 S ffff91599eaf0000 0 5843 2 0x00000080 [Wed Feb 6 09:38:50 2019][1696430.367554] Call Trace: [Wed Feb 6 09:38:50 2019][1696430.370184] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:50 2019][1696430.376285] [] schedule+0x29/0x70 [Wed Feb 6 09:38:50 2019][1696430.381484] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.388102] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:50 2019][1696430.393886] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.400864] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696430.405917] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.412198] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696430.418815] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.425093] ptlrpc_hr01_003 S ffff91599eaf30c0 0 5845 2 0x00000080 [Wed Feb 6 09:38:50 2019][1696430.432394] Call Trace: [Wed Feb 6 09:38:50 2019][1696430.435024] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:50 2019][1696430.441122] [] schedule+0x29/0x70 [Wed Feb 6 09:38:50 2019][1696430.446318] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.452941] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:50 2019][1696430.458727] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.465711] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696430.470776] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.477052] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696430.483686] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.489965] ptlrpc_hr01_004 S ffff91599eaf4100 0 5846 2 0x00000080 [Wed Feb 6 09:38:50 2019][1696430.497263] Call Trace: [Wed Feb 6 09:38:50 2019][1696430.499896] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:50 2019][1696430.505995] [] schedule+0x29/0x70 [Wed Feb 6 09:38:50 2019][1696430.511187] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.517808] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:50 2019][1696430.523592] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.530575] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696430.535638] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.541910] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696430.548545] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.554815] ptlrpc_hr01_005 S ffff9159a4bea080 0 5847 2 0x00000080 [Wed Feb 6 09:38:50 2019][1696430.562091] Call Trace: [Wed Feb 6 09:38:50 2019][1696430.564721] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:50 2019][1696430.570840] [] schedule+0x29/0x70 [Wed Feb 6 09:38:50 2019][1696430.576023] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.582650] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:50 2019][1696430.588446] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.595413] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696430.600486] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.606757] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696430.613394] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.619673] ptlrpc_hr02_000 S ffff9159a4bec100 0 5848 2 0x00000080 [Wed Feb 6 09:38:50 2019][1696430.626970] Call Trace: [Wed Feb 6 09:38:50 2019][1696430.629602] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:50 2019][1696430.635707] [] schedule+0x29/0x70 [Wed Feb 6 09:38:50 2019][1696430.640899] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.647517] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:50 2019][1696430.653316] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.660287] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696430.665350] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.671636] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696430.678261] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.684550] ptlrpc_hr02_001 S ffff9159a4bed140 0 5849 2 0x00000080 [Wed Feb 6 09:38:50 2019][1696430.691825] Call Trace: [Wed Feb 6 09:38:50 2019][1696430.694472] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:50 2019][1696430.700570] [] schedule+0x29/0x70 [Wed Feb 6 09:38:50 2019][1696430.705765] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.712391] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:50 2019][1696430.718191] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.725158] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696430.730235] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.736511] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696430.743148] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.749423] ptlrpc_hr02_002 S ffff9159a4be9040 0 5850 2 0x00000080 [Wed Feb 6 09:38:50 2019][1696430.756716] Call Trace: [Wed Feb 6 09:38:50 2019][1696430.759358] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:50 2019][1696430.765454] [] schedule+0x29/0x70 [Wed Feb 6 09:38:50 2019][1696430.770648] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.777273] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:50 2019][1696430.783054] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.790036] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696430.795102] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.801395] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696430.808012] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.814311] ptlrpc_hr02_003 S ffff9159f9301040 0 5852 2 0x00000080 [Wed Feb 6 09:38:50 2019][1696430.821607] Call Trace: [Wed Feb 6 09:38:50 2019][1696430.824265] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:50 2019][1696430.830365] [] schedule+0x29/0x70 [Wed Feb 6 09:38:50 2019][1696430.835566] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.842188] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:50 2019][1696430.847993] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:50 2019][1696430.854960] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:50 2019][1696430.860042] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.866317] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:50 2019][1696430.872957] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:50 2019][1696430.879229] ptlrpc_hr02_004 S ffff9159f9304100 0 5853 2 0x00000080 [Wed Feb 6 09:38:51 2019][1696430.886537] Call Trace: [Wed Feb 6 09:38:51 2019][1696430.889168] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:51 2019][1696430.895289] [] schedule+0x29/0x70 [Wed Feb 6 09:38:51 2019][1696430.900465] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696430.907116] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:51 2019][1696430.912899] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696430.919892] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:51 2019][1696430.924958] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696430.931251] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:51 2019][1696430.937869] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696430.944166] ptlrpc_hr02_005 S ffff9159f9306180 0 5854 2 0x00000080 [Wed Feb 6 09:38:51 2019][1696430.951449] Call Trace: [Wed Feb 6 09:38:51 2019][1696430.954105] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:51 2019][1696430.960200] [] schedule+0x29/0x70 [Wed Feb 6 09:38:51 2019][1696430.965401] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696430.972026] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:51 2019][1696430.977832] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696430.984802] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:51 2019][1696430.989892] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696430.996160] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:51 2019][1696431.002805] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.009073] ptlrpc_hr03_000 S ffff9159f9302080 0 5855 2 0x00000080 [Wed Feb 6 09:38:51 2019][1696431.016369] Call Trace: [Wed Feb 6 09:38:51 2019][1696431.019000] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:51 2019][1696431.025104] [] schedule+0x29/0x70 [Wed Feb 6 09:38:51 2019][1696431.030294] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696431.036914] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:51 2019][1696431.042712] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696431.049678] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:51 2019][1696431.054762] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.061037] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:51 2019][1696431.067657] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.073944] ptlrpc_hr03_001 S ffff9159f9305140 0 5856 2 0x00000080 [Wed Feb 6 09:38:51 2019][1696431.081239] Call Trace: [Wed Feb 6 09:38:51 2019][1696431.083868] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:51 2019][1696431.089974] [] schedule+0x29/0x70 [Wed Feb 6 09:38:51 2019][1696431.095166] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696431.101786] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:51 2019][1696431.107579] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696431.114553] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:51 2019][1696431.119626] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.125897] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:51 2019][1696431.132531] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.138801] ptlrpc_hr03_002 S ffff9159f93030c0 0 5857 2 0x00000080 [Wed Feb 6 09:38:51 2019][1696431.146092] Call Trace: [Wed Feb 6 09:38:51 2019][1696431.148721] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:51 2019][1696431.154815] [] schedule+0x29/0x70 [Wed Feb 6 09:38:51 2019][1696431.160010] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696431.166628] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:51 2019][1696431.172414] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696431.179393] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:51 2019][1696431.184452] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.190734] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:51 2019][1696431.197351] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.203647] ptlrpc_hr03_003 S ffff9159f9300000 0 5858 2 0x00000080 [Wed Feb 6 09:38:51 2019][1696431.210919] Call Trace: [Wed Feb 6 09:38:51 2019][1696431.213555] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:51 2019][1696431.219655] [] schedule+0x29/0x70 [Wed Feb 6 09:38:51 2019][1696431.224847] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696431.231468] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:51 2019][1696431.237266] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696431.244234] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:51 2019][1696431.249306] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.255575] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:51 2019][1696431.262207] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.268484] ptlrpc_hr03_004 S ffff9129bef90000 0 5859 2 0x00000080 [Wed Feb 6 09:38:51 2019][1696431.275770] Call Trace: [Wed Feb 6 09:38:51 2019][1696431.278423] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:51 2019][1696431.284524] [] schedule+0x29/0x70 [Wed Feb 6 09:38:51 2019][1696431.289721] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696431.296339] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:51 2019][1696431.302119] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696431.309104] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:51 2019][1696431.314168] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.320443] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:51 2019][1696431.327078] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.333354] ptlrpc_hr03_005 S ffff9129bef930c0 0 5860 2 0x00000080 [Wed Feb 6 09:38:51 2019][1696431.340649] Call Trace: [Wed Feb 6 09:38:51 2019][1696431.343275] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:51 2019][1696431.349369] [] schedule+0x29/0x70 [Wed Feb 6 09:38:51 2019][1696431.354566] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696431.361187] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:51 2019][1696431.366969] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Wed Feb 6 09:38:51 2019][1696431.373944] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:51 2019][1696431.379000] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.385281] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:51 2019][1696431.391898] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.398176] kiblnd_connd S ffff9129bef91040 0 5873 2 0x00000080 [Wed Feb 6 09:38:51 2019][1696431.405477] Call Trace: [Wed Feb 6 09:38:51 2019][1696431.408106] [] schedule+0x29/0x70 [Wed Feb 6 09:38:51 2019][1696431.413254] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:38:51 2019][1696431.419283] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:38:51 2019][1696431.425818] [] kiblnd_connd+0x60c/0xa00 [ko2iblnd] [Wed Feb 6 09:38:51 2019][1696431.432446] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:51 2019][1696431.438219] [] ? kiblnd_check_conns+0x870/0x870 [ko2iblnd] [Wed Feb 6 09:38:51 2019][1696431.445532] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:51 2019][1696431.450606] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.456885] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:51 2019][1696431.463507] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.469796] kiblnd_sd_00_00 S ffff9129bef96180 0 5875 2 0x00000080 [Wed Feb 6 09:38:51 2019][1696431.477072] Call Trace: [Wed Feb 6 09:38:51 2019][1696431.479707] [] schedule+0x29/0x70 [Wed Feb 6 09:38:51 2019][1696431.484871] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:38:51 2019][1696431.491925] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:51 2019][1696431.498199] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:51 2019][1696431.503882] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:51 2019][1696431.509645] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:51 2019][1696431.516527] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:51 2019][1696431.521602] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.527874] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:51 2019][1696431.534498] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.540786] kiblnd_sd_00_01 S ffff9129bef92080 0 5876 2 0x00000080 [Wed Feb 6 09:38:51 2019][1696431.548058] Call Trace: [Wed Feb 6 09:38:51 2019][1696431.550697] [] schedule+0x29/0x70 [Wed Feb 6 09:38:51 2019][1696431.555862] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:38:51 2019][1696431.562911] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:51 2019][1696431.569184] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:51 2019][1696431.574865] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:51 2019][1696431.580626] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:51 2019][1696431.587516] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:51 2019][1696431.592575] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.598862] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:51 2019][1696431.605479] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.611771] kiblnd_sd_00_02 S ffff9129bef94100 0 5877 2 0x00000080 [Wed Feb 6 09:38:51 2019][1696431.619045] Call Trace: [Wed Feb 6 09:38:51 2019][1696431.621680] [] schedule+0x29/0x70 [Wed Feb 6 09:38:51 2019][1696431.626845] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:38:51 2019][1696431.633897] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:51 2019][1696431.640174] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:51 2019][1696431.645856] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:51 2019][1696431.651618] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:51 2019][1696431.658517] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:51 2019][1696431.663583] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.669870] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:51 2019][1696431.676487] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.682778] kiblnd_sd_00_03 S ffff9129bef95140 0 5878 2 0x00000080 [Wed Feb 6 09:38:51 2019][1696431.690065] Call Trace: [Wed Feb 6 09:38:51 2019][1696431.692698] [] schedule+0x29/0x70 [Wed Feb 6 09:38:51 2019][1696431.697866] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:38:51 2019][1696431.704925] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:51 2019][1696431.711206] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:51 2019][1696431.716889] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:51 2019][1696431.722649] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:51 2019][1696431.729542] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:51 2019][1696431.734607] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.740881] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:51 2019][1696431.747517] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.753796] kiblnd_sd_01_00 S ffff9159f6cf4100 0 5879 2 0x00000080 [Wed Feb 6 09:38:51 2019][1696431.761098] Call Trace: [Wed Feb 6 09:38:51 2019][1696431.763738] [] schedule+0x29/0x70 [Wed Feb 6 09:38:51 2019][1696431.768889] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:38:51 2019][1696431.775953] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:51 2019][1696431.782231] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:51 2019][1696431.787912] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:51 2019][1696431.793665] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:51 2019][1696431.800558] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:51 2019][1696431.805615] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.811906] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:51 2019][1696431.818530] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:51 2019][1696431.824817] kiblnd_sd_01_01 W ffff9159f6cf1040 0 5880 2 0x00000080 [Wed Feb 6 09:38:51 2019][1696431.832094] Call Trace: [Wed Feb 6 09:38:51 2019][1696431.834723] [] schedule+0x29/0x70 [Wed Feb 6 09:38:51 2019][1696431.839884] [] ? kiblnd_scheduler+0x4e8/0x1180 [ko2iblnd] [Wed Feb 6 09:38:51 2019][1696431.847111] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:51 2019][1696431.853388] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:51 2019][1696431.859068] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:52 2019][1696431.864831] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696431.871721] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:38:52 2019][1696431.876954] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696431.883241] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:52 2019][1696431.890034] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696431.896316] kiblnd_sd_01_02 S ffff9159f6cf30c0 0 5881 2 0x00000080 [Wed Feb 6 09:38:52 2019][1696431.903590] Call Trace: [Wed Feb 6 09:38:52 2019][1696431.906230] [] schedule+0x29/0x70 [Wed Feb 6 09:38:52 2019][1696431.911381] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696431.918446] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:52 2019][1696431.924724] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:52 2019][1696431.930405] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:52 2019][1696431.936159] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696431.943048] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:52 2019][1696431.948108] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696431.954395] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:52 2019][1696431.961022] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696431.967311] kiblnd_sd_01_03 R running task 0 5882 2 0x00000080 [Wed Feb 6 09:38:52 2019][1696431.974595] Call Trace: [Wed Feb 6 09:38:52 2019][1696431.977227] [] schedule+0x29/0x70 [Wed Feb 6 09:38:52 2019][1696431.982382] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696431.989451] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:52 2019][1696431.995719] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:52 2019][1696432.001406] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:52 2019][1696432.007170] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.014054] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:52 2019][1696432.019132] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.025407] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:52 2019][1696432.032038] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.038308] kiblnd_sd_02_00 S ffff9159f6cf5140 0 5883 2 0x00000080 [Wed Feb 6 09:38:52 2019][1696432.045592] Call Trace: [Wed Feb 6 09:38:52 2019][1696432.048223] [] schedule+0x29/0x70 [Wed Feb 6 09:38:52 2019][1696432.053376] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.060445] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:52 2019][1696432.066723] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:52 2019][1696432.072391] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:52 2019][1696432.078162] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.085043] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:52 2019][1696432.090118] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.096391] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:52 2019][1696432.103025] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.109298] kiblnd_sd_02_01 S ffff9159f6cf0000 0 5884 2 0x00000080 [Wed Feb 6 09:38:52 2019][1696432.116595] Call Trace: [Wed Feb 6 09:38:52 2019][1696432.119227] [] schedule+0x29/0x70 [Wed Feb 6 09:38:52 2019][1696432.124373] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.131443] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:52 2019][1696432.137724] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:52 2019][1696432.143388] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:52 2019][1696432.149161] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.156043] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:52 2019][1696432.161118] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.167397] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:52 2019][1696432.174033] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.180306] kiblnd_sd_02_02 S ffff9159f6fa9040 0 5885 2 0x00000080 [Wed Feb 6 09:38:52 2019][1696432.187592] Call Trace: [Wed Feb 6 09:38:52 2019][1696432.190226] [] schedule+0x29/0x70 [Wed Feb 6 09:38:52 2019][1696432.195374] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.202444] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:52 2019][1696432.208722] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:52 2019][1696432.214389] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:52 2019][1696432.220160] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.227043] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:52 2019][1696432.232118] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.238397] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:52 2019][1696432.245030] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.251298] kiblnd_sd_02_03 S ffff9159f6fae180 0 5886 2 0x00000080 [Wed Feb 6 09:38:52 2019][1696432.258594] Call Trace: [Wed Feb 6 09:38:52 2019][1696432.261229] [] schedule+0x29/0x70 [Wed Feb 6 09:38:52 2019][1696432.266372] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.273438] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:52 2019][1696432.279711] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:52 2019][1696432.285379] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:52 2019][1696432.291150] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.298033] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:52 2019][1696432.303104] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.309374] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:52 2019][1696432.316011] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.322279] kiblnd_sd_03_00 S ffff9159f6fac100 0 5887 2 0x00000080 [Wed Feb 6 09:38:52 2019][1696432.329555] Call Trace: [Wed Feb 6 09:38:52 2019][1696432.332200] [] schedule+0x29/0x70 [Wed Feb 6 09:38:52 2019][1696432.337346] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.344417] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:52 2019][1696432.350693] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:52 2019][1696432.356362] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:52 2019][1696432.362130] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.369016] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:52 2019][1696432.374092] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.380369] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:52 2019][1696432.387003] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.393278] kiblnd_sd_03_01 S ffff9159f6fab0c0 0 5888 2 0x00000080 [Wed Feb 6 09:38:52 2019][1696432.400572] Call Trace: [Wed Feb 6 09:38:52 2019][1696432.403209] [] schedule+0x29/0x70 [Wed Feb 6 09:38:52 2019][1696432.408358] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.415430] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:52 2019][1696432.421699] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:52 2019][1696432.427379] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:52 2019][1696432.433132] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.440025] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:52 2019][1696432.445085] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.451359] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:52 2019][1696432.457992] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.464263] kiblnd_sd_03_02 S ffff9159f6fa8000 0 5889 2 0x00000080 [Wed Feb 6 09:38:52 2019][1696432.471563] Call Trace: [Wed Feb 6 09:38:52 2019][1696432.474195] [] schedule+0x29/0x70 [Wed Feb 6 09:38:52 2019][1696432.479348] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.486411] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:52 2019][1696432.492684] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:52 2019][1696432.498360] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:52 2019][1696432.504114] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.511009] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:52 2019][1696432.516074] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.522349] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:52 2019][1696432.528983] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.535254] kiblnd_sd_03_03 S ffff9159ef235140 0 5890 2 0x00000080 [Wed Feb 6 09:38:52 2019][1696432.542557] Call Trace: [Wed Feb 6 09:38:52 2019][1696432.545197] [] schedule+0x29/0x70 [Wed Feb 6 09:38:52 2019][1696432.550343] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.557411] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:38:52 2019][1696432.563690] [] ? __switch_to+0xce/0x580 [Wed Feb 6 09:38:52 2019][1696432.569357] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:52 2019][1696432.575130] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Wed Feb 6 09:38:52 2019][1696432.582009] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:52 2019][1696432.587088] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.593365] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:52 2019][1696432.599983] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.606274] monitor_thread S ffff9159ef236180 0 5891 2 0x00000080 [Wed Feb 6 09:38:52 2019][1696432.613563] Call Trace: [Wed Feb 6 09:38:52 2019][1696432.616195] [] schedule+0x29/0x70 [Wed Feb 6 09:38:52 2019][1696432.621343] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:38:52 2019][1696432.627373] [] ? _raw_spin_lock+0x20/0x30 [Wed Feb 6 09:38:52 2019][1696432.633213] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:38:52 2019][1696432.639766] [] ? prepare_to_wait+0x56/0x90 [Wed Feb 6 09:38:52 2019][1696432.645709] [] lnet_monitor_thread+0x492/0x560 [lnet] [Wed Feb 6 09:38:52 2019][1696432.652595] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:52 2019][1696432.658630] [] ? lnet_recover_peer_nis+0x670/0x670 [lnet] [Wed Feb 6 09:38:52 2019][1696432.665856] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:52 2019][1696432.670932] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.677211] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:52 2019][1696432.683843] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.690123] lnet_discovery S ffff9149f3e9e180 0 5894 2 0x00000080 [Wed Feb 6 09:38:52 2019][1696432.697412] Call Trace: [Wed Feb 6 09:38:52 2019][1696432.700041] [] schedule+0x29/0x70 [Wed Feb 6 09:38:52 2019][1696432.705186] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:38:52 2019][1696432.711213] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:38:52 2019][1696432.717744] [] ? prepare_to_wait+0x56/0x90 [Wed Feb 6 09:38:52 2019][1696432.723696] [] lnet_peer_discovery+0xdf/0x1150 [lnet] [Wed Feb 6 09:38:52 2019][1696432.730580] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:52 2019][1696432.736616] [] ? lnet_peer_merge_data+0xde0/0xde0 [lnet] [Wed Feb 6 09:38:52 2019][1696432.743759] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:52 2019][1696432.748838] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.755110] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:52 2019][1696432.761727] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.768019] ptlrpcd_rcv S ffff9149f3e98000 0 5895 2 0x00000080 [Wed Feb 6 09:38:52 2019][1696432.775301] Call Trace: [Wed Feb 6 09:38:52 2019][1696432.777965] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:52 2019][1696432.784666] [] schedule+0x29/0x70 [Wed Feb 6 09:38:52 2019][1696432.789873] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:52 2019][1696432.795880] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:52 2019][1696432.801668] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:52 2019][1696432.808397] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:52 2019][1696432.813462] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.819736] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:52 2019][1696432.826370] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:52 2019][1696432.832641] ptlrpcd_00_00 S ffff9159ef2330c0 0 5896 2 0x00000080 [Wed Feb 6 09:38:52 2019][1696432.839931] Call Trace: [Wed Feb 6 09:38:52 2019][1696432.842582] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:52 2019][1696432.849287] [] schedule+0x29/0x70 [Wed Feb 6 09:38:52 2019][1696432.854483] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:52 2019][1696432.860493] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696432.866284] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696432.873009] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696432.878070] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696432.884361] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:53 2019][1696432.890988] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696432.897260] ptlrpcd_00_01 S ffff9159ef234100 0 5897 2 0x00000080 [Wed Feb 6 09:38:53 2019][1696432.904559] Call Trace: [Wed Feb 6 09:38:53 2019][1696432.907212] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:53 2019][1696432.913918] [] schedule+0x29/0x70 [Wed Feb 6 09:38:53 2019][1696432.919112] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696432.925125] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696432.930924] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696432.937631] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696432.942707] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696432.948986] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:53 2019][1696432.955618] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696432.961890] ptlrpcd_00_02 S ffff9159ef232080 0 5898 2 0x00000080 [Wed Feb 6 09:38:53 2019][1696432.969186] Call Trace: [Wed Feb 6 09:38:53 2019][1696432.971846] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:53 2019][1696432.978558] [] schedule+0x29/0x70 [Wed Feb 6 09:38:53 2019][1696432.983750] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696432.989762] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696432.995561] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.002277] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696433.007355] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.013631] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:53 2019][1696433.020264] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.026545] ptlrpcd_00_03 S ffff9159ef231040 0 5899 2 0x00000080 [Wed Feb 6 09:38:53 2019][1696433.033826] Call Trace: [Wed Feb 6 09:38:53 2019][1696433.036490] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:53 2019][1696433.043193] [] schedule+0x29/0x70 [Wed Feb 6 09:38:53 2019][1696433.048391] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.054410] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696433.060193] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.066913] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696433.071975] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.078265] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:53 2019][1696433.084887] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.091169] ptlrpcd_00_04 S ffff9159f6e9d140 0 5900 2 0x00000080 [Wed Feb 6 09:38:53 2019][1696433.098469] Call Trace: [Wed Feb 6 09:38:53 2019][1696433.101139] [] ? ptlrpc_check_set.part.23+0x11bd/0x1df0 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.108972] [] schedule+0x29/0x70 [Wed Feb 6 09:38:53 2019][1696433.114135] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:38:53 2019][1696433.120144] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:38:53 2019][1696433.126713] [] ptlrpcd+0x278/0x550 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.132748] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696433.138532] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.145252] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696433.150318] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.156601] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:53 2019][1696433.163219] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.169511] ptlrpcd_00_05 S ffff9159f6e99040 0 5901 2 0x00000080 [Wed Feb 6 09:38:53 2019][1696433.176809] Call Trace: [Wed Feb 6 09:38:53 2019][1696433.179466] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:53 2019][1696433.186174] [] schedule+0x29/0x70 [Wed Feb 6 09:38:53 2019][1696433.191365] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.197377] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696433.203173] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.209884] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696433.214959] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.221238] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:53 2019][1696433.227873] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.234150] ptlrpcd_00_06 S ffff9159f6e9b0c0 0 5902 2 0x00000080 [Wed Feb 6 09:38:53 2019][1696433.241432] Call Trace: [Wed Feb 6 09:38:53 2019][1696433.244082] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:53 2019][1696433.250789] [] schedule+0x29/0x70 [Wed Feb 6 09:38:53 2019][1696433.255984] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.261997] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696433.267784] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.274508] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696433.279566] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.285854] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:53 2019][1696433.292470] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.298756] ptlrpcd_00_07 S ffff9159f6e9a080 0 5903 2 0x00000080 [Wed Feb 6 09:38:53 2019][1696433.306040] Call Trace: [Wed Feb 6 09:38:53 2019][1696433.308699] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:53 2019][1696433.315419] [] schedule+0x29/0x70 [Wed Feb 6 09:38:53 2019][1696433.320599] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.326616] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696433.332418] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.339124] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696433.344187] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.350476] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:53 2019][1696433.357100] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.363387] ptlrpcd_00_08 S ffff9159f7a86180 0 5904 2 0x00000080 [Wed Feb 6 09:38:53 2019][1696433.370665] Call Trace: [Wed Feb 6 09:38:53 2019][1696433.373328] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:53 2019][1696433.380029] [] schedule+0x29/0x70 [Wed Feb 6 09:38:53 2019][1696433.385226] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.391239] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696433.397024] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.403742] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696433.408810] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.415095] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:53 2019][1696433.421719] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.428006] ptlrpcd_00_09 S ffff9159f7a84100 0 5905 2 0x00000080 [Wed Feb 6 09:38:53 2019][1696433.435284] Call Trace: [Wed Feb 6 09:38:53 2019][1696433.437948] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:53 2019][1696433.444654] [] schedule+0x29/0x70 [Wed Feb 6 09:38:53 2019][1696433.449849] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.455869] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696433.461666] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.468371] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696433.473451] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.479726] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:53 2019][1696433.486346] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.492636] ptlrpcd_00_10 S ffff9159f7a81040 0 5906 2 0x00000080 [Wed Feb 6 09:38:53 2019][1696433.499915] Call Trace: [Wed Feb 6 09:38:53 2019][1696433.502561] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:53 2019][1696433.509273] [] schedule+0x29/0x70 [Wed Feb 6 09:38:53 2019][1696433.514464] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.520476] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696433.526276] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.532987] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696433.538049] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.544334] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:53 2019][1696433.550952] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.557240] ptlrpcd_00_11 S ffff9159f6ba9040 0 5907 2 0x00000080 [Wed Feb 6 09:38:53 2019][1696433.564509] Call Trace: [Wed Feb 6 09:38:53 2019][1696433.567172] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:53 2019][1696433.573877] [] schedule+0x29/0x70 [Wed Feb 6 09:38:53 2019][1696433.579071] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.585093] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696433.590890] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.597598] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696433.602673] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.608950] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:53 2019][1696433.615584] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.621853] ptlrpcd_01_00 S ffff9159f6bad140 0 5908 2 0x00000080 [Wed Feb 6 09:38:53 2019][1696433.629150] Call Trace: [Wed Feb 6 09:38:53 2019][1696433.631809] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:53 2019][1696433.638521] [] schedule+0x29/0x70 [Wed Feb 6 09:38:53 2019][1696433.643713] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.649732] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696433.655521] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.662238] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696433.667306] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.673580] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:53 2019][1696433.680217] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.686493] ptlrpcd_01_01 S ffff914aa95bc100 0 5909 2 0x00000080 [Wed Feb 6 09:38:53 2019][1696433.693783] Call Trace: [Wed Feb 6 09:38:53 2019][1696433.696431] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:53 2019][1696433.703141] [] schedule+0x29/0x70 [Wed Feb 6 09:38:53 2019][1696433.708331] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.714347] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696433.720133] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.726851] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696433.731910] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.738197] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:53 2019][1696433.744820] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.751110] ptlrpcd_01_02 S ffff9159fa26c100 0 5910 2 0x00000080 [Wed Feb 6 09:38:53 2019][1696433.758378] Call Trace: [Wed Feb 6 09:38:53 2019][1696433.761028] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:53 2019][1696433.767753] [] schedule+0x29/0x70 [Wed Feb 6 09:38:53 2019][1696433.772932] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.778952] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696433.784751] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.791459] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696433.796537] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.802817] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:53 2019][1696433.809455] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:53 2019][1696433.815726] ptlrpcd_01_03 S ffff9159fa26b0c0 0 5911 2 0x00000080 [Wed Feb 6 09:38:53 2019][1696433.823023] Call Trace: [Wed Feb 6 09:38:53 2019][1696433.825676] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:53 2019][1696433.832385] [] schedule+0x29/0x70 [Wed Feb 6 09:38:53 2019][1696433.837578] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.843591] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:53 2019][1696433.849388] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:53 2019][1696433.856098] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:53 2019][1696433.861174] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696433.867449] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696433.874085] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696433.880363] ptlrpcd_01_04 S ffff9159fa26a080 0 5912 2 0x00000080 [Wed Feb 6 09:38:54 2019][1696433.887662] Call Trace: [Wed Feb 6 09:38:54 2019][1696433.890311] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:54 2019][1696433.897022] [] schedule+0x29/0x70 [Wed Feb 6 09:38:54 2019][1696433.902213] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696433.908227] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:54 2019][1696433.914024] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696433.920730] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:54 2019][1696433.925815] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696433.932086] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696433.938723] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696433.945003] ptlrpcd_01_05 S ffff9159fa268000 0 5913 2 0x00000080 [Wed Feb 6 09:38:54 2019][1696433.952305] Call Trace: [Wed Feb 6 09:38:54 2019][1696433.954957] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:54 2019][1696433.961667] [] schedule+0x29/0x70 [Wed Feb 6 09:38:54 2019][1696433.966861] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696433.972872] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:54 2019][1696433.978656] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696433.985377] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:54 2019][1696433.990443] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696433.996718] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696434.003352] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.009628] ptlrpcd_01_06 S ffff9129c185a080 0 5914 2 0x00000080 [Wed Feb 6 09:38:54 2019][1696434.016920] Call Trace: [Wed Feb 6 09:38:54 2019][1696434.019567] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:54 2019][1696434.026279] [] schedule+0x29/0x70 [Wed Feb 6 09:38:54 2019][1696434.031468] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.037487] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:54 2019][1696434.043270] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.049989] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:54 2019][1696434.055052] [] ? finish_task_switch+0x54/0x1c0 [Wed Feb 6 09:38:54 2019][1696434.061340] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.067607] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696434.074227] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.080513] ptlrpcd_01_07 S ffff9129c185c100 0 5915 2 0x00000080 [Wed Feb 6 09:38:54 2019][1696434.087810] Call Trace: [Wed Feb 6 09:38:54 2019][1696434.090478] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:54 2019][1696434.097181] [] schedule+0x29/0x70 [Wed Feb 6 09:38:54 2019][1696434.102373] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.108384] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:54 2019][1696434.114162] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.120886] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:54 2019][1696434.125948] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.132223] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696434.138859] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.145133] ptlrpcd_01_08 S ffff9129c185e180 0 5916 2 0x00000080 [Wed Feb 6 09:38:54 2019][1696434.152424] Call Trace: [Wed Feb 6 09:38:54 2019][1696434.155073] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:54 2019][1696434.161782] [] schedule+0x29/0x70 [Wed Feb 6 09:38:54 2019][1696434.166978] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.172995] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:54 2019][1696434.178777] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.185497] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:54 2019][1696434.190556] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.196846] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696434.203461] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.209736] ptlrpcd_01_09 S ffff9129c185d140 0 5917 2 0x00000080 [Wed Feb 6 09:38:54 2019][1696434.217034] Call Trace: [Wed Feb 6 09:38:54 2019][1696434.219691] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:54 2019][1696434.226394] [] schedule+0x29/0x70 [Wed Feb 6 09:38:54 2019][1696434.231589] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.237607] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:54 2019][1696434.243384] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.250108] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:54 2019][1696434.255173] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.261447] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696434.268081] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.274358] ptlrpcd_01_10 S ffff9129c1858000 0 5918 2 0x00000080 [Wed Feb 6 09:38:54 2019][1696434.281654] Call Trace: [Wed Feb 6 09:38:54 2019][1696434.284306] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:54 2019][1696434.291019] [] schedule+0x29/0x70 [Wed Feb 6 09:38:54 2019][1696434.296212] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.302235] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:54 2019][1696434.308034] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.314747] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:54 2019][1696434.319830] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.326099] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696434.332720] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.339012] ptlrpcd_01_11 S ffff9129c1859040 0 5919 2 0x00000080 [Wed Feb 6 09:38:54 2019][1696434.346308] Call Trace: [Wed Feb 6 09:38:54 2019][1696434.348956] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:54 2019][1696434.355664] [] schedule+0x29/0x70 [Wed Feb 6 09:38:54 2019][1696434.360864] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.366881] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:54 2019][1696434.372678] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.379381] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:54 2019][1696434.384454] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.390730] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696434.397352] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.403636] ptlrpcd_02_00 S ffff911b29689040 0 5920 2 0x00000080 [Wed Feb 6 09:38:54 2019][1696434.410908] Call Trace: [Wed Feb 6 09:38:54 2019][1696434.413575] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:54 2019][1696434.420286] [] schedule+0x29/0x70 [Wed Feb 6 09:38:54 2019][1696434.425478] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.431500] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:54 2019][1696434.437300] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.444016] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:54 2019][1696434.449090] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.455366] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696434.462007] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.468284] ptlrpcd_02_01 S ffff9129f6525140 0 5921 2 0x00000080 [Wed Feb 6 09:38:54 2019][1696434.475553] Call Trace: [Wed Feb 6 09:38:54 2019][1696434.478219] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:54 2019][1696434.484932] [] schedule+0x29/0x70 [Wed Feb 6 09:38:54 2019][1696434.490123] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.496138] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:54 2019][1696434.501932] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.508640] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:54 2019][1696434.513719] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.519998] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696434.526618] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.532906] ptlrpcd_02_02 S ffff9129f6520000 0 5922 2 0x00000080 [Wed Feb 6 09:38:54 2019][1696434.540184] Call Trace: [Wed Feb 6 09:38:54 2019][1696434.542832] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:54 2019][1696434.549553] [] schedule+0x29/0x70 [Wed Feb 6 09:38:54 2019][1696434.554743] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.560755] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:54 2019][1696434.566557] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.573268] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:54 2019][1696434.578325] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.584611] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696434.591226] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.597507] ptlrpcd_02_03 S ffff9129fc01c100 0 5923 2 0x00000080 [Wed Feb 6 09:38:54 2019][1696434.604778] Call Trace: [Wed Feb 6 09:38:54 2019][1696434.607438] [] ? ptlrpc_check_set.part.23+0x11bd/0x1df0 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.615274] [] schedule+0x29/0x70 [Wed Feb 6 09:38:54 2019][1696434.620435] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:38:54 2019][1696434.626452] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:38:54 2019][1696434.633032] [] ptlrpcd+0x278/0x550 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.639043] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:54 2019][1696434.644843] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.651547] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:54 2019][1696434.656617] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.662889] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696434.669525] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.675800] ptlrpcd_02_04 S ffff91291a3e4100 0 5924 2 0x00000080 [Wed Feb 6 09:38:54 2019][1696434.683099] Call Trace: [Wed Feb 6 09:38:54 2019][1696434.685750] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:54 2019][1696434.692457] [] schedule+0x29/0x70 [Wed Feb 6 09:38:54 2019][1696434.697648] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.703664] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:54 2019][1696434.709451] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.716178] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:54 2019][1696434.721243] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.727531] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696434.734156] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.740441] ptlrpcd_02_05 S ffff91291a3e30c0 0 5925 2 0x00000080 [Wed Feb 6 09:38:54 2019][1696434.747721] Call Trace: [Wed Feb 6 09:38:54 2019][1696434.750392] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:54 2019][1696434.757099] [] schedule+0x29/0x70 [Wed Feb 6 09:38:54 2019][1696434.762296] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.768312] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:54 2019][1696434.774106] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.780818] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:54 2019][1696434.785894] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.792172] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696434.798805] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.805079] ptlrpcd_02_06 S ffff91291a3e0000 0 5926 2 0x00000080 [Wed Feb 6 09:38:54 2019][1696434.812377] Call Trace: [Wed Feb 6 09:38:54 2019][1696434.815026] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:54 2019][1696434.821736] [] schedule+0x29/0x70 [Wed Feb 6 09:38:54 2019][1696434.826930] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.832948] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:54 2019][1696434.838746] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:54 2019][1696434.845460] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:54 2019][1696434.850519] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.856808] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:54 2019][1696434.863429] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:54 2019][1696434.869716] ptlrpcd_02_07 S ffff91291a3e2080 0 5927 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696434.876988] Call Trace: [Wed Feb 6 09:38:55 2019][1696434.879640] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:55 2019][1696434.886347] [] schedule+0x29/0x70 [Wed Feb 6 09:38:55 2019][1696434.891540] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696434.897561] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:55 2019][1696434.903357] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696434.910069] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:55 2019][1696434.915143] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696434.921422] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:55 2019][1696434.928057] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696434.934334] ptlrpcd_02_08 S ffff91291a3e1040 0 5928 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696434.941634] Call Trace: [Wed Feb 6 09:38:55 2019][1696434.944282] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:55 2019][1696434.950991] [] schedule+0x29/0x70 [Wed Feb 6 09:38:55 2019][1696434.956185] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696434.962201] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:55 2019][1696434.967984] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696434.974711] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:55 2019][1696434.979777] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696434.986062] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:55 2019][1696434.992679] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696434.998968] ptlrpcd_02_09 S ffff91291a3e6180 0 5929 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696435.006245] Call Trace: [Wed Feb 6 09:38:55 2019][1696435.008907] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:55 2019][1696435.015615] [] schedule+0x29/0x70 [Wed Feb 6 09:38:55 2019][1696435.020801] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.026821] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:55 2019][1696435.032618] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.039326] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:55 2019][1696435.044399] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.050668] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:55 2019][1696435.057304] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.063573] ptlrpcd_02_10 S ffff91291a3e5140 0 5930 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696435.070871] Call Trace: [Wed Feb 6 09:38:55 2019][1696435.073544] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:55 2019][1696435.080252] [] schedule+0x29/0x70 [Wed Feb 6 09:38:55 2019][1696435.085439] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.091458] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:55 2019][1696435.097257] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.103961] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:55 2019][1696435.109040] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.115316] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:55 2019][1696435.121939] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.128224] ptlrpcd_02_11 S ffff9129a52b9040 0 5931 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696435.135508] Call Trace: [Wed Feb 6 09:38:55 2019][1696435.138177] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:55 2019][1696435.144889] [] schedule+0x29/0x70 [Wed Feb 6 09:38:55 2019][1696435.150081] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.156094] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:55 2019][1696435.161896] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.168610] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:55 2019][1696435.173686] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.179960] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:55 2019][1696435.186595] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.192867] ptlrpcd_03_00 S ffff9129a52be180 0 5932 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696435.200160] Call Trace: [Wed Feb 6 09:38:55 2019][1696435.202823] [] ? ptlrpc_check_set.part.23+0x11bd/0x1df0 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.210677] [] schedule+0x29/0x70 [Wed Feb 6 09:38:55 2019][1696435.215821] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:38:55 2019][1696435.221835] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:38:55 2019][1696435.228410] [] ptlrpcd+0x278/0x550 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.234427] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:55 2019][1696435.240224] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.246934] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:55 2019][1696435.252004] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.258274] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:55 2019][1696435.264910] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.271183] ptlrpcd_03_01 S ffff9129a52bc100 0 5933 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696435.278456] Call Trace: [Wed Feb 6 09:38:55 2019][1696435.281121] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:55 2019][1696435.287833] [] schedule+0x29/0x70 [Wed Feb 6 09:38:55 2019][1696435.293023] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.299035] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:55 2019][1696435.304836] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.311545] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:55 2019][1696435.316606] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.322896] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:55 2019][1696435.329519] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.335805] ptlrpcd_03_02 S ffff9129a52b8000 0 5934 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696435.343074] Call Trace: [Wed Feb 6 09:38:55 2019][1696435.345724] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:55 2019][1696435.352453] [] schedule+0x29/0x70 [Wed Feb 6 09:38:55 2019][1696435.357631] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.363649] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:55 2019][1696435.369449] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.376157] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:55 2019][1696435.381231] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.387507] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:55 2019][1696435.394142] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.400409] ptlrpcd_03_03 S ffff9129a52bd140 0 5935 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696435.407692] Call Trace: [Wed Feb 6 09:38:55 2019][1696435.410373] [] ? ptlrpc_check_set.part.23+0x11bd/0x1df0 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.418210] [] schedule+0x29/0x70 [Wed Feb 6 09:38:55 2019][1696435.423365] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:38:55 2019][1696435.429381] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:38:55 2019][1696435.435955] [] ptlrpcd+0x278/0x550 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.441972] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:55 2019][1696435.447769] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.454478] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:55 2019][1696435.459550] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.465821] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:55 2019][1696435.472455] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.478724] ptlrpcd_03_04 S ffff9129a52ba080 0 5936 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696435.486013] Call Trace: [Wed Feb 6 09:38:55 2019][1696435.488667] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:55 2019][1696435.495393] [] schedule+0x29/0x70 [Wed Feb 6 09:38:55 2019][1696435.500580] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.506602] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:55 2019][1696435.512385] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.519098] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:55 2019][1696435.524172] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.530442] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:55 2019][1696435.537074] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.543348] ptlrpcd_03_05 S ffff9129a52bb0c0 0 5937 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696435.550650] Call Trace: [Wed Feb 6 09:38:55 2019][1696435.553299] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:55 2019][1696435.560005] [] schedule+0x29/0x70 [Wed Feb 6 09:38:55 2019][1696435.565200] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.571218] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:55 2019][1696435.577019] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.583724] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:55 2019][1696435.588788] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.595079] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:55 2019][1696435.601699] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.607974] ptlrpcd_03_06 S ffff9129f33b6180 0 5938 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696435.615268] Call Trace: [Wed Feb 6 09:38:55 2019][1696435.617917] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:55 2019][1696435.624629] [] schedule+0x29/0x70 [Wed Feb 6 09:38:55 2019][1696435.629818] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.635829] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:55 2019][1696435.641629] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.648336] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:55 2019][1696435.653403] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.659686] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:55 2019][1696435.666308] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.672594] ptlrpcd_03_07 S ffff9129f33b5140 0 5939 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696435.679869] Call Trace: [Wed Feb 6 09:38:55 2019][1696435.682531] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:55 2019][1696435.689239] [] schedule+0x29/0x70 [Wed Feb 6 09:38:55 2019][1696435.694431] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.700445] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:55 2019][1696435.706244] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.712952] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:55 2019][1696435.718027] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.724302] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:55 2019][1696435.730935] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.737206] ptlrpcd_03_08 S ffff9129e9aa6180 0 5940 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696435.744502] Call Trace: [Wed Feb 6 09:38:55 2019][1696435.747163] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:55 2019][1696435.753876] [] schedule+0x29/0x70 [Wed Feb 6 09:38:55 2019][1696435.759065] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.765080] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:55 2019][1696435.770881] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.777591] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:55 2019][1696435.782660] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.788934] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:55 2019][1696435.795570] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.801848] ptlrpcd_03_09 S ffff9129e9aa30c0 0 5941 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696435.809147] Call Trace: [Wed Feb 6 09:38:55 2019][1696435.811798] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:55 2019][1696435.818506] [] schedule+0x29/0x70 [Wed Feb 6 09:38:55 2019][1696435.823698] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.829709] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:55 2019][1696435.835506] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:55 2019][1696435.842213] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:55 2019][1696435.847268] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.853558] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:55 2019][1696435.860186] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:55 2019][1696435.866456] ptlrpcd_03_10 S ffff9129e9aa4100 0 5942 2 0x00000080 [Wed Feb 6 09:38:55 2019][1696435.873761] Call Trace: [Wed Feb 6 09:38:56 2019][1696435.876419] [] ? lu_env_refill+0x24/0x30 [obdclass] [Wed Feb 6 09:38:56 2019][1696435.883125] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696435.888319] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696435.894328] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:56 2019][1696435.900127] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696435.906837] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:56 2019][1696435.911900] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696435.918188] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:56 2019][1696435.924809] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696435.931097] ptlrpcd_03_11 S ffff9129e9aa2080 0 5943 2 0x00000080 [Wed Feb 6 09:38:56 2019][1696435.938369] Call Trace: [Wed Feb 6 09:38:56 2019][1696435.941027] [] ? ptlrpc_check_set.part.23+0x11bd/0x1df0 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696435.948870] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696435.954033] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:38:56 2019][1696435.960053] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:38:56 2019][1696435.966631] [] ptlrpcd+0x278/0x550 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696435.972643] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:56 2019][1696435.978438] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696435.985145] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:56 2019][1696435.990204] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696435.996494] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:56 2019][1696436.003119] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.009407] kmmpd-dm-0 S ffff9129e9aa1040 0 6009 2 0x00000080 [Wed Feb 6 09:38:56 2019][1696436.016704] Call Trace: [Wed Feb 6 09:38:56 2019][1696436.019354] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696436.024500] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:38:56 2019][1696436.030526] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:38:56 2019][1696436.037069] [] ? write_mmp_block+0x12a/0x160 [ldiskfs] [Wed Feb 6 09:38:56 2019][1696436.044047] [] schedule_timeout_interruptible+0x1e/0x20 [Wed Feb 6 09:38:56 2019][1696436.051103] [] kmmpd+0x216/0x430 [ldiskfs] [Wed Feb 6 09:38:56 2019][1696436.057056] [] ? __dump_mmp_msg+0x70/0x70 [ldiskfs] [Wed Feb 6 09:38:56 2019][1696436.063761] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:56 2019][1696436.068822] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.075110] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:56 2019][1696436.081736] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.088023] jbd2/dm-0-8 S ffff9129e9aa5140 0 6010 2 0x00000080 [Wed Feb 6 09:38:56 2019][1696436.095294] Call Trace: [Wed Feb 6 09:38:56 2019][1696436.097939] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696436.103102] [] kjournald2+0x203/0x260 [jbd2] [Wed Feb 6 09:38:56 2019][1696436.109212] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:56 2019][1696436.115229] [] ? commit_timeout+0x10/0x10 [jbd2] [Wed Feb 6 09:38:56 2019][1696436.121694] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:56 2019][1696436.126761] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.133050] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:56 2019][1696436.139673] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.145962] ldiskfs-rsv-con S ffff9129f5cc30c0 0 6011 2 0x00000080 [Wed Feb 6 09:38:56 2019][1696436.153255] Call Trace: [Wed Feb 6 09:38:56 2019][1696436.155898] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696436.161049] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:56 2019][1696436.166902] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:56 2019][1696436.173092] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:56 2019][1696436.178169] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.184447] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:56 2019][1696436.191078] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.197353] ldlm_cb00_000 S ffff9129f5cc6180 0 6019 2 0x00000080 [Wed Feb 6 09:38:56 2019][1696436.204653] Call Trace: [Wed Feb 6 09:38:56 2019][1696436.207282] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696436.212466] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.219364] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:56 2019][1696436.225150] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.231640] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.239220] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:56 2019][1696436.244297] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.250573] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:56 2019][1696436.257211] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.263491] lc_watchdogd S ffff9129f5cc5140 0 6020 2 0x00000080 [Wed Feb 6 09:38:56 2019][1696436.270792] Call Trace: [Wed Feb 6 09:38:56 2019][1696436.273426] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696436.278582] [] lcw_dispatch_main+0x749/0x940 [libcfs] [Wed Feb 6 09:38:56 2019][1696436.285478] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:56 2019][1696436.291494] [] ? lc_watchdog_add+0x790/0x790 [libcfs] [Wed Feb 6 09:38:56 2019][1696436.298370] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:56 2019][1696436.303446] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.309717] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:56 2019][1696436.316340] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.322626] ldlm_cb00_001 S ffff9129f5cc1040 0 6021 2 0x00000080 [Wed Feb 6 09:38:56 2019][1696436.329895] Call Trace: [Wed Feb 6 09:38:56 2019][1696436.332538] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696436.337717] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.344613] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:56 2019][1696436.350390] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.356887] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.364467] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:56 2019][1696436.369542] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.375821] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:56 2019][1696436.382451] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.388722] ldlm_cb01_000 S ffff9129f5cc2080 0 6022 2 0x00000080 [Wed Feb 6 09:38:56 2019][1696436.396027] Call Trace: [Wed Feb 6 09:38:56 2019][1696436.398654] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696436.403832] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.410728] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:56 2019][1696436.416513] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.422988] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.430575] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:56 2019][1696436.435638] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.441925] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:56 2019][1696436.448545] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.454832] ldlm_cb01_001 S ffff9129ea33e180 0 6023 2 0x00000080 [Wed Feb 6 09:38:56 2019][1696436.462121] Call Trace: [Wed Feb 6 09:38:56 2019][1696436.464758] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696436.469954] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.476833] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:56 2019][1696436.482616] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.489111] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.496688] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:56 2019][1696436.501766] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.508043] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:56 2019][1696436.514678] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.520957] ldlm_cb02_000 S ffff9129ea338000 0 6024 2 0x00000080 [Wed Feb 6 09:38:56 2019][1696436.528246] Call Trace: [Wed Feb 6 09:38:56 2019][1696436.530879] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696436.536059] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.542958] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:56 2019][1696436.548745] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.555233] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.562808] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:56 2019][1696436.567885] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.574158] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:56 2019][1696436.580795] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.587066] ldlm_cb02_001 S ffff9129ea33a080 0 6026 2 0x00000080 [Wed Feb 6 09:38:56 2019][1696436.594363] Call Trace: [Wed Feb 6 09:38:56 2019][1696436.597011] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696436.602184] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.609076] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:56 2019][1696436.614862] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.621338] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.628923] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:56 2019][1696436.633980] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.640269] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:56 2019][1696436.646895] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.653185] ldlm_cb03_000 S ffff9129ea33b0c0 0 6027 2 0x00000080 [Wed Feb 6 09:38:56 2019][1696436.660482] Call Trace: [Wed Feb 6 09:38:56 2019][1696436.663128] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696436.668306] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.675200] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:56 2019][1696436.680983] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.687472] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.695045] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:56 2019][1696436.700121] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.706389] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:56 2019][1696436.713010] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.719301] ldlm_cb03_001 S ffff9129ea339040 0 6028 2 0x00000080 [Wed Feb 6 09:38:56 2019][1696436.726591] Call Trace: [Wed Feb 6 09:38:56 2019][1696436.729226] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696436.734405] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.741297] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:56 2019][1696436.747073] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.753548] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.761138] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:56 2019][1696436.766202] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.772477] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:56 2019][1696436.779110] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.785379] ldlm_cn00_000 S ffff9129ea33d140 0 6029 2 0x00000080 [Wed Feb 6 09:38:56 2019][1696436.792672] Call Trace: [Wed Feb 6 09:38:56 2019][1696436.795301] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696436.800472] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.807370] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:38:56 2019][1696436.814340] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:56 2019][1696436.820121] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.826609] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:56 2019][1696436.834183] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:56 2019][1696436.839259] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.845528] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:56 2019][1696436.852149] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:56 2019][1696436.858441] ldlm_cn00_001 S ffff9129f3d20000 0 6030 2 0x00000080 [Wed Feb 6 09:38:56 2019][1696436.865726] Call Trace: [Wed Feb 6 09:38:56 2019][1696436.868357] [] schedule+0x29/0x70 [Wed Feb 6 09:38:56 2019][1696436.873536] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696436.880427] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:57 2019][1696436.886203] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696436.892684] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696436.900277] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:57 2019][1696436.905341] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696436.911629] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:57 2019][1696436.918253] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696436.924545] ldlm_cn00_002 S ffff9129f3d21040 0 6031 2 0x00000080 [Wed Feb 6 09:38:57 2019][1696436.931820] Call Trace: [Wed Feb 6 09:38:57 2019][1696436.934450] [] schedule+0x29/0x70 [Wed Feb 6 09:38:57 2019][1696436.939630] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696436.946524] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:57 2019][1696436.952310] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696436.958788] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696436.966378] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:57 2019][1696436.971444] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696436.977719] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:57 2019][1696436.984353] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696436.990631] ldlm_cn01_000 S ffff9129f3d26180 0 6032 2 0x00000080 [Wed Feb 6 09:38:57 2019][1696436.997936] Call Trace: [Wed Feb 6 09:38:57 2019][1696437.000570] [] schedule+0x29/0x70 [Wed Feb 6 09:38:57 2019][1696437.005752] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.012648] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:57 2019][1696437.018425] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.024921] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.032498] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:57 2019][1696437.037577] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.043854] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:57 2019][1696437.050489] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.056756] ldlm_cn01_001 S ffff9129f3d24100 0 6033 2 0x00000080 [Wed Feb 6 09:38:57 2019][1696437.064055] Call Trace: [Wed Feb 6 09:38:57 2019][1696437.066692] [] schedule+0x29/0x70 [Wed Feb 6 09:38:57 2019][1696437.071872] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.078771] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:57 2019][1696437.084559] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.091056] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.098630] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:57 2019][1696437.103700] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.109974] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:57 2019][1696437.116608] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.122876] ldlm_cn01_002 S ffff9129f3d22080 0 6034 2 0x00000080 [Wed Feb 6 09:38:57 2019][1696437.130173] Call Trace: [Wed Feb 6 09:38:57 2019][1696437.132802] [] schedule+0x29/0x70 [Wed Feb 6 09:38:57 2019][1696437.137992] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.144875] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:57 2019][1696437.150661] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.157156] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.164729] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:57 2019][1696437.169810] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.176086] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:57 2019][1696437.182709] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.188996] ldlm_cn02_000 S ffff9129f3d25140 0 6035 2 0x00000080 [Wed Feb 6 09:38:57 2019][1696437.196294] Call Trace: [Wed Feb 6 09:38:57 2019][1696437.198946] [] schedule+0x29/0x70 [Wed Feb 6 09:38:57 2019][1696437.204128] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.211028] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:57 2019][1696437.216814] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.223311] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.230885] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:57 2019][1696437.235963] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.242241] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:57 2019][1696437.248863] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.255140] ldlm_cn02_001 S ffff911b295a8000 0 6036 2 0x00000080 [Wed Feb 6 09:38:57 2019][1696437.262431] Call Trace: [Wed Feb 6 09:38:57 2019][1696437.265068] [] schedule+0x29/0x70 [Wed Feb 6 09:38:57 2019][1696437.270247] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.277141] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:57 2019][1696437.282930] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.289424] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.296998] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:57 2019][1696437.302059] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.308334] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:57 2019][1696437.314969] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.321247] ldlm_cn02_002 S ffff911b295ab0c0 0 6037 2 0x00000080 [Wed Feb 6 09:38:57 2019][1696437.328550] Call Trace: [Wed Feb 6 09:38:57 2019][1696437.331187] [] schedule+0x29/0x70 [Wed Feb 6 09:38:57 2019][1696437.336366] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.343257] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:57 2019][1696437.349047] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.355538] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.363117] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:57 2019][1696437.368191] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.374464] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:57 2019][1696437.381098] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.387375] ldlm_cn03_000 S ffff9129f3049040 0 6038 2 0x00000080 [Wed Feb 6 09:38:57 2019][1696437.394674] Call Trace: [Wed Feb 6 09:38:57 2019][1696437.397304] [] schedule+0x29/0x70 [Wed Feb 6 09:38:57 2019][1696437.402488] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.409386] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:57 2019][1696437.415173] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.421658] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.429237] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:57 2019][1696437.434312] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.440591] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:57 2019][1696437.447228] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.453499] ldlm_cn03_001 S ffff9129f304c100 0 6039 2 0x00000080 [Wed Feb 6 09:38:57 2019][1696437.460786] Call Trace: [Wed Feb 6 09:38:57 2019][1696437.463416] [] schedule+0x29/0x70 [Wed Feb 6 09:38:57 2019][1696437.468595] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.475491] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:57 2019][1696437.481277] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.487758] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.495342] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:57 2019][1696437.500402] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.506687] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:57 2019][1696437.513302] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.519593] ldlm_cn03_002 S ffff9129f304d140 0 6040 2 0x00000080 [Wed Feb 6 09:38:57 2019][1696437.526883] Call Trace: [Wed Feb 6 09:38:57 2019][1696437.529521] [] schedule+0x29/0x70 [Wed Feb 6 09:38:57 2019][1696437.534700] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.541601] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:57 2019][1696437.547389] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.553882] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.561460] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:57 2019][1696437.566534] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.572813] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:57 2019][1696437.579448] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.585726] ldlm_bl_01 S ffff9129f304b0c0 0 6041 2 0x00000080 [Wed Feb 6 09:38:57 2019][1696437.593026] Call Trace: [Wed Feb 6 09:38:57 2019][1696437.595656] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:38:57 2019][1696437.601525] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:38:57 2019][1696437.608942] [] schedule+0x29/0x70 [Wed Feb 6 09:38:57 2019][1696437.614122] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.621188] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:57 2019][1696437.626971] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.634558] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:57 2019][1696437.639615] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.645891] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:57 2019][1696437.652527] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.658805] ldlm_bl_02 S ffff9129f304e180 0 6042 2 0x00000080 [Wed Feb 6 09:38:57 2019][1696437.666103] Call Trace: [Wed Feb 6 09:38:57 2019][1696437.668733] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:38:57 2019][1696437.674593] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:38:57 2019][1696437.682011] [] schedule+0x29/0x70 [Wed Feb 6 09:38:57 2019][1696437.687188] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.694241] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:57 2019][1696437.700042] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.707620] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:57 2019][1696437.712688] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.718976] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:57 2019][1696437.725603] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.731887] ldlm_elt S ffff9139f615b0c0 0 6043 2 0x00000080 [Wed Feb 6 09:38:57 2019][1696437.739166] Call Trace: [Wed Feb 6 09:38:57 2019][1696437.741797] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:57 2019][1696437.747923] [] schedule+0x29/0x70 [Wed Feb 6 09:38:57 2019][1696437.753105] [] expired_lock_main+0x1c2/0xc00 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.759987] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:57 2019][1696437.765784] [] ? ldlm_server_glimpse_ast+0x3f0/0x3f0 [ptlrpc] [Wed Feb 6 09:38:57 2019][1696437.773352] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:57 2019][1696437.778418] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.784706] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:57 2019][1696437.791331] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.797618] mgs_params_noti S ffff9139f6158000 0 6044 2 0x00000080 [Wed Feb 6 09:38:57 2019][1696437.804903] Call Trace: [Wed Feb 6 09:38:57 2019][1696437.807548] [] ? mgc_name2resid+0x151/0x220 [mgc] [Wed Feb 6 09:38:57 2019][1696437.814081] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:57 2019][1696437.820194] [] schedule+0x29/0x70 [Wed Feb 6 09:38:57 2019][1696437.825345] [] mgs_ir_notify+0x1c0/0x2b0 [mgs] [Wed Feb 6 09:38:57 2019][1696437.831633] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:57 2019][1696437.837393] [] ? lprocfs_ir_set_state+0x170/0x170 [mgs] [Wed Feb 6 09:38:57 2019][1696437.844451] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:57 2019][1696437.849523] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.855793] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:57 2019][1696437.862428] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:57 2019][1696437.868695] ll_mgs_0000 S ffff9139f615c100 0 6046 2 0x00000080 [Wed Feb 6 09:38:57 2019][1696437.875975] Call Trace: [Wed Feb 6 09:38:57 2019][1696437.878620] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696437.883803] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696437.890702] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:58 2019][1696437.896490] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696437.902972] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696437.910548] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:58 2019][1696437.915616] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696437.921885] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:58 2019][1696437.928512] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696437.934799] ll_mgs_0001 S ffff9139f615a080 0 6047 2 0x00000080 [Wed Feb 6 09:38:58 2019][1696437.942085] Call Trace: [Wed Feb 6 09:38:58 2019][1696437.944712] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696437.949897] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696437.956781] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:58 2019][1696437.962567] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696437.969064] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696437.976646] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:58 2019][1696437.981720] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696437.987990] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:58 2019][1696437.994627] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.000903] ll_mgs_0002 S ffff912aa9ad2080 0 6048 2 0x00000080 [Wed Feb 6 09:38:58 2019][1696438.008194] Call Trace: [Wed Feb 6 09:38:58 2019][1696438.010824] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696438.016008] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696438.022918] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:58 2019][1696438.028722] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696438.035216] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696438.042789] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:58 2019][1696438.047865] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.054134] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:58 2019][1696438.060754] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.067045] ll_evictor S ffff912aa9ad0000 0 6049 2 0x00000080 [Wed Feb 6 09:38:58 2019][1696438.074335] Call Trace: [Wed Feb 6 09:38:58 2019][1696438.076970] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:58 2019][1696438.083076] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696438.088265] [] ping_evictor_main+0x4be/0x700 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696438.095145] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:58 2019][1696438.100952] [] ? ptlrpc_del_timeout_client+0x190/0x190 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696438.108703] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:58 2019][1696438.113765] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.120052] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:58 2019][1696438.126671] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.132953] ll_cfg_requeue S ffff912aa9ad30c0 0 6050 2 0x00000080 [Wed Feb 6 09:38:58 2019][1696438.140226] Call Trace: [Wed Feb 6 09:38:58 2019][1696438.142855] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:58 2019][1696438.148967] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696438.154120] [] mgc_requeue_thread+0x5f5/0x870 [mgc] [Wed Feb 6 09:38:58 2019][1696438.160835] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:58 2019][1696438.166605] [] ? mgc_process_config+0x13f0/0x13f0 [mgc] [Wed Feb 6 09:38:58 2019][1696438.173662] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:58 2019][1696438.178737] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.185012] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:58 2019][1696438.191651] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.197930] mgs_fir_notify S ffff912aa9ad6180 0 6062 2 0x00000080 [Wed Feb 6 09:38:58 2019][1696438.205223] Call Trace: [Wed Feb 6 09:38:58 2019][1696438.207863] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:38:58 2019][1696438.213977] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696438.219131] [] mgs_ir_notify+0x1c0/0x2b0 [mgs] [Wed Feb 6 09:38:58 2019][1696438.225413] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:58 2019][1696438.231188] [] ? lprocfs_ir_set_state+0x170/0x170 [mgs] [Wed Feb 6 09:38:58 2019][1696438.238237] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:58 2019][1696438.243315] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.249595] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:58 2019][1696438.256222] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.262495] ll_mgs_0003 S ffff9139ef7db0c0 0 6070 2 0x00000080 [Wed Feb 6 09:38:58 2019][1696438.269787] Call Trace: [Wed Feb 6 09:38:58 2019][1696438.272422] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696438.277602] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696438.284500] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:58 2019][1696438.290286] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696438.296776] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696438.304349] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:58 2019][1696438.309426] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.315705] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:58 2019][1696438.322335] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.328606] ldlm_cn01_003 S ffff91396b9f9040 0 6071 2 0x00000080 [Wed Feb 6 09:38:58 2019][1696438.335897] Call Trace: [Wed Feb 6 09:38:58 2019][1696438.338527] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696438.343699] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696438.350591] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:58 2019][1696438.356370] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696438.362856] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696438.370449] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:58 2019][1696438.375513] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.381798] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:58 2019][1696438.388416] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.394703] ldlm_cn02_003 S ffff91396b9fe180 0 6072 2 0x00000080 [Wed Feb 6 09:38:58 2019][1696438.401983] Call Trace: [Wed Feb 6 09:38:58 2019][1696438.404612] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696438.409784] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696438.416676] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:58 2019][1696438.422457] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696438.428942] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696438.436533] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:58 2019][1696438.441600] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.447875] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:58 2019][1696438.454504] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.460780] kmmpd-dm-3 S ffff91396b9fc100 0 6148 2 0x00000080 [Wed Feb 6 09:38:58 2019][1696438.468077] Call Trace: [Wed Feb 6 09:38:58 2019][1696438.470708] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696438.475860] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:38:58 2019][1696438.481886] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:38:58 2019][1696438.488439] [] ? write_mmp_block+0x12a/0x160 [ldiskfs] [Wed Feb 6 09:38:58 2019][1696438.495426] [] schedule_timeout_interruptible+0x1e/0x20 [Wed Feb 6 09:38:58 2019][1696438.502488] [] kmmpd+0x216/0x430 [ldiskfs] [Wed Feb 6 09:38:58 2019][1696438.508438] [] ? __dump_mmp_msg+0x70/0x70 [ldiskfs] [Wed Feb 6 09:38:58 2019][1696438.515143] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:58 2019][1696438.520203] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.526498] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:58 2019][1696438.533122] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.539395] kmmpd-dm-4 S ffff91396b9f8000 0 6149 2 0x00000080 [Wed Feb 6 09:38:58 2019][1696438.546678] Call Trace: [Wed Feb 6 09:38:58 2019][1696438.549305] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696438.554446] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:38:58 2019][1696438.560475] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:38:58 2019][1696438.567012] [] ? write_mmp_block+0x12a/0x160 [ldiskfs] [Wed Feb 6 09:38:58 2019][1696438.573986] [] schedule_timeout_interruptible+0x1e/0x20 [Wed Feb 6 09:38:58 2019][1696438.581061] [] kmmpd+0x216/0x430 [ldiskfs] [Wed Feb 6 09:38:58 2019][1696438.586992] [] ? __dump_mmp_msg+0x70/0x70 [ldiskfs] [Wed Feb 6 09:38:58 2019][1696438.593716] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:58 2019][1696438.598780] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.605066] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:58 2019][1696438.611684] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.617975] jbd2/dm-4-8 S ffff91396b9fa080 0 6150 2 0x00000080 [Wed Feb 6 09:38:58 2019][1696438.625261] Call Trace: [Wed Feb 6 09:38:58 2019][1696438.627890] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696438.633055] [] kjournald2+0x203/0x260 [jbd2] [Wed Feb 6 09:38:58 2019][1696438.639178] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:58 2019][1696438.645194] [] ? commit_timeout+0x10/0x10 [jbd2] [Wed Feb 6 09:38:58 2019][1696438.651651] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:58 2019][1696438.656712] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.663003] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:58 2019][1696438.669636] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.675904] jbd2/dm-3-8 S ffff91396b9fd140 0 6151 2 0x00000080 [Wed Feb 6 09:38:58 2019][1696438.683200] Call Trace: [Wed Feb 6 09:38:58 2019][1696438.685828] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696438.690979] [] kjournald2+0x203/0x260 [jbd2] [Wed Feb 6 09:38:58 2019][1696438.697094] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:38:58 2019][1696438.703120] [] ? commit_timeout+0x10/0x10 [jbd2] [Wed Feb 6 09:38:58 2019][1696438.709567] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:58 2019][1696438.714642] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.720916] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:58 2019][1696438.727537] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.733823] ldiskfs-rsv-con S ffff912aa98ba080 0 6152 2 0x00000080 [Wed Feb 6 09:38:58 2019][1696438.741099] Call Trace: [Wed Feb 6 09:38:58 2019][1696438.743729] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696438.748884] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:58 2019][1696438.754722] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:58 2019][1696438.760908] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:58 2019][1696438.765977] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.772245] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:58 2019][1696438.778871] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.785158] ldiskfs-rsv-con S ffff912aa98b8000 0 6153 2 0x00000080 [Wed Feb 6 09:38:58 2019][1696438.792438] Call Trace: [Wed Feb 6 09:38:58 2019][1696438.795078] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696438.800222] [] rescuer_thread+0x31e/0x3c0 [Wed Feb 6 09:38:58 2019][1696438.806078] [] ? process_one_work+0x440/0x440 [Wed Feb 6 09:38:58 2019][1696438.812259] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:58 2019][1696438.817315] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.823605] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:58 2019][1696438.830229] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:58 2019][1696438.836505] mdt00_000 S ffff9139f1ff6180 0 6167 2 0x00000080 [Wed Feb 6 09:38:58 2019][1696438.843804] Call Trace: [Wed Feb 6 09:38:58 2019][1696438.846435] [] schedule+0x29/0x70 [Wed Feb 6 09:38:58 2019][1696438.851618] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:38:58 2019][1696438.858517] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:59 2019][1696438.864300] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696438.870788] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696438.878379] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1696438.883445] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1696438.889730] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1696438.896347] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1696438.902640] mdt00_001 S ffff9139f1ff4100 0 6168 2 0x00000080 [Wed Feb 6 09:38:59 2019][1696438.909915] Call Trace: [Wed Feb 6 09:38:59 2019][1696438.912557] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:38:59 2019][1696438.919466] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696438.927228] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1696438.932404] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696438.939501] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696438.946579] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:38:59 2019][1696438.953739] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:59 2019][1696438.959522] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696438.966844] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:38:59 2019][1696438.974206] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696438.982234] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:38:59 2019][1696438.988961] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:38:59 2019][1696438.995960] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:38:59 2019][1696439.002697] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.010744] [] ? htable_lookup+0x8b/0x130 [obdclass] [Wed Feb 6 09:38:59 2019][1696439.017558] [] ? lu_object_find_at+0x1b3/0x280 [obdclass] [Wed Feb 6 09:38:59 2019][1696439.024809] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:38:59 2019][1696439.031963] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:38:59 2019][1696439.039062] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.045718] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.052820] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.059976] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:38:59 2019][1696439.066698] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:38:59 2019][1696439.073344] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:38:59 2019][1696439.080165] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.087064] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:38:59 2019][1696439.094382] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:38:59 2019][1696439.100957] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.108220] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.115935] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.122243] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.129345] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.137103] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:38:59 2019][1696439.144382] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.152243] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.159218] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:38:59 2019][1696439.165663] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:38:59 2019][1696439.171724] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.178195] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.185774] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1696439.190847] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1696439.197126] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1696439.203760] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1696439.210036] mdt00_002 D ffff9139f1ff2080 0 6169 2 0x00000080 [Wed Feb 6 09:38:59 2019][1696439.217329] Call Trace: [Wed Feb 6 09:38:59 2019][1696439.219972] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [Wed Feb 6 09:38:59 2019][1696439.227115] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1696439.232270] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 09:38:59 2019][1696439.238891] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 09:38:59 2019][1696439.245788] [] down_write+0x2d/0x3d [Wed Feb 6 09:38:59 2019][1696439.251120] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 09:38:59 2019][1696439.258009] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 09:38:59 2019][1696439.264908] [] ? ___slab_alloc+0x209/0x4f0 [Wed Feb 6 09:38:59 2019][1696439.270858] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:38:59 2019][1696439.278009] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:59 2019][1696439.283789] [] ? lu_buf_alloc+0x48/0x320 [obdclass] [Wed Feb 6 09:38:59 2019][1696439.290515] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:38:59 2019][1696439.298525] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:38:59 2019][1696439.305952] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:38:59 2019][1696439.313189] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:38:59 2019][1696439.319925] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:38:59 2019][1696439.327250] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:38:59 2019][1696439.333897] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:38:59 2019][1696439.340551] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.347193] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:38:59 2019][1696439.353823] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:38:59 2019][1696439.360483] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.367375] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:38:59 2019][1696439.374714] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:38:59 2019][1696439.381267] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.388545] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.396235] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.402547] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.409652] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.417411] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:38:59 2019][1696439.424670] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.432553] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.439519] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:38:59 2019][1696439.445980] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:38:59 2019][1696439.452024] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.458501] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.466092] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1696439.471156] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1696439.477431] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1696439.484062] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:38:59 2019][1696439.490331] mdt01_000 S ffff9139f1ff5140 0 6170 2 0x00000080 [Wed Feb 6 09:38:59 2019][1696439.497616] Call Trace: [Wed Feb 6 09:38:59 2019][1696439.500250] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:38:59 2019][1696439.507160] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.514921] [] schedule+0x29/0x70 [Wed Feb 6 09:38:59 2019][1696439.520097] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.527192] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.534268] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:38:59 2019][1696439.541425] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:38:59 2019][1696439.547209] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.554538] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:38:59 2019][1696439.561899] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.569920] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:38:59 2019][1696439.576643] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:38:59 2019][1696439.583618] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:38:59 2019][1696439.590350] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.598396] [] ? htable_lookup+0x8b/0x130 [obdclass] [Wed Feb 6 09:38:59 2019][1696439.605205] [] ? lu_object_find_at+0x1b3/0x280 [obdclass] [Wed Feb 6 09:38:59 2019][1696439.612443] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:38:59 2019][1696439.619608] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:38:59 2019][1696439.626695] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.633360] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.640466] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.647625] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:38:59 2019][1696439.654361] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:38:59 2019][1696439.660998] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:38:59 2019][1696439.667831] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.667968] LNet: Service thread pid 15239 was inactive for 1192.08s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 09:38:59 2019][1696439.667970] LNet: Skipped 4 previous similar messages [Wed Feb 6 09:38:59 2019][1696439.667972] Pid: 15239, comm: mdt01_052 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 09:38:59 2019][1696439.667972] Call Trace: [Wed Feb 6 09:38:59 2019][1696439.668029] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.668057] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.668074] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:38:59 2019][1696439.668085] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:38:59 2019][1696439.668095] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:38:59 2019][1696439.668106] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:38:59 2019][1696439.668116] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:38:59 2019][1696439.668141] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.668170] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.668214] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.668250] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.668282] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.668314] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.668318] [] kthread+0xd1/0xe0 [Wed Feb 6 09:38:59 2019][1696439.668322] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:38:59 2019][1696439.668347] [] 0xffffffffffffffff [Wed Feb 6 09:38:59 2019][1696439.668349] LustreError: dumping log to /tmp/lustre-log.1549474777.15239 [Wed Feb 6 09:38:59 2019][1696439.824030] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:38:59 2019][1696439.831347] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:38:59 2019][1696439.837918] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.845198] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.852903] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:38:59 2019][1696439.859219] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696439.866330] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696439.874087] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:00 2019][1696439.881359] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696439.889220] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696439.896213] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:00 2019][1696439.902660] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:00 2019][1696439.908709] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696439.915203] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696439.922781] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1696439.927858] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1696439.934135] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1696439.940769] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1696439.947041] mdt01_001 S ffff9139f1ff30c0 0 6171 2 0x00000080 [Wed Feb 6 09:39:00 2019][1696439.954329] Call Trace: [Wed Feb 6 09:39:00 2019][1696439.956966] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:39:00 2019][1696439.963886] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696439.971649] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1696439.976827] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696439.983921] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696439.991014] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:00 2019][1696439.998171] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:00 2019][1696440.003951] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.011290] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:39:00 2019][1696440.018633] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.026652] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:00 2019][1696440.033458] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:39:00 2019][1696440.040580] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:00 2019][1696440.047376] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.055416] [] ? htable_lookup+0x8b/0x130 [obdclass] [Wed Feb 6 09:39:00 2019][1696440.062227] [] ? lu_object_find_at+0x1b3/0x280 [obdclass] [Wed Feb 6 09:39:00 2019][1696440.069478] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:00 2019][1696440.076621] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:39:00 2019][1696440.083713] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.090380] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.097466] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.104630] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:39:00 2019][1696440.111340] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:00 2019][1696440.117974] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:39:00 2019][1696440.124802] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.131693] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:00 2019][1696440.139014] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:00 2019][1696440.145595] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.152850] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.160557] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.166859] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.173948] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.181767] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:00 2019][1696440.189030] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.196895] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.203874] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:00 2019][1696440.210317] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:00 2019][1696440.216361] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.222849] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.230427] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1696440.235488] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1696440.241778] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1696440.248402] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1696440.254678] mdt01_002 S ffff9139f61f1040 0 6172 2 0x00000080 [Wed Feb 6 09:39:00 2019][1696440.261965] Call Trace: [Wed Feb 6 09:39:00 2019][1696440.264597] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1696440.269775] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.276667] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:00 2019][1696440.282449] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.288932] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.296517] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1696440.301578] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1696440.307904] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1696440.314559] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1696440.320829] mdt02_000 S ffff9139f8946180 0 6173 2 0x00000080 [Wed Feb 6 09:39:00 2019][1696440.328134] Call Trace: [Wed Feb 6 09:39:00 2019][1696440.330775] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1696440.335948] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.342842] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:00 2019][1696440.348628] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.355111] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.362690] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1696440.367765] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1696440.374034] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1696440.380659] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1696440.386948] mdt02_001 S ffff9139f8942080 0 6174 2 0x00000080 [Wed Feb 6 09:39:00 2019][1696440.394231] Call Trace: [Wed Feb 6 09:39:00 2019][1696440.396864] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1696440.402054] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.408934] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:00 2019][1696440.414726] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.421220] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.428795] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1696440.433871] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1696440.440149] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1696440.446793] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1696440.453071] mdt02_002 S ffff9149d5f6e180 0 6175 2 0x00000080 [Wed Feb 6 09:39:00 2019][1696440.460360] Call Trace: [Wed Feb 6 09:39:00 2019][1696440.462990] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1696440.468169] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.475065] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:00 2019][1696440.480848] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.487324] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.494912] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1696440.499981] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1696440.506274] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1696440.512899] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1696440.519168] mdt03_000 S ffff9149d5f69040 0 6176 2 0x00000080 [Wed Feb 6 09:39:00 2019][1696440.526463] Call Trace: [Wed Feb 6 09:39:00 2019][1696440.529092] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1696440.534273] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.541167] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:00 2019][1696440.546954] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.553441] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.561016] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:00 2019][1696440.566091] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1696440.572371] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:00 2019][1696440.579005] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:00 2019][1696440.585284] mdt03_001 S ffff9149d5f6a080 0 6177 2 0x00000080 [Wed Feb 6 09:39:00 2019][1696440.592587] Call Trace: [Wed Feb 6 09:39:00 2019][1696440.595225] [] schedule+0x29/0x70 [Wed Feb 6 09:39:00 2019][1696440.600369] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:00 2019][1696440.606401] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:00 2019][1696440.612941] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [Wed Feb 6 09:39:00 2019][1696440.619831] [] osp_precreate_reserve+0x2e8/0x800 [osp] [Wed Feb 6 09:39:00 2019][1696440.626813] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:00 2019][1696440.632574] [] osp_declare_create+0x199/0x5b0 [osp] [Wed Feb 6 09:39:00 2019][1696440.639314] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Wed Feb 6 09:39:00 2019][1696440.646644] [] lod_sub_declare_create+0xdf/0x210 [lod] [Wed Feb 6 09:39:00 2019][1696440.653637] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [Wed Feb 6 09:39:00 2019][1696440.660876] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [Wed Feb 6 09:39:00 2019][1696440.668301] [] ? lod_qos_calc_weight+0x80/0x80 [lod] [Wed Feb 6 09:39:00 2019][1696440.675106] [] ? lod_obj_is_ost_use_skip_cb+0x30/0x30 [lod] [Wed Feb 6 09:39:00 2019][1696440.682529] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [Wed Feb 6 09:39:00 2019][1696440.689513] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:00 2019][1696440.696669] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:00 2019][1696440.702433] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:39:00 2019][1696440.710459] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:39:00 2019][1696440.717884] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:39:00 2019][1696440.725138] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:39:00 2019][1696440.731862] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:00 2019][1696440.739191] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:39:00 2019][1696440.745836] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:39:00 2019][1696440.752496] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.759133] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:00 2019][1696440.765758] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:39:00 2019][1696440.772420] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.779311] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:00 2019][1696440.786635] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:00 2019][1696440.793213] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.800476] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.808188] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.814495] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:00 2019][1696440.821588] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696440.829360] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:01 2019][1696440.836620] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696440.844524] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696440.851498] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:01 2019][1696440.857957] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:01 2019][1696440.864007] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696440.870498] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696440.878078] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1696440.883144] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696440.889440] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1696440.896065] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696440.902358] mdt03_002 S ffff9149d5f6c100 0 6178 2 0x00000080 [Wed Feb 6 09:39:01 2019][1696440.909636] Call Trace: [Wed Feb 6 09:39:01 2019][1696440.912287] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:39:01 2019][1696440.919193] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696440.926961] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1696440.932134] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696440.939212] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696440.946302] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:01 2019][1696440.953443] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:01 2019][1696440.959237] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696440.966556] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:39:01 2019][1696440.973933] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696440.981966] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:01 2019][1696440.988682] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:39:01 2019][1696440.995659] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:01 2019][1696441.002407] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.010430] [] ? lod_attr_get+0xab/0x130 [lod] [Wed Feb 6 09:39:01 2019][1696441.016725] [] ? mdd_la_get+0x84/0x1a0 [mdd] [Wed Feb 6 09:39:01 2019][1696441.022840] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:01 2019][1696441.030012] [] ? mdt_attr_get_complex+0x4ec/0x850 [mdt] [Wed Feb 6 09:39:01 2019][1696441.037080] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:39:01 2019][1696441.044179] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.050829] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.057935] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.065084] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:39:01 2019][1696441.071800] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:01 2019][1696441.078445] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:39:01 2019][1696441.085267] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.092166] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:01 2019][1696441.099488] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:01 2019][1696441.106091] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.113351] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.121063] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.127380] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.134482] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.142243] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:01 2019][1696441.149514] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.157381] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.164369] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:01 2019][1696441.170817] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:01 2019][1696441.176876] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.183360] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.190940] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1696441.196020] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.202300] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1696441.208934] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.215209] mdt_rdpg00_000 S ffff913aa9a35140 0 6179 2 0x00000080 [Wed Feb 6 09:39:01 2019][1696441.222508] Call Trace: [Wed Feb 6 09:39:01 2019][1696441.225141] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1696441.230322] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.237255] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:01 2019][1696441.243079] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.249573] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.257151] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1696441.262228] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.268505] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1696441.275141] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.281418] mdt_rdpg00_001 S ffff913aa9a32080 0 6180 2 0x00000080 [Wed Feb 6 09:39:01 2019][1696441.288725] Call Trace: [Wed Feb 6 09:39:01 2019][1696441.291355] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1696441.296533] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.303429] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:01 2019][1696441.309216] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.315694] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.323283] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1696441.328350] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.334631] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1696441.341265] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.347534] mdt_rdpg01_000 S ffff913aa9a34100 0 6181 2 0x00000080 [Wed Feb 6 09:39:01 2019][1696441.354831] Call Trace: [Wed Feb 6 09:39:01 2019][1696441.357470] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1696441.362646] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.369572] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:01 2019][1696441.375361] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.381843] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.389424] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1696441.394495] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.400764] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1696441.407402] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.413668] mdt_rdpg01_001 S ffff913aa9a36180 0 6182 2 0x00000080 [Wed Feb 6 09:39:01 2019][1696441.420939] Call Trace: [Wed Feb 6 09:39:01 2019][1696441.423585] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1696441.428762] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.435656] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:01 2019][1696441.441433] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.447906] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.455493] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1696441.460551] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.466828] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1696441.473461] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.479730] mdt_rdpg02_000 S ffff9159f4ee8000 0 6183 2 0x00000080 [Wed Feb 6 09:39:01 2019][1696441.487023] Call Trace: [Wed Feb 6 09:39:01 2019][1696441.489656] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1696441.494829] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.501766] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:01 2019][1696441.507549] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.514036] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.521628] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1696441.526688] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.532979] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1696441.539601] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.545889] mdt_rdpg02_001 S ffff9159f4ee9040 0 6184 2 0x00000080 [Wed Feb 6 09:39:01 2019][1696441.553172] Call Trace: [Wed Feb 6 09:39:01 2019][1696441.555805] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1696441.560977] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.567868] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:01 2019][1696441.573646] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.580123] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.587708] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1696441.592770] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.599059] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1696441.605675] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.611955] mdt_rdpg03_000 S ffff9159f4eea080 0 6185 2 0x00000080 [Wed Feb 6 09:39:01 2019][1696441.619245] Call Trace: [Wed Feb 6 09:39:01 2019][1696441.621878] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1696441.627058] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.634035] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:01 2019][1696441.639819] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.646309] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.653883] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1696441.658958] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.665226] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1696441.671864] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.678145] mdt_rdpg03_001 S ffff9159f4eeb0c0 0 6186 2 0x00000080 [Wed Feb 6 09:39:01 2019][1696441.685426] Call Trace: [Wed Feb 6 09:39:01 2019][1696441.688068] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1696441.693242] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.700134] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:01 2019][1696441.705911] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.712385] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.719976] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1696441.725040] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.731315] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:01 2019][1696441.737949] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.744224] mdt_attr00_000 S ffff9159f4eec100 0 6187 2 0x00000080 [Wed Feb 6 09:39:01 2019][1696441.751520] Call Trace: [Wed Feb 6 09:39:01 2019][1696441.754147] [] schedule+0x29/0x70 [Wed Feb 6 09:39:01 2019][1696441.759315] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.766218] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:01 2019][1696441.771999] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.778517] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:01 2019][1696441.786106] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:01 2019][1696441.791185] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:01 2019][1696441.797526] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696441.804297] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696441.810584] mdt_attr00_001 S ffff9159f4eed140 0 6188 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696441.817854] Call Trace: [Wed Feb 6 09:39:02 2019][1696441.820484] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696441.825667] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696441.832566] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696441.838356] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696441.844835] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696441.852413] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696441.857489] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696441.863759] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696441.870380] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696441.876673] mdt_attr01_000 S ffff9159f4eee180 0 6189 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696441.883952] Call Trace: [Wed Feb 6 09:39:02 2019][1696441.886590] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696441.891764] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696441.898660] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696441.904448] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696441.910925] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696441.918514] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696441.923571] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696441.929849] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696441.936490] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696441.942841] mdt_attr01_001 S ffff9149fa3a30c0 0 6190 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696441.950198] Call Trace: [Wed Feb 6 09:39:02 2019][1696441.952873] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696441.958084] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696441.965064] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696441.970940] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696441.977501] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696441.985095] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696441.990156] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696441.996433] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696442.003070] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.009348] mdt_attr02_000 S ffff9149fa3a6180 0 6191 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696442.016625] Call Trace: [Wed Feb 6 09:39:02 2019][1696442.019274] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696442.024449] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.031343] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696442.037129] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.043618] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.051193] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696442.056267] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.062546] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696442.069182] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.075455] mdt_attr02_001 S ffff9149fa3a0000 0 6192 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696442.082760] Call Trace: [Wed Feb 6 09:39:02 2019][1696442.085388] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696442.090568] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.097462] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696442.103243] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.109851] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.117532] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696442.122672] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.129057] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696442.135746] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.142026] mdt_attr03_000 S ffff9149fa3a1040 0 6193 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696442.149329] Call Trace: [Wed Feb 6 09:39:02 2019][1696442.151956] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696442.157126] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.164023] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696442.169809] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.176284] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.183875] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696442.188934] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.195222] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696442.201839] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.208124] mdt_attr03_001 S ffff9149fa3a5140 0 6194 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696442.215396] Call Trace: [Wed Feb 6 09:39:02 2019][1696442.218040] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696442.223212] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.230109] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696442.235897] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.242378] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.249961] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696442.255026] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.261317] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696442.267941] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.274217] mdt_out00_000 S ffff9149b5232080 0 6195 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696442.281511] Call Trace: [Wed Feb 6 09:39:02 2019][1696442.284147] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696442.289326] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.296227] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696442.302012] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.308484] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.316071] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696442.321129] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.327405] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696442.334049] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.340326] mdt_out00_001 S ffff9149b5231040 0 6196 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696442.347627] Call Trace: [Wed Feb 6 09:39:02 2019][1696442.350257] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696442.355433] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.362330] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696442.368116] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.374586] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.382179] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696442.387243] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.393518] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696442.400148] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.406419] mdt_out01_000 S ffff9149b5236180 0 6197 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696442.413710] Call Trace: [Wed Feb 6 09:39:02 2019][1696442.416342] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696442.421520] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.428416] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696442.434200] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.440675] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.448261] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696442.453326] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.459616] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696442.466239] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.472511] mdt_out01_001 S ffff9149b5235140 0 6198 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696442.479806] Call Trace: [Wed Feb 6 09:39:02 2019][1696442.482439] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696442.487615] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.494508] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696442.500296] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.506783] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.514355] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696442.519433] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.525699] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696442.532319] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.538605] mdt_out02_000 S ffff9149b52330c0 0 6199 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696442.545891] Call Trace: [Wed Feb 6 09:39:02 2019][1696442.548534] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696442.553713] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.560607] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696442.566390] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.572876] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.580450] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696442.585525] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.591794] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696442.598417] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.604705] mdt_out02_001 S ffff9149b5234100 0 6200 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696442.611987] Call Trace: [Wed Feb 6 09:39:02 2019][1696442.614625] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696442.619804] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.626697] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696442.632483] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.638970] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.646544] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696442.651624] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.657900] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696442.664536] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.670826] mdt_out03_000 S ffff9149b5230000 0 6201 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696442.678107] Call Trace: [Wed Feb 6 09:39:02 2019][1696442.680758] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696442.685932] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.692825] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696442.698610] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.705098] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.712676] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696442.717751] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.724026] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696442.730667] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.736942] mdt_out03_001 S ffff9149fa32b0c0 0 6202 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696442.744235] Call Trace: [Wed Feb 6 09:39:02 2019][1696442.746870] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696442.752057] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.758947] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696442.764744] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.771218] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.778810] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696442.783874] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.790149] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696442.796786] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.803057] mdt_seqs_0000 S ffff9149fa32a080 0 6203 2 0x00000080 [Wed Feb 6 09:39:02 2019][1696442.810343] Call Trace: [Wed Feb 6 09:39:02 2019][1696442.812974] [] schedule+0x29/0x70 [Wed Feb 6 09:39:02 2019][1696442.818150] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.825051] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:02 2019][1696442.830833] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.837314] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:02 2019][1696442.844905] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:02 2019][1696442.849968] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:02 2019][1696442.856241] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:02 2019][1696442.862878] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696442.869155] mdt_seqs_0001 S ffff9149fa32e180 0 6204 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696442.876455] Call Trace: [Wed Feb 6 09:39:03 2019][1696442.879084] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1696442.884261] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696442.891155] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:03 2019][1696442.896931] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696442.903408] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696442.910992] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1696442.916054] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696442.922328] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1696442.928963] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696442.935240] mdt_seqm_0000 S ffff9149fa329040 0 6205 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696442.942529] Call Trace: [Wed Feb 6 09:39:03 2019][1696442.945161] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1696442.950338] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696442.957232] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:03 2019][1696442.963008] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696442.969482] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696442.977074] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1696442.982133] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696442.988411] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1696442.995053] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.001328] mdt_seqm_0001 S ffff9149fa32d140 0 6206 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696443.008625] Call Trace: [Wed Feb 6 09:39:03 2019][1696443.011255] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1696443.016432] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.023330] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:03 2019][1696443.029110] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.035585] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.043171] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1696443.048232] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.054506] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1696443.061140] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.067419] mdt_fld_0000 S ffff9149f865a080 0 6207 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696443.074717] Call Trace: [Wed Feb 6 09:39:03 2019][1696443.077347] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1696443.082521] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.089422] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:03 2019][1696443.095205] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.101693] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.109272] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1696443.114332] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.120621] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1696443.127247] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.133534] mdt_fld_0001 S ffff9149f865c100 0 6208 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696443.140801] Call Trace: [Wed Feb 6 09:39:03 2019][1696443.143445] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1696443.148619] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.155511] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:03 2019][1696443.161287] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.167765] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.175350] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1696443.180410] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.186684] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1696443.193320] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.199598] mdt_io00_000 S ffff9149f865d140 0 6209 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696443.206897] Call Trace: [Wed Feb 6 09:39:03 2019][1696443.209526] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1696443.214706] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.221600] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:03 2019][1696443.227384] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.233874] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.241451] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1696443.246512] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.252801] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1696443.259423] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.265709] mdt_io00_001 S ffff9149f865b0c0 0 6210 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696443.272982] Call Trace: [Wed Feb 6 09:39:03 2019][1696443.275612] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1696443.280788] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.287687] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:03 2019][1696443.293470] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.299946] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.307533] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1696443.312592] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.318872] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1696443.325490] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.331762] mdt_io00_002 S ffff9149f8658000 0 6211 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696443.339055] Call Trace: [Wed Feb 6 09:39:03 2019][1696443.341691] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1696443.346868] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.353758] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:03 2019][1696443.359537] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.366029] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.373599] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1696443.378662] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.384950] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1696443.391572] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.397840] mdt_io01_000 S ffff9159b55c0000 0 6212 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696443.405138] Call Trace: [Wed Feb 6 09:39:03 2019][1696443.407776] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1696443.412951] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.419848] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:03 2019][1696443.425632] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.432108] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.439702] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1696443.444763] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.451061] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1696443.457681] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.463956] mdt_io01_001 S ffff9159b55c1040 0 6213 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696443.471259] Call Trace: [Wed Feb 6 09:39:03 2019][1696443.473893] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1696443.479075] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.485970] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:03 2019][1696443.491748] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.498222] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.505815] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1696443.510878] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.517152] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1696443.523788] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.530066] mdt_io01_002 S ffff9159b55c2080 0 6214 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696443.537366] Call Trace: [Wed Feb 6 09:39:03 2019][1696443.539993] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1696443.545175] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.552068] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:03 2019][1696443.557853] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.564344] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.571920] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1696443.576981] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.583268] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1696443.589892] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.596178] mdt_io02_000 S ffff9159b55c30c0 0 6215 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696443.603454] Call Trace: [Wed Feb 6 09:39:03 2019][1696443.606101] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1696443.611274] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.618170] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:03 2019][1696443.623953] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.630432] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.638017] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1696443.643082] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.649370] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1696443.655997] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.662279] mdt_io02_001 S ffff9159b55c4100 0 6216 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696443.669561] Call Trace: [Wed Feb 6 09:39:03 2019][1696443.672204] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1696443.677382] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.684279] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:03 2019][1696443.690061] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.696554] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.704131] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1696443.709191] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.715478] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1696443.722096] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.728371] mdt_io02_002 S ffff9159b55c5140 0 6217 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696443.735676] Call Trace: [Wed Feb 6 09:39:03 2019][1696443.738313] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1696443.743493] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.750394] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:03 2019][1696443.756175] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.762669] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.770243] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1696443.775306] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.781593] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1696443.788220] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.794506] mdt_io03_000 S ffff9159b55c6180 0 6218 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696443.801773] Call Trace: [Wed Feb 6 09:39:03 2019][1696443.804416] [] schedule+0x29/0x70 [Wed Feb 6 09:39:03 2019][1696443.809589] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.816488] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:03 2019][1696443.822275] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.828757] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:03 2019][1696443.836352] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:03 2019][1696443.841410] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.847695] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:03 2019][1696443.854316] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:03 2019][1696443.860592] mdt_io03_001 S ffff9159a7200000 0 6219 2 0x00000080 [Wed Feb 6 09:39:03 2019][1696443.867889] Call Trace: [Wed Feb 6 09:39:03 2019][1696443.870527] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1696443.875705] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:04 2019][1696443.882606] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:04 2019][1696443.888391] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:04 2019][1696443.894863] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:04 2019][1696443.902450] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1696443.907504] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696443.913775] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1696443.920414] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696443.926690] mdt_io03_002 S ffff9159a7201040 0 6220 2 0x00000080 [Wed Feb 6 09:39:04 2019][1696443.933974] Call Trace: [Wed Feb 6 09:39:04 2019][1696443.936621] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1696443.941798] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:04 2019][1696443.948692] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:04 2019][1696443.954473] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:04 2019][1696443.960964] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:04 2019][1696443.968542] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1696443.973602] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696443.979892] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1696443.986513] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696443.992807] qmt_reba_fir-QM S ffff9159a72030c0 0 6232 2 0x00000080 [Wed Feb 6 09:39:04 2019][1696444.000084] Call Trace: [Wed Feb 6 09:39:04 2019][1696444.002717] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:04 2019][1696444.008813] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1696444.013984] [] qmt_reba_thread+0x9e0/0xda0 [lquota] [Wed Feb 6 09:39:04 2019][1696444.020690] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:04 2019][1696444.026453] [] ? qmt_glimpse_lock.isra.19+0x690/0x690 [lquota] [Wed Feb 6 09:39:04 2019][1696444.034129] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1696444.039195] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.045482] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1696444.052106] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.058399] osp_up2-0 S ffff9159a7204100 0 6241 2 0x00000080 [Wed Feb 6 09:39:04 2019][1696444.065684] Call Trace: [Wed Feb 6 09:39:04 2019][1696444.068339] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Wed Feb 6 09:39:04 2019][1696444.075406] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Wed Feb 6 09:39:04 2019][1696444.082390] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1696444.087543] [] osp_send_update_thread+0x2f8/0x590 [osp] [Wed Feb 6 09:39:04 2019][1696444.094614] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:04 2019][1696444.100379] [] ? osp_invalidate_request+0x390/0x390 [osp] [Wed Feb 6 09:39:04 2019][1696444.107622] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1696444.112676] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.118965] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1696444.125581] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.131852] osp-pre-2-0 S ffff9159a7205140 0 6242 2 0x00000080 [Wed Feb 6 09:39:04 2019][1696444.139150] Call Trace: [Wed Feb 6 09:39:04 2019][1696444.141781] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:04 2019][1696444.147193] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1696444.152358] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:04 2019][1696444.159324] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:04 2019][1696444.165096] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:04 2019][1696444.171805] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1696444.176882] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.183161] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1696444.189796] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.196074] osp_up1-0 S ffff9159a7206180 0 6244 2 0x00000080 [Wed Feb 6 09:39:04 2019][1696444.203363] Call Trace: [Wed Feb 6 09:39:04 2019][1696444.206021] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Wed Feb 6 09:39:04 2019][1696444.213083] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Wed Feb 6 09:39:04 2019][1696444.220077] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1696444.225234] [] osp_send_update_thread+0x2f8/0x590 [osp] [Wed Feb 6 09:39:04 2019][1696444.232292] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:04 2019][1696444.238060] [] ? osp_invalidate_request+0x390/0x390 [osp] [Wed Feb 6 09:39:04 2019][1696444.245292] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1696444.250367] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.256643] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1696444.263281] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.269558] osp-pre-1-0 S ffff91599d648000 0 6245 2 0x00000080 [Wed Feb 6 09:39:04 2019][1696444.276845] Call Trace: [Wed Feb 6 09:39:04 2019][1696444.279496] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:04 2019][1696444.284899] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1696444.290066] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:04 2019][1696444.297031] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:04 2019][1696444.302803] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:04 2019][1696444.309510] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1696444.314577] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.320864] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1696444.327489] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.333776] osp_up3-0 S ffff91599d649040 0 6246 2 0x00000080 [Wed Feb 6 09:39:04 2019][1696444.341048] Call Trace: [Wed Feb 6 09:39:04 2019][1696444.343730] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Wed Feb 6 09:39:04 2019][1696444.350791] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Wed Feb 6 09:39:04 2019][1696444.357775] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1696444.362933] [] osp_send_update_thread+0x2f8/0x590 [osp] [Wed Feb 6 09:39:04 2019][1696444.370006] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:04 2019][1696444.375767] [] ? osp_invalidate_request+0x390/0x390 [osp] [Wed Feb 6 09:39:04 2019][1696444.383002] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1696444.388060] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.394349] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1696444.400962] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.407229] osp-pre-3-0 S ffff91599d64a080 0 6247 2 0x00000080 [Wed Feb 6 09:39:04 2019][1696444.414528] Call Trace: [Wed Feb 6 09:39:04 2019][1696444.417163] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:04 2019][1696444.422570] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1696444.427738] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:04 2019][1696444.434708] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:04 2019][1696444.440466] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:04 2019][1696444.447188] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1696444.452244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.458538] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1696444.465163] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.471436] osp-syn-12-0 S ffff91599d64b0c0 0 6248 2 0x00000080 [Wed Feb 6 09:39:04 2019][1696444.478736] Call Trace: [Wed Feb 6 09:39:04 2019][1696444.481366] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:04 2019][1696444.487468] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1696444.492636] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:04 2019][1696444.499947] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:04 2019][1696444.505724] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.513058] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:04 2019][1696444.519606] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.526851] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.534116] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.541447] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.549225] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.556461] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.564238] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.571908] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:04 2019][1696444.578365] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:04 2019][1696444.584406] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:04 2019][1696444.590961] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.597772] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:04 2019][1696444.604219] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:04 2019][1696444.609907] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:04 2019][1696444.617483] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1696444.622543] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.628834] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1696444.635455] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.641732] osp-pre-12-0 S ffff91599d64c100 0 6249 2 0x00000080 [Wed Feb 6 09:39:04 2019][1696444.649030] Call Trace: [Wed Feb 6 09:39:04 2019][1696444.651660] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:04 2019][1696444.657062] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1696444.662233] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:04 2019][1696444.669199] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:04 2019][1696444.674958] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:04 2019][1696444.681684] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1696444.686745] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.693033] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1696444.699651] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.705926] osp-syn-13-0 S ffff91599d64d140 0 6250 2 0x00000080 [Wed Feb 6 09:39:04 2019][1696444.713230] Call Trace: [Wed Feb 6 09:39:04 2019][1696444.715860] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:04 2019][1696444.721963] [] schedule+0x29/0x70 [Wed Feb 6 09:39:04 2019][1696444.727127] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:04 2019][1696444.734440] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:04 2019][1696444.740224] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.747544] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:04 2019][1696444.754101] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.761357] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.768601] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.775929] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.783705] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.790947] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.798728] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.806383] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:04 2019][1696444.812851] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:04 2019][1696444.818867] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:04 2019][1696444.825414] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:04 2019][1696444.832235] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:04 2019][1696444.838682] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:04 2019][1696444.844351] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:04 2019][1696444.851931] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:04 2019][1696444.856989] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:04 2019][1696444.863281] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:04 2019][1696444.869902] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696444.876176] osp-pre-13-0 S ffff91599d64e180 0 6251 2 0x00000080 [Wed Feb 6 09:39:05 2019][1696444.883461] Call Trace: [Wed Feb 6 09:39:05 2019][1696444.886098] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:05 2019][1696444.891512] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1696444.896678] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:05 2019][1696444.903647] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:05 2019][1696444.909415] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:05 2019][1696444.916123] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1696444.921198] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696444.927472] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1696444.934100] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696444.940391] osp-syn-39-0 S ffff9159f6e08000 0 6252 2 0x00000080 [Wed Feb 6 09:39:05 2019][1696444.947688] Call Trace: [Wed Feb 6 09:39:05 2019][1696444.950338] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:05 2019][1696444.956437] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1696444.961603] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:05 2019][1696444.968920] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:05 2019][1696444.974699] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:05 2019][1696444.982017] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:05 2019][1696444.988582] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:05 2019][1696444.995825] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.003084] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.010411] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.018191] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.025428] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.033209] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.040866] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:05 2019][1696445.047319] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:05 2019][1696445.053347] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:05 2019][1696445.059899] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.066709] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:05 2019][1696445.073160] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:05 2019][1696445.078847] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:05 2019][1696445.086419] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1696445.091483] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696445.097769] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1696445.104389] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696445.110677] osp-pre-39-0 S ffff9159f6e09040 0 6253 2 0x00000080 [Wed Feb 6 09:39:05 2019][1696445.117970] Call Trace: [Wed Feb 6 09:39:05 2019][1696445.120613] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:05 2019][1696445.126018] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1696445.131184] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:05 2019][1696445.138150] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:05 2019][1696445.143900] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:05 2019][1696445.150618] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1696445.155677] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696445.161952] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1696445.168588] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696445.174855] osp-syn-46-0 S ffff9159f6e0a080 0 6254 2 0x00000080 [Wed Feb 6 09:39:05 2019][1696445.182135] Call Trace: [Wed Feb 6 09:39:05 2019][1696445.184779] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:05 2019][1696445.190883] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1696445.196050] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:05 2019][1696445.203364] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:05 2019][1696445.209144] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.216460] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:05 2019][1696445.223019] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.230260] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.237500] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.244847] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.252606] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.259868] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.267626] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.275306] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:05 2019][1696445.281756] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:05 2019][1696445.287776] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:05 2019][1696445.294343] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.301141] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:05 2019][1696445.307605] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:05 2019][1696445.313282] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:05 2019][1696445.320869] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1696445.325931] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696445.332221] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1696445.338844] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696445.345131] osp-pre-46-0 S ffff9159f6e0b0c0 0 6255 2 0x00000080 [Wed Feb 6 09:39:05 2019][1696445.352411] Call Trace: [Wed Feb 6 09:39:05 2019][1696445.355040] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:05 2019][1696445.360468] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1696445.365619] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:05 2019][1696445.372587] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:05 2019][1696445.378357] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:05 2019][1696445.385064] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1696445.390136] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696445.396407] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1696445.403045] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696445.409320] osp-syn-10-0 S ffff9159f6e0c100 0 6256 2 0x00000080 [Wed Feb 6 09:39:05 2019][1696445.416597] Call Trace: [Wed Feb 6 09:39:05 2019][1696445.419245] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:05 2019][1696445.425348] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1696445.430516] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:05 2019][1696445.437830] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:05 2019][1696445.443610] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.450929] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:05 2019][1696445.457496] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.464737] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.471998] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.479323] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.487100] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.494338] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.502120] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.509784] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:05 2019][1696445.516245] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:05 2019][1696445.522268] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:05 2019][1696445.528834] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.535636] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:05 2019][1696445.542097] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:05 2019][1696445.547771] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:05 2019][1696445.555344] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1696445.560417] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696445.566690] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1696445.573323] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696445.579590] osp-pre-10-0 S ffff9159f6e0d140 0 6257 2 0x00000080 [Wed Feb 6 09:39:05 2019][1696445.586894] Call Trace: [Wed Feb 6 09:39:05 2019][1696445.589522] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:05 2019][1696445.594932] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1696445.600097] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:05 2019][1696445.607069] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:05 2019][1696445.612826] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:05 2019][1696445.619545] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1696445.624612] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696445.630895] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1696445.637510] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696445.643790] osp-syn-11-0 S ffff9159f6e0e180 0 6258 2 0x00000080 [Wed Feb 6 09:39:05 2019][1696445.651060] Call Trace: [Wed Feb 6 09:39:05 2019][1696445.653692] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:05 2019][1696445.659798] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1696445.664978] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:05 2019][1696445.672295] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:05 2019][1696445.678079] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.685394] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:05 2019][1696445.691955] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.699199] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.706453] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.713783] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.721547] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.728801] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.736562] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.744240] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:05 2019][1696445.750682] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:05 2019][1696445.756697] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:05 2019][1696445.763268] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:05 2019][1696445.770072] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:05 2019][1696445.776520] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:05 2019][1696445.782207] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:05 2019][1696445.789781] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1696445.794841] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696445.801131] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1696445.807749] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696445.814025] osp-pre-11-0 S ffff9159ea2b8000 0 6259 2 0x00000080 [Wed Feb 6 09:39:05 2019][1696445.821327] Call Trace: [Wed Feb 6 09:39:05 2019][1696445.823962] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:05 2019][1696445.829371] [] schedule+0x29/0x70 [Wed Feb 6 09:39:05 2019][1696445.834537] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:05 2019][1696445.841503] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:05 2019][1696445.847266] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:05 2019][1696445.853980] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:05 2019][1696445.859038] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:05 2019][1696445.865328] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:05 2019][1696445.871949] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696445.878219] osp-syn-14-0 S ffff9159ea2b9040 0 6260 2 0x00000080 [Wed Feb 6 09:39:06 2019][1696445.885519] Call Trace: [Wed Feb 6 09:39:06 2019][1696445.888147] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:06 2019][1696445.894243] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1696445.899409] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:06 2019][1696445.906723] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:06 2019][1696445.912494] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:06 2019][1696445.919827] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:06 2019][1696445.926377] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:06 2019][1696445.933621] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:06 2019][1696445.940876] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:06 2019][1696445.948204] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:06 2019][1696445.955965] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:06 2019][1696445.963225] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:06 2019][1696445.970985] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:06 2019][1696445.978639] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:06 2019][1696445.985101] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:06 2019][1696445.991117] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:06 2019][1696445.997674] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.004489] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:06 2019][1696446.010933] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:06 2019][1696446.016626] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:06 2019][1696446.024202] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1696446.029264] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.035552] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1696446.042172] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.048446] osp-pre-14-0 S ffff9159ea2ba080 0 6261 2 0x00000080 [Wed Feb 6 09:39:06 2019][1696446.055735] Call Trace: [Wed Feb 6 09:39:06 2019][1696446.058370] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:06 2019][1696446.063781] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1696446.068945] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:06 2019][1696446.075912] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:06 2019][1696446.081680] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:06 2019][1696446.088392] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1696446.093464] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.099737] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1696446.106371] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.112642] osp-syn-26-0 S ffff9159ea2bb0c0 0 6262 2 0x00000080 [Wed Feb 6 09:39:06 2019][1696446.119933] Call Trace: [Wed Feb 6 09:39:06 2019][1696446.122562] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:06 2019][1696446.128668] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1696446.133831] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:06 2019][1696446.141147] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:06 2019][1696446.146934] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.154251] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:06 2019][1696446.160817] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.168064] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.175306] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.182649] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.190416] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.197662] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.205442] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.213098] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:06 2019][1696446.219561] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:06 2019][1696446.225586] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:06 2019][1696446.232157] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.238958] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:06 2019][1696446.245425] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:06 2019][1696446.251101] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:06 2019][1696446.258693] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1696446.263758] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.270043] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1696446.276664] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.282950] osp-pre-26-0 S ffff9159ea2bc100 0 6263 2 0x00000080 [Wed Feb 6 09:39:06 2019][1696446.290233] Call Trace: [Wed Feb 6 09:39:06 2019][1696446.292871] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:06 2019][1696446.298292] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1696446.303445] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:06 2019][1696446.310409] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:06 2019][1696446.316186] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:06 2019][1696446.322888] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1696446.327955] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.334241] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1696446.340868] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.347155] osp-syn-31-0 S ffff9159ea2bd140 0 6264 2 0x00000080 [Wed Feb 6 09:39:06 2019][1696446.354433] Call Trace: [Wed Feb 6 09:39:06 2019][1696446.357062] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:06 2019][1696446.363176] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1696446.368325] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:06 2019][1696446.375636] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:06 2019][1696446.381420] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.388738] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:06 2019][1696446.395296] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.402554] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.409802] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.417125] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:06 2019][1696446.424124] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.431888] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.439153] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.446915] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.454593] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:06 2019][1696446.461041] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:06 2019][1696446.467060] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:06 2019][1696446.473634] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.480436] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:06 2019][1696446.486884] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:06 2019][1696446.492571] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:06 2019][1696446.500143] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1696446.505220] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.511494] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1696446.518115] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.524404] osp-pre-31-0 S ffff9159ea2be180 0 6265 2 0x00000080 [Wed Feb 6 09:39:06 2019][1696446.531677] Call Trace: [Wed Feb 6 09:39:06 2019][1696446.534308] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:06 2019][1696446.539719] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1696446.544884] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:06 2019][1696446.551856] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:06 2019][1696446.557613] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:06 2019][1696446.564340] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1696446.569404] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.575678] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1696446.582311] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.588579] osp-syn-42-0 S ffff9159ff308000 0 6266 2 0x00000080 [Wed Feb 6 09:39:06 2019][1696446.595848] Call Trace: [Wed Feb 6 09:39:06 2019][1696446.598492] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:06 2019][1696446.604591] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1696446.609759] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:06 2019][1696446.617073] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:06 2019][1696446.622864] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.630187] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:06 2019][1696446.636750] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.643990] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.651246] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.658578] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.666372] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.673615] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.681379] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.689057] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:06 2019][1696446.695506] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:06 2019][1696446.701535] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:06 2019][1696446.708091] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.714907] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:06 2019][1696446.721349] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:06 2019][1696446.727039] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:06 2019][1696446.734612] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1696446.739676] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.745962] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1696446.752585] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.758876] osp-pre-42-0 S ffff9159ff309040 0 6267 2 0x00000080 [Wed Feb 6 09:39:06 2019][1696446.766152] Call Trace: [Wed Feb 6 09:39:06 2019][1696446.768785] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:06 2019][1696446.774193] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1696446.779362] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:06 2019][1696446.786330] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:06 2019][1696446.792098] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:06 2019][1696446.798806] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:06 2019][1696446.803884] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.810158] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:06 2019][1696446.816795] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:06 2019][1696446.823062] osp-syn-9-0 S ffff9159ff30a080 0 6268 2 0x00000080 [Wed Feb 6 09:39:06 2019][1696446.830331] Call Trace: [Wed Feb 6 09:39:06 2019][1696446.832965] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:06 2019][1696446.839079] [] schedule+0x29/0x70 [Wed Feb 6 09:39:06 2019][1696446.844237] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:06 2019][1696446.851555] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:06 2019][1696446.857340] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:06 2019][1696446.864663] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:06 2019][1696446.871230] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:07 2019][1696446.878471] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:07 2019][1696446.885730] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:07 2019][1696446.893057] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:07 2019][1696446.900837] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:07 2019][1696446.908078] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:07 2019][1696446.915856] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:07 2019][1696446.923516] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:07 2019][1696446.929978] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:07 2019][1696446.935999] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:07 2019][1696446.942565] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:07 2019][1696446.949361] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:07 2019][1696446.955806] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:07 2019][1696446.961492] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:07 2019][1696446.969068] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1696446.974132] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696446.980419] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1696446.987044] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696446.993333] osp-pre-9-0 S ffff9159ff30b0c0 0 6269 2 0x00000080 [Wed Feb 6 09:39:07 2019][1696447.000617] Call Trace: [Wed Feb 6 09:39:07 2019][1696447.003260] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:07 2019][1696447.008664] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1696447.013831] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:07 2019][1696447.020805] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:07 2019][1696447.026562] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:07 2019][1696447.033286] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1696447.038344] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696447.044627] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1696447.051247] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696447.057537] osp-syn-2-0 S ffff9159ff30c100 0 6270 2 0x00000080 [Wed Feb 6 09:39:07 2019][1696447.064823] Call Trace: [Wed Feb 6 09:39:07 2019][1696447.067465] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:07 2019][1696447.073566] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1696447.078733] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:07 2019][1696447.086046] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:07 2019][1696447.091827] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.099145] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:07 2019][1696447.105713] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.112952] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.120195] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.127539] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.135304] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.142551] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.150331] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.158001] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:07 2019][1696447.164458] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:07 2019][1696447.170474] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:07 2019][1696447.177030] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.183833] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:07 2019][1696447.190292] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:07 2019][1696447.195964] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:07 2019][1696447.203555] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1696447.208614] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696447.214902] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1696447.221524] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696447.227812] osp-pre-2-0 S ffff9159ff30d140 0 6271 2 0x00000080 [Wed Feb 6 09:39:07 2019][1696447.235084] Call Trace: [Wed Feb 6 09:39:07 2019][1696447.237711] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:07 2019][1696447.243112] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1696447.248283] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:07 2019][1696447.255251] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:07 2019][1696447.261009] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:07 2019][1696447.267735] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1696447.272801] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696447.279088] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1696447.285712] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696447.291998] osp-syn-43-0 S ffff9159ff30e180 0 6272 2 0x00000080 [Wed Feb 6 09:39:07 2019][1696447.299270] Call Trace: [Wed Feb 6 09:39:07 2019][1696447.301902] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:07 2019][1696447.308003] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1696447.313173] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:07 2019][1696447.320483] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:07 2019][1696447.326270] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.333581] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:07 2019][1696447.340128] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.347391] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.354631] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.361955] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:07 2019][1696447.368952] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.376713] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.383970] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.391733] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.399412] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:07 2019][1696447.405859] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:07 2019][1696447.411893] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:07 2019][1696447.418443] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.425248] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:07 2019][1696447.431713] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:07 2019][1696447.437389] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:07 2019][1696447.444982] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1696447.450044] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696447.456333] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1696447.462951] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696447.469225] osp-pre-43-0 S ffff9159a9298000 0 6273 2 0x00000080 [Wed Feb 6 09:39:07 2019][1696447.476525] Call Trace: [Wed Feb 6 09:39:07 2019][1696447.479155] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:07 2019][1696447.484563] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1696447.489731] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:07 2019][1696447.496696] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:07 2019][1696447.502458] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:07 2019][1696447.509178] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1696447.514242] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696447.520517] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1696447.527157] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696447.533429] osp-syn-19-0 S ffff9159a9299040 0 6274 2 0x00000080 [Wed Feb 6 09:39:07 2019][1696447.540721] Call Trace: [Wed Feb 6 09:39:07 2019][1696447.543350] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:07 2019][1696447.549456] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1696447.554622] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:07 2019][1696447.561935] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:07 2019][1696447.567721] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.575042] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:07 2019][1696447.581601] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.588842] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.596086] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.603439] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.611203] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.618450] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.626227] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.633885] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:07 2019][1696447.640349] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:07 2019][1696447.646362] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:07 2019][1696447.652910] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.659724] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:07 2019][1696447.666174] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:07 2019][1696447.671858] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:07 2019][1696447.679434] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1696447.684509] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696447.690788] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1696447.697417] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696447.703688] osp-pre-19-0 S ffff9159a929a080 0 6275 2 0x00000080 [Wed Feb 6 09:39:07 2019][1696447.710982] Call Trace: [Wed Feb 6 09:39:07 2019][1696447.713614] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:07 2019][1696447.719024] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1696447.724190] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:07 2019][1696447.731163] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:07 2019][1696447.736935] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:07 2019][1696447.743645] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:07 2019][1696447.748707] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696447.754996] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:07 2019][1696447.761620] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:07 2019][1696447.767908] osp-syn-18-0 S ffff9159a929b0c0 0 6276 2 0x00000080 [Wed Feb 6 09:39:07 2019][1696447.775186] Call Trace: [Wed Feb 6 09:39:07 2019][1696447.777834] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:07 2019][1696447.783937] [] schedule+0x29/0x70 [Wed Feb 6 09:39:07 2019][1696447.789100] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:07 2019][1696447.796416] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:07 2019][1696447.802206] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.809528] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:07 2019][1696447.816082] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.823340] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.830580] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.837918] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:07 2019][1696447.844896] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.852679] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.859919] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.867684] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:07 2019][1696447.875361] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:08 2019][1696447.881805] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:08 2019][1696447.887837] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:08 2019][1696447.894386] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:08 2019][1696447.901202] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:08 2019][1696447.907651] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:08 2019][1696447.913335] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:08 2019][1696447.920910] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1696447.925986] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696447.932260] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1696447.938899] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696447.945177] osp-pre-18-0 S ffff9159a929c100 0 6277 2 0x00000080 [Wed Feb 6 09:39:08 2019][1696447.952467] Call Trace: [Wed Feb 6 09:39:08 2019][1696447.955096] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:08 2019][1696447.960532] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1696447.965688] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:08 2019][1696447.972660] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:08 2019][1696447.978433] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:08 2019][1696447.985140] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1696447.990214] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696447.996486] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1696448.003119] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696448.009398] osp-syn-25-0 S ffff9159a929d140 0 6278 2 0x00000080 [Wed Feb 6 09:39:08 2019][1696448.016698] Call Trace: [Wed Feb 6 09:39:08 2019][1696448.019327] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:08 2019][1696448.025429] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1696448.030596] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:08 2019][1696448.037912] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:08 2019][1696448.043683] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.051015] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:08 2019][1696448.057568] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.064826] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.072073] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.079421] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.087181] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.094437] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.102200] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.109865] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:08 2019][1696448.116329] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:08 2019][1696448.122349] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:08 2019][1696448.128903] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.135705] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:08 2019][1696448.142160] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:08 2019][1696448.147826] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:08 2019][1696448.155414] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1696448.160480] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696448.166754] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1696448.173390] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696448.179669] osp-pre-25-0 S ffff9159a929e180 0 6279 2 0x00000080 [Wed Feb 6 09:39:08 2019][1696448.186959] Call Trace: [Wed Feb 6 09:39:08 2019][1696448.189588] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:08 2019][1696448.194997] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1696448.200161] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:08 2019][1696448.207129] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:08 2019][1696448.212888] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:08 2019][1696448.219613] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1696448.224676] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696448.230951] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1696448.237584] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696448.243857] osp-syn-17-0 S ffff9159fad08000 0 6280 2 0x00000080 [Wed Feb 6 09:39:08 2019][1696448.251153] Call Trace: [Wed Feb 6 09:39:08 2019][1696448.253785] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:08 2019][1696448.259889] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1696448.265056] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:08 2019][1696448.272369] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:08 2019][1696448.278154] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.285478] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:08 2019][1696448.292042] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.299285] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.306546] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.313883] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.321659] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.328898] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.336669] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.344329] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:08 2019][1696448.350793] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:08 2019][1696448.356814] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:08 2019][1696448.363373] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.370173] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:08 2019][1696448.376641] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:08 2019][1696448.382310] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:08 2019][1696448.389880] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1696448.394956] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696448.401227] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1696448.407850] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696448.414137] osp-pre-17-0 S ffff9159fad09040 0 6281 2 0x00000080 [Wed Feb 6 09:39:08 2019][1696448.421416] Call Trace: [Wed Feb 6 09:39:08 2019][1696448.424043] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:08 2019][1696448.429463] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1696448.434617] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:08 2019][1696448.441590] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:08 2019][1696448.447361] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:08 2019][1696448.454069] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1696448.459141] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696448.465417] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1696448.472052] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696448.478325] osp-syn-0-0 S ffff9159fad0a080 0 6282 2 0x00000080 [Wed Feb 6 09:39:08 2019][1696448.485618] Call Trace: [Wed Feb 6 09:39:08 2019][1696448.488253] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:08 2019][1696448.494354] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1696448.499518] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:08 2019][1696448.506833] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:08 2019][1696448.512618] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.519936] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:08 2019][1696448.526489] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.533749] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.540996] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.548342] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.556101] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.563359] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.571120] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.578799] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:08 2019][1696448.585243] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:08 2019][1696448.591262] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:08 2019][1696448.597822] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.604620] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:08 2019][1696448.611085] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:08 2019][1696448.616756] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:08 2019][1696448.624343] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1696448.629402] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696448.635678] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1696448.642312] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696448.648589] osp-pre-0-0 S ffff9159fad0b0c0 0 6283 2 0x00000080 [Wed Feb 6 09:39:08 2019][1696448.655879] Call Trace: [Wed Feb 6 09:39:08 2019][1696448.658511] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:08 2019][1696448.663919] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1696448.669087] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:08 2019][1696448.676052] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:08 2019][1696448.681814] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:08 2019][1696448.688532] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1696448.693596] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696448.699885] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1696448.706511] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696448.712800] osp-syn-5-0 S ffff9159fad0c100 0 6284 2 0x00000080 [Wed Feb 6 09:39:08 2019][1696448.720075] Call Trace: [Wed Feb 6 09:39:08 2019][1696448.722721] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:08 2019][1696448.728817] [] schedule+0x29/0x70 [Wed Feb 6 09:39:08 2019][1696448.733988] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:08 2019][1696448.741299] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:08 2019][1696448.747070] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.754405] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:08 2019][1696448.760954] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.768216] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.775465] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.782813] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.790578] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.797823] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.805588] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.813266] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:08 2019][1696448.819717] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:08 2019][1696448.825751] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:08 2019][1696448.832301] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:08 2019][1696448.839117] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:08 2019][1696448.845564] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:08 2019][1696448.851249] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:08 2019][1696448.858823] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:08 2019][1696448.863893] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:08 2019][1696448.870164] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:08 2019][1696448.876804] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696448.883077] osp-pre-5-0 S ffff9159fad0d140 0 6285 2 0x00000080 [Wed Feb 6 09:39:09 2019][1696448.890370] Call Trace: [Wed Feb 6 09:39:09 2019][1696448.893005] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:09 2019][1696448.898428] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1696448.903582] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:09 2019][1696448.910555] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:09 2019][1696448.916325] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:09 2019][1696448.923036] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1696448.928113] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696448.934389] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1696448.941024] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696448.947294] osp-syn-44-0 S ffff9159fad0e180 0 6286 2 0x00000080 [Wed Feb 6 09:39:09 2019][1696448.954596] Call Trace: [Wed Feb 6 09:39:09 2019][1696448.957225] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:09 2019][1696448.963323] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1696448.968483] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:09 2019][1696448.975800] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:09 2019][1696448.981573] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:09 2019][1696448.988901] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:09 2019][1696448.995453] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.002712] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.009954] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.017301] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.025055] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.032314] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.040078] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.047742] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:09 2019][1696449.054210] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:09 2019][1696449.060227] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:09 2019][1696449.066787] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.073594] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:09 2019][1696449.080058] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:09 2019][1696449.085732] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:09 2019][1696449.093321] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1696449.098381] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696449.104660] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1696449.111296] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696449.117572] osp-pre-44-0 S ffff91599f680000 0 6287 2 0x00000080 [Wed Feb 6 09:39:09 2019][1696449.124867] Call Trace: [Wed Feb 6 09:39:09 2019][1696449.127502] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:09 2019][1696449.132915] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1696449.138078] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:09 2019][1696449.145048] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:09 2019][1696449.150817] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:09 2019][1696449.157527] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1696449.162600] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696449.168868] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1696449.175508] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696449.181780] osp-syn-32-0 S ffff91599f681040 0 6288 2 0x00000080 [Wed Feb 6 09:39:09 2019][1696449.189048] Call Trace: [Wed Feb 6 09:39:09 2019][1696449.191692] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:09 2019][1696449.197793] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1696449.202959] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:09 2019][1696449.210274] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:09 2019][1696449.216058] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.223382] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:09 2019][1696449.229946] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.237191] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.244449] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.251775] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.259543] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.266803] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.274572] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.282247] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:09 2019][1696449.288700] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:09 2019][1696449.294735] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:09 2019][1696449.301279] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.308087] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:09 2019][1696449.314532] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:09 2019][1696449.320208] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:09 2019][1696449.327801] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1696449.332867] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696449.339142] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1696449.345778] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696449.352051] osp-pre-32-0 S ffff91599f682080 0 6289 2 0x00000080 [Wed Feb 6 09:39:09 2019][1696449.359329] Call Trace: [Wed Feb 6 09:39:09 2019][1696449.361957] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:09 2019][1696449.367358] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1696449.372524] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:09 2019][1696449.379491] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:09 2019][1696449.385254] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:09 2019][1696449.391980] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1696449.397045] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696449.403333] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1696449.409955] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696449.416244] osp-syn-21-0 S ffff91599f6830c0 0 6290 2 0x00000080 [Wed Feb 6 09:39:09 2019][1696449.423516] Call Trace: [Wed Feb 6 09:39:09 2019][1696449.426143] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:09 2019][1696449.432241] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1696449.437409] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:09 2019][1696449.444724] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:09 2019][1696449.450507] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.457830] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:09 2019][1696449.464388] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.471629] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.478891] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.486228] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.494003] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.501245] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.509025] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.516678] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:09 2019][1696449.523136] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:09 2019][1696449.529161] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:09 2019][1696449.535714] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.542528] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:09 2019][1696449.548976] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:09 2019][1696449.554661] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:09 2019][1696449.562236] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1696449.567310] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696449.573589] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1696449.580228] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696449.586505] osp-pre-21-0 S ffff91599f684100 0 6291 2 0x00000080 [Wed Feb 6 09:39:09 2019][1696449.593803] Call Trace: [Wed Feb 6 09:39:09 2019][1696449.596434] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:09 2019][1696449.601845] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1696449.607008] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:09 2019][1696449.613978] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:09 2019][1696449.619750] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:09 2019][1696449.626458] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1696449.631532] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696449.637803] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1696449.644438] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696449.650713] osp-syn-16-0 S ffff91599f685140 0 6292 2 0x00000080 [Wed Feb 6 09:39:09 2019][1696449.658006] Call Trace: [Wed Feb 6 09:39:09 2019][1696449.660635] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:09 2019][1696449.666757] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1696449.671924] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:09 2019][1696449.679239] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:09 2019][1696449.685025] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.692349] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:09 2019][1696449.698915] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.706157] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.713410] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.720743] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.728522] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.735761] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.743541] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.751199] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:09 2019][1696449.757663] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:09 2019][1696449.763685] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:09 2019][1696449.770252] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:09 2019][1696449.777050] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:09 2019][1696449.783515] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:09 2019][1696449.789193] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:09 2019][1696449.796784] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:09 2019][1696449.801850] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696449.808139] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:09 2019][1696449.814764] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:09 2019][1696449.821051] osp-pre-16-0 S ffff91599f686180 0 6293 2 0x00000080 [Wed Feb 6 09:39:09 2019][1696449.828340] Call Trace: [Wed Feb 6 09:39:09 2019][1696449.830977] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:09 2019][1696449.836397] [] schedule+0x29/0x70 [Wed Feb 6 09:39:09 2019][1696449.841543] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:09 2019][1696449.848511] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:09 2019][1696449.854283] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:09 2019][1696449.860991] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1696449.866067] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696449.872347] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1696449.878982] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696449.885258] osp-syn-41-0 S ffff9159a7268000 0 6294 2 0x00000080 [Wed Feb 6 09:39:10 2019][1696449.892550] Call Trace: [Wed Feb 6 09:39:10 2019][1696449.895180] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:10 2019][1696449.901283] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1696449.906450] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:10 2019][1696449.913765] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:10 2019][1696449.919545] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:10 2019][1696449.926865] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:10 2019][1696449.933433] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:10 2019][1696449.940675] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:10 2019][1696449.947931] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:10 2019][1696449.955257] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:10 2019][1696449.963037] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:10 2019][1696449.970274] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:10 2019][1696449.978036] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:10 2019][1696449.985715] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:10 2019][1696449.992165] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:10 2019][1696449.998197] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:10 2019][1696450.004744] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.011561] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:10 2019][1696450.018006] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:10 2019][1696450.023672] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:10 2019][1696450.031269] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1696450.036331] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.042605] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1696450.049241] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.055520] osp-pre-41-0 S ffff9159a7269040 0 6295 2 0x00000080 [Wed Feb 6 09:39:10 2019][1696450.062813] Call Trace: [Wed Feb 6 09:39:10 2019][1696450.065440] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:10 2019][1696450.070849] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1696450.076015] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:10 2019][1696450.082985] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:10 2019][1696450.088746] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:10 2019][1696450.095471] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1696450.100535] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.106824] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1696450.113450] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.119756] osp-syn-4-0 S ffff9159a726a080 0 6296 2 0x00000080 [Wed Feb 6 09:39:10 2019][1696450.127041] Call Trace: [Wed Feb 6 09:39:10 2019][1696450.129686] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:10 2019][1696450.135821] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1696450.140986] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:10 2019][1696450.148308] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:10 2019][1696450.154121] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.161459] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:10 2019][1696450.168017] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.175259] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.182519] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.189844] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.197626] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.204901] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.212694] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.220356] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:10 2019][1696450.226817] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:10 2019][1696450.232866] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:10 2019][1696450.239421] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.246221] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:10 2019][1696450.252702] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:10 2019][1696450.258481] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:10 2019][1696450.266066] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1696450.271149] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.277437] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1696450.284095] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.290433] osp-pre-4-0 S ffff9159a726b0c0 0 6297 2 0x00000080 [Wed Feb 6 09:39:10 2019][1696450.297736] Call Trace: [Wed Feb 6 09:39:10 2019][1696450.300378] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:10 2019][1696450.305782] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1696450.310984] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:10 2019][1696450.317954] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:10 2019][1696450.323711] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:10 2019][1696450.330476] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1696450.335566] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.341857] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1696450.348493] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.354770] osp-syn-23-0 S ffff9159a726c100 0 6298 2 0x00000080 [Wed Feb 6 09:39:10 2019][1696450.362091] Call Trace: [Wed Feb 6 09:39:10 2019][1696450.364747] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:10 2019][1696450.370849] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1696450.376016] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:10 2019][1696450.383329] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:10 2019][1696450.389124] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.396460] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:10 2019][1696450.403023] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.410264] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.417506] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.424838] [] ? discard_slab+0x39/0x50 [Wed Feb 6 09:39:10 2019][1696450.430524] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.438301] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.445542] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.453310] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.460987] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:10 2019][1696450.467433] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:10 2019][1696450.473463] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:10 2019][1696450.480010] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.486817] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:10 2019][1696450.493265] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:10 2019][1696450.498937] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:10 2019][1696450.506525] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1696450.511588] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.517864] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1696450.524499] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.530778] osp-pre-23-0 S ffff9159a726d140 0 6299 2 0x00000080 [Wed Feb 6 09:39:10 2019][1696450.538078] Call Trace: [Wed Feb 6 09:39:10 2019][1696450.540706] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:10 2019][1696450.546121] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1696450.551285] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:10 2019][1696450.558252] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:10 2019][1696450.564024] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:10 2019][1696450.570733] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1696450.575809] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.582086] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1696450.588722] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.595000] osp-syn-33-0 S ffff9159a726e180 0 6300 2 0x00000080 [Wed Feb 6 09:39:10 2019][1696450.602298] Call Trace: [Wed Feb 6 09:39:10 2019][1696450.604928] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:10 2019][1696450.611035] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1696450.616200] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:10 2019][1696450.623514] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:10 2019][1696450.629298] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.636625] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:10 2019][1696450.643185] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.650432] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.657688] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.665016] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.672793] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.680032] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.687806] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.695463] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:10 2019][1696450.701911] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:10 2019][1696450.707947] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:10 2019][1696450.714497] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:10 2019][1696450.721297] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:10 2019][1696450.727757] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:10 2019][1696450.733430] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:10 2019][1696450.741015] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1696450.746082] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.752367] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1696450.758987] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.765274] osp-pre-33-0 S ffff9159fc470000 0 6301 2 0x00000080 [Wed Feb 6 09:39:10 2019][1696450.772571] Call Trace: [Wed Feb 6 09:39:10 2019][1696450.775212] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:10 2019][1696450.780631] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1696450.785778] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:10 2019][1696450.792753] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:10 2019][1696450.798523] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:10 2019][1696450.805238] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:10 2019][1696450.810315] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.816586] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:10 2019][1696450.823206] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:10 2019][1696450.829494] osp-syn-20-0 S ffff9159fc471040 0 6302 2 0x00000080 [Wed Feb 6 09:39:10 2019][1696450.836776] Call Trace: [Wed Feb 6 09:39:10 2019][1696450.839428] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:10 2019][1696450.845534] [] schedule+0x29/0x70 [Wed Feb 6 09:39:10 2019][1696450.850698] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:10 2019][1696450.858016] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:10 2019][1696450.863799] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:11 2019][1696450.871122] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:11 2019][1696450.877691] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:11 2019][1696450.884939] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:11 2019][1696450.892198] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:11 2019][1696450.899518] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:11 2019][1696450.906515] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:11 2019][1696450.914274] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:11 2019][1696450.921536] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:11 2019][1696450.929294] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:11 2019][1696450.936974] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:11 2019][1696450.943422] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:11 2019][1696450.949447] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:11 2019][1696450.956014] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:11 2019][1696450.962810] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:11 2019][1696450.969277] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:11 2019][1696450.974950] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:11 2019][1696450.982534] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1696450.987592] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696450.993881] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1696451.000506] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696451.006781] osp-pre-20-0 S ffff9159fc472080 0 6303 2 0x00000080 [Wed Feb 6 09:39:11 2019][1696451.014076] Call Trace: [Wed Feb 6 09:39:11 2019][1696451.016710] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:11 2019][1696451.022120] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1696451.027287] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:11 2019][1696451.034254] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:11 2019][1696451.040027] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:11 2019][1696451.046731] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1696451.051806] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696451.058079] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1696451.064699] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696451.070986] osp-syn-7-0 S ffff9159fc4730c0 0 6304 2 0x00000080 [Wed Feb 6 09:39:11 2019][1696451.078259] Call Trace: [Wed Feb 6 09:39:11 2019][1696451.080890] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:11 2019][1696451.086992] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1696451.092159] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:11 2019][1696451.099473] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:11 2019][1696451.105261] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.112580] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:11 2019][1696451.119142] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.126390] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.133636] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.140982] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.148740] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.155999] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.163760] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.171438] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:11 2019][1696451.177883] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:11 2019][1696451.183915] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:11 2019][1696451.190461] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.197274] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:11 2019][1696451.203727] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:11 2019][1696451.209398] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:11 2019][1696451.216984] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1696451.222049] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696451.228338] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1696451.234960] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696451.241250] osp-pre-7-0 S ffff9159fc474100 0 6305 2 0x00000080 [Wed Feb 6 09:39:11 2019][1696451.248518] Call Trace: [Wed Feb 6 09:39:11 2019][1696451.251147] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:11 2019][1696451.256551] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1696451.261714] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:11 2019][1696451.268687] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:11 2019][1696451.274457] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:11 2019][1696451.281164] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1696451.286239] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696451.292510] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1696451.299146] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696451.305425] osp-syn-40-0 S ffff9159fc475140 0 6306 2 0x00000080 [Wed Feb 6 09:39:11 2019][1696451.312723] Call Trace: [Wed Feb 6 09:39:11 2019][1696451.315352] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:11 2019][1696451.321446] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1696451.326612] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:11 2019][1696451.333929] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:11 2019][1696451.339705] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.347035] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:11 2019][1696451.353582] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.360839] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.368084] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.375428] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.383187] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.390446] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.398207] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.405873] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:11 2019][1696451.412336] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:11 2019][1696451.418354] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:11 2019][1696451.424915] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.431710] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:11 2019][1696451.438156] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:11 2019][1696451.443843] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:11 2019][1696451.451418] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1696451.456480] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696451.462770] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1696451.469394] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696451.475678] osp-pre-40-0 S ffff9159fc476180 0 6307 2 0x00000080 [Wed Feb 6 09:39:11 2019][1696451.482958] Call Trace: [Wed Feb 6 09:39:11 2019][1696451.485600] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:11 2019][1696451.491006] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1696451.496172] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:11 2019][1696451.503143] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:11 2019][1696451.508901] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:11 2019][1696451.515628] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1696451.520692] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696451.526980] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1696451.533604] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696451.539891] osp-syn-45-0 S ffff9159fafe0000 0 6308 2 0x00000080 [Wed Feb 6 09:39:11 2019][1696451.547165] Call Trace: [Wed Feb 6 09:39:11 2019][1696451.549795] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:11 2019][1696451.555896] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1696451.561064] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:11 2019][1696451.568379] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:11 2019][1696451.574162] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.581478] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:11 2019][1696451.588041] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.595285] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.602542] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.609867] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:11 2019][1696451.616862] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.624622] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.631881] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.639641] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.647321] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:11 2019][1696451.653772] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:11 2019][1696451.659805] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:11 2019][1696451.666351] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.673168] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:11 2019][1696451.679614] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:11 2019][1696451.685289] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:11 2019][1696451.692874] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1696451.697938] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696451.704224] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1696451.710842] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696451.717133] osp-pre-45-0 S ffff9159fafe1040 0 6309 2 0x00000080 [Wed Feb 6 09:39:11 2019][1696451.724421] Call Trace: [Wed Feb 6 09:39:11 2019][1696451.727055] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:11 2019][1696451.732460] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1696451.737624] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:11 2019][1696451.744592] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:11 2019][1696451.750364] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:11 2019][1696451.757072] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:11 2019][1696451.762144] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696451.768415] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:11 2019][1696451.775053] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:11 2019][1696451.781332] osp-syn-6-0 S ffff9159fafe2080 0 6310 2 0x00000080 [Wed Feb 6 09:39:11 2019][1696451.788617] Call Trace: [Wed Feb 6 09:39:11 2019][1696451.791254] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:11 2019][1696451.797372] [] schedule+0x29/0x70 [Wed Feb 6 09:39:11 2019][1696451.802538] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:11 2019][1696451.809854] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:11 2019][1696451.815639] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.822962] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:11 2019][1696451.829518] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.836763] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.844019] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.851349] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.859127] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:11 2019][1696451.866369] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:12 2019][1696451.874149] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:12 2019][1696451.881811] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:12 2019][1696451.888267] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:12 2019][1696451.894294] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:12 2019][1696451.900847] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:12 2019][1696451.907646] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:12 2019][1696451.914108] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:12 2019][1696451.919782] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:12 2019][1696451.927373] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1696451.932439] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696451.938727] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1696451.945350] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696451.951640] osp-pre-6-0 S ffff9159fafe30c0 0 6311 2 0x00000080 [Wed Feb 6 09:39:12 2019][1696451.958929] Call Trace: [Wed Feb 6 09:39:12 2019][1696451.961569] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:12 2019][1696451.966974] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1696451.972143] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:12 2019][1696451.979110] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:12 2019][1696451.984869] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:12 2019][1696451.991592] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1696451.996650] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696452.002940] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1696452.009566] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696452.015854] osp-syn-24-0 S ffff9159fafe4100 0 6312 2 0x00000080 [Wed Feb 6 09:39:12 2019][1696452.023141] Call Trace: [Wed Feb 6 09:39:12 2019][1696452.025786] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:12 2019][1696452.031890] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1696452.037058] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:12 2019][1696452.044372] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:12 2019][1696452.050157] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.057480] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:12 2019][1696452.064048] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.071293] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.078551] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.085873] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:12 2019][1696452.092874] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.100632] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.107884] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.115644] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.123325] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:12 2019][1696452.129775] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:12 2019][1696452.135808] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:12 2019][1696452.142358] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.149171] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:12 2019][1696452.155621] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:12 2019][1696452.161305] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:12 2019][1696452.168879] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1696452.173954] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696452.180223] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1696452.186856] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696452.193125] osp-pre-24-0 S ffff9159fafe5140 0 6313 2 0x00000080 [Wed Feb 6 09:39:12 2019][1696452.200404] Call Trace: [Wed Feb 6 09:39:12 2019][1696452.203035] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:12 2019][1696452.208459] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1696452.213611] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:12 2019][1696452.220576] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:12 2019][1696452.226351] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:12 2019][1696452.233057] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1696452.238115] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696452.244409] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1696452.251034] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696452.257310] osp-syn-47-0 S ffff9159fafe6180 0 6314 2 0x00000080 [Wed Feb 6 09:39:12 2019][1696452.264602] Call Trace: [Wed Feb 6 09:39:12 2019][1696452.267237] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:12 2019][1696452.273343] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1696452.278508] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:12 2019][1696452.285824] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:12 2019][1696452.291605] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.298924] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:12 2019][1696452.305490] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.312732] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.319988] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.327312] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.335083] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.342323] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.350084] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.357763] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:12 2019][1696452.364215] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:12 2019][1696452.370247] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:12 2019][1696452.376793] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.383610] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:12 2019][1696452.390056] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:12 2019][1696452.395728] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:12 2019][1696452.403316] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1696452.408381] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696452.414655] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1696452.421293] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696452.427565] osp-pre-47-0 S ffff9159a8390000 0 6315 2 0x00000080 [Wed Feb 6 09:39:12 2019][1696452.434852] Call Trace: [Wed Feb 6 09:39:12 2019][1696452.437480] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:12 2019][1696452.442889] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1696452.448056] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:12 2019][1696452.455029] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:12 2019][1696452.460784] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:12 2019][1696452.467502] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1696452.472565] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696452.478853] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1696452.485469] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696452.491749] osp-syn-38-0 S ffff9159a8391040 0 6316 2 0x00000080 [Wed Feb 6 09:39:12 2019][1696452.499042] Call Trace: [Wed Feb 6 09:39:12 2019][1696452.501678] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:12 2019][1696452.507781] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1696452.512949] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:12 2019][1696452.520261] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:12 2019][1696452.526046] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.533362] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:12 2019][1696452.539929] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.547174] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.554430] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.561747] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:12 2019][1696452.568734] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.576511] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.583753] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.591530] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.599192] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:12 2019][1696452.605651] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:12 2019][1696452.611667] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:12 2019][1696452.618232] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.625033] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:12 2019][1696452.631498] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:12 2019][1696452.637173] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:12 2019][1696452.644763] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1696452.649825] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696452.656108] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1696452.662726] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696452.669012] osp-pre-38-0 S ffff9159a8392080 0 6317 2 0x00000080 [Wed Feb 6 09:39:12 2019][1696452.676301] Call Trace: [Wed Feb 6 09:39:12 2019][1696452.678945] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:12 2019][1696452.684351] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1696452.689520] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:12 2019][1696452.696483] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:12 2019][1696452.702255] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:12 2019][1696452.708961] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:12 2019][1696452.714014] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696452.720304] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:12 2019][1696452.726924] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:12 2019][1696452.733197] osp-syn-15-0 S ffff9159a83930c0 0 6318 2 0x00000080 [Wed Feb 6 09:39:12 2019][1696452.740497] Call Trace: [Wed Feb 6 09:39:12 2019][1696452.743127] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:12 2019][1696452.749233] [] schedule+0x29/0x70 [Wed Feb 6 09:39:12 2019][1696452.754396] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:12 2019][1696452.761713] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:12 2019][1696452.767492] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.774813] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:12 2019][1696452.781381] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.788617] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.795868] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.803199] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.810964] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.818222] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.825983] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.833662] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:12 2019][1696452.840113] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:12 2019][1696452.846146] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:12 2019][1696452.852694] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:12 2019][1696452.859509] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:12 2019][1696452.865959] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:13 2019][1696452.871640] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:13 2019][1696452.879219] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1696452.884297] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696452.890570] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1696452.897202] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696452.903476] osp-pre-15-0 S ffff9159a8394100 0 6319 2 0x00000080 [Wed Feb 6 09:39:13 2019][1696452.910768] Call Trace: [Wed Feb 6 09:39:13 2019][1696452.913395] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:13 2019][1696452.918801] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1696452.923967] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:13 2019][1696452.930933] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:13 2019][1696452.936706] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:13 2019][1696452.943413] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1696452.948488] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696452.954768] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1696452.961403] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696452.967682] osp-syn-29-0 S ffff9159a8395140 0 6320 2 0x00000080 [Wed Feb 6 09:39:13 2019][1696452.974980] Call Trace: [Wed Feb 6 09:39:13 2019][1696452.977609] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:13 2019][1696452.983713] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1696452.988879] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:13 2019][1696452.996196] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:13 2019][1696453.001966] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.009306] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:13 2019][1696453.015855] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.023103] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.030343] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.037684] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.045442] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.052682] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.060463] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.068121] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:13 2019][1696453.074584] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:13 2019][1696453.080607] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:13 2019][1696453.087173] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.093971] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:13 2019][1696453.100438] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:13 2019][1696453.106112] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:13 2019][1696453.113692] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1696453.118750] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696453.125038] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1696453.131658] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696453.137930] osp-pre-29-0 S ffff9159a8396180 0 6321 2 0x00000080 [Wed Feb 6 09:39:13 2019][1696453.145226] Call Trace: [Wed Feb 6 09:39:13 2019][1696453.147858] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:13 2019][1696453.153264] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1696453.158429] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:13 2019][1696453.165396] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:13 2019][1696453.171167] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:13 2019][1696453.177873] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1696453.182927] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696453.189223] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1696453.195846] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696453.202121] osp-syn-1-0 S ffff9159f7048000 0 6322 2 0x00000080 [Wed Feb 6 09:39:13 2019][1696453.209415] Call Trace: [Wed Feb 6 09:39:13 2019][1696453.212052] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:13 2019][1696453.218156] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1696453.223321] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:13 2019][1696453.230635] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:13 2019][1696453.236419] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.243743] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:13 2019][1696453.250313] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.257558] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.264819] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.272146] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.279924] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.287164] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.294944] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.302605] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:13 2019][1696453.309062] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:13 2019][1696453.315082] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:13 2019][1696453.321644] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.328441] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:13 2019][1696453.334899] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:13 2019][1696453.340570] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:13 2019][1696453.348149] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1696453.353224] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696453.359500] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1696453.366137] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696453.372413] osp-pre-1-0 S ffff9159f7049040 0 6323 2 0x00000080 [Wed Feb 6 09:39:13 2019][1696453.379688] Call Trace: [Wed Feb 6 09:39:13 2019][1696453.382330] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:13 2019][1696453.387734] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1696453.392895] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:13 2019][1696453.399864] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:13 2019][1696453.405622] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:13 2019][1696453.412340] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1696453.417399] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696453.423682] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1696453.430305] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696453.436585] osp-syn-22-0 S ffff9159f704a080 0 6324 2 0x00000080 [Wed Feb 6 09:39:13 2019][1696453.443862] Call Trace: [Wed Feb 6 09:39:13 2019][1696453.446500] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:13 2019][1696453.452602] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1696453.457768] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:13 2019][1696453.465082] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:13 2019][1696453.470866] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.478181] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:13 2019][1696453.484749] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.491989] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.499235] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.506575] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:13 2019][1696453.513561] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.521342] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.528577] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.536352] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.544011] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:13 2019][1696453.550459] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:13 2019][1696453.556492] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:13 2019][1696453.563041] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.569855] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:13 2019][1696453.576310] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:13 2019][1696453.581999] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:13 2019][1696453.589573] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1696453.594634] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696453.600925] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1696453.607551] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696453.613836] osp-pre-22-0 S ffff9159f704b0c0 0 6325 2 0x00000080 [Wed Feb 6 09:39:13 2019][1696453.621122] Call Trace: [Wed Feb 6 09:39:13 2019][1696453.623765] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:13 2019][1696453.629169] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1696453.634336] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:13 2019][1696453.641304] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:13 2019][1696453.647063] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:13 2019][1696453.653782] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1696453.658848] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696453.665150] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1696453.671769] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696453.678059] osp-syn-3-0 S ffff9159f704c100 0 6326 2 0x00000080 [Wed Feb 6 09:39:13 2019][1696453.685353] Call Trace: [Wed Feb 6 09:39:13 2019][1696453.687993] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:13 2019][1696453.694092] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1696453.699262] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:13 2019][1696453.706576] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:13 2019][1696453.712347] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.719680] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:13 2019][1696453.726228] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.733483] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.740729] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.748072] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.755834] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.763096] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.770855] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.778522] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:13 2019][1696453.784980] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:13 2019][1696453.790996] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:13 2019][1696453.797555] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:13 2019][1696453.804351] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:13 2019][1696453.810808] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:13 2019][1696453.816476] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:13 2019][1696453.824061] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:13 2019][1696453.829126] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696453.835414] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:13 2019][1696453.842037] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:13 2019][1696453.848330] osp-pre-3-0 S ffff9159f704d140 0 6327 2 0x00000080 [Wed Feb 6 09:39:13 2019][1696453.855607] Call Trace: [Wed Feb 6 09:39:13 2019][1696453.858235] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:13 2019][1696453.863646] [] schedule+0x29/0x70 [Wed Feb 6 09:39:13 2019][1696453.868813] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:14 2019][1696453.875780] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:14 2019][1696453.881552] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:14 2019][1696453.888261] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1696453.893334] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696453.899603] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1696453.906237] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696453.912505] osp-syn-27-0 S ffff9159f704e180 0 6328 2 0x00000080 [Wed Feb 6 09:39:14 2019][1696453.919786] Call Trace: [Wed Feb 6 09:39:14 2019][1696453.922439] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:14 2019][1696453.928543] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1696453.933712] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:14 2019][1696453.941025] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:14 2019][1696453.946809] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:14 2019][1696453.954126] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:14 2019][1696453.960688] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:14 2019][1696453.967932] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:14 2019][1696453.975191] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:14 2019][1696453.982518] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:14 2019][1696453.990297] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:14 2019][1696453.997536] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.005318] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.012983] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:14 2019][1696454.019449] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:14 2019][1696454.025469] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:14 2019][1696454.032027] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.038827] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:14 2019][1696454.045288] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:14 2019][1696454.050961] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:14 2019][1696454.058552] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1696454.063618] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.069905] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1696454.076530] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.082819] osp-pre-27-0 S ffff9159f7818000 0 6329 2 0x00000080 [Wed Feb 6 09:39:14 2019][1696454.090087] Call Trace: [Wed Feb 6 09:39:14 2019][1696454.092729] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:14 2019][1696454.098137] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1696454.103302] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:14 2019][1696454.110268] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:14 2019][1696454.116043] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:14 2019][1696454.122751] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1696454.127814] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.134102] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1696454.140729] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.147010] osp-syn-37-0 S ffff9159f7819040 0 6330 2 0x00000080 [Wed Feb 6 09:39:14 2019][1696454.154294] Call Trace: [Wed Feb 6 09:39:14 2019][1696454.156934] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:14 2019][1696454.163031] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1696454.168198] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:14 2019][1696454.175518] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:14 2019][1696454.181289] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.188621] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:14 2019][1696454.195168] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.202429] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.209668] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.217008] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:14 2019][1696454.223984] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.231764] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.239003] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.246758] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.254431] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:14 2019][1696454.260873] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:14 2019][1696454.266894] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:14 2019][1696454.273452] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.280251] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:14 2019][1696454.286713] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:14 2019][1696454.292380] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:14 2019][1696454.299965] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1696454.305022] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.311296] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1696454.317933] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.324201] osp-pre-37-0 S ffff9159f781a080 0 6331 2 0x00000080 [Wed Feb 6 09:39:14 2019][1696454.331489] Call Trace: [Wed Feb 6 09:39:14 2019][1696454.334115] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1696454.339273] [] osp_precreate_thread+0x45f/0x1380 [osp] [Wed Feb 6 09:39:14 2019][1696454.346244] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:14 2019][1696454.351999] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:14 2019][1696454.358718] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1696454.363778] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.370054] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1696454.376690] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.382967] osp-syn-36-0 S ffff9159f781b0c0 0 6332 2 0x00000080 [Wed Feb 6 09:39:14 2019][1696454.390259] Call Trace: [Wed Feb 6 09:39:14 2019][1696454.392902] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:14 2019][1696454.398996] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1696454.404164] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:14 2019][1696454.411477] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:14 2019][1696454.417248] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.424583] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:14 2019][1696454.431134] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.438379] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.445638] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.452963] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.460744] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.467986] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.475750] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.483430] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:14 2019][1696454.489879] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:14 2019][1696454.495913] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:14 2019][1696454.502462] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.509276] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:14 2019][1696454.515723] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:14 2019][1696454.521414] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:14 2019][1696454.528982] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1696454.534047] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.540342] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1696454.546961] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.553245] osp-pre-36-0 S ffff9159f781c100 0 6333 2 0x00000080 [Wed Feb 6 09:39:14 2019][1696454.560515] Call Trace: [Wed Feb 6 09:39:14 2019][1696454.563147] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:14 2019][1696454.568569] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1696454.573716] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:14 2019][1696454.580684] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:14 2019][1696454.586459] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:14 2019][1696454.593162] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1696454.598239] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.604513] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1696454.611129] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.617416] osp-syn-8-0 S ffff9159f781d140 0 6334 2 0x00000080 [Wed Feb 6 09:39:14 2019][1696454.624696] Call Trace: [Wed Feb 6 09:39:14 2019][1696454.627324] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:14 2019][1696454.633419] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1696454.638586] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:14 2019][1696454.645899] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:14 2019][1696454.651672] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.659010] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:14 2019][1696454.665573] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.672819] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.680077] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.687405] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.695184] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.702422] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.710205] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.717864] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:14 2019][1696454.724311] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:14 2019][1696454.730349] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:14 2019][1696454.736901] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:14 2019][1696454.743716] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:14 2019][1696454.750165] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:14 2019][1696454.755846] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:14 2019][1696454.763422] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1696454.768498] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.774770] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1696454.781394] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.787678] osp-pre-8-0 S ffff9159f781e180 0 6335 2 0x00000080 [Wed Feb 6 09:39:14 2019][1696454.794966] Call Trace: [Wed Feb 6 09:39:14 2019][1696454.797610] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:14 2019][1696454.803013] [] schedule+0x29/0x70 [Wed Feb 6 09:39:14 2019][1696454.808179] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:14 2019][1696454.815150] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:14 2019][1696454.820905] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:14 2019][1696454.827623] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:14 2019][1696454.832683] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.838958] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:14 2019][1696454.845594] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:14 2019][1696454.851872] osp-syn-35-0 S ffff9159ab260000 0 6336 2 0x00000080 [Wed Feb 6 09:39:14 2019][1696454.859161] Call Trace: [Wed Feb 6 09:39:14 2019][1696454.861789] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:14 2019][1696454.867883] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1696454.873051] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:15 2019][1696454.880363] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:15 2019][1696454.886136] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:15 2019][1696454.893473] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:15 2019][1696454.900022] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:15 2019][1696454.907281] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:15 2019][1696454.914522] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:15 2019][1696454.921862] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:15 2019][1696454.929622] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:15 2019][1696454.936877] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:15 2019][1696454.944637] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:15 2019][1696454.952302] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:15 2019][1696454.958764] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:15 2019][1696454.964786] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:15 2019][1696454.971349] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:15 2019][1696454.978150] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:15 2019][1696454.984608] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:15 2019][1696454.990276] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:15 2019][1696454.997860] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1696455.002926] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.009215] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1696455.015835] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.022118] osp-pre-35-0 S ffff9159ab261040 0 6337 2 0x00000080 [Wed Feb 6 09:39:15 2019][1696455.029400] Call Trace: [Wed Feb 6 09:39:15 2019][1696455.032037] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:15 2019][1696455.037446] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1696455.042614] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:15 2019][1696455.049581] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:15 2019][1696455.055352] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:15 2019][1696455.062058] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1696455.067138] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.073413] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1696455.080037] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.086324] osp-syn-28-0 S ffff9159ab262080 0 6338 2 0x00000080 [Wed Feb 6 09:39:15 2019][1696455.093621] Call Trace: [Wed Feb 6 09:39:15 2019][1696455.096256] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:15 2019][1696455.102372] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1696455.107522] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:15 2019][1696455.114844] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:15 2019][1696455.120628] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.127952] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:15 2019][1696455.134513] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.141755] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.149004] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.156328] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:15 2019][1696455.163313] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.171090] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.178332] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.186112] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.193780] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:15 2019][1696455.200241] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:15 2019][1696455.206266] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:15 2019][1696455.212827] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.219624] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:15 2019][1696455.226088] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:15 2019][1696455.231760] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:15 2019][1696455.239341] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1696455.244424] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.250704] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1696455.257326] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.263611] osp-pre-28-0 S ffff9159ab2630c0 0 6339 2 0x00000080 [Wed Feb 6 09:39:15 2019][1696455.270890] Call Trace: [Wed Feb 6 09:39:15 2019][1696455.273536] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:15 2019][1696455.278939] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1696455.284106] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:15 2019][1696455.291074] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:15 2019][1696455.296834] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:15 2019][1696455.303561] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1696455.308624] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.314912] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1696455.321537] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.327825] osp-syn-30-0 S ffff9159ab264100 0 6340 2 0x00000080 [Wed Feb 6 09:39:15 2019][1696455.335093] Call Trace: [Wed Feb 6 09:39:15 2019][1696455.337736] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:15 2019][1696455.343834] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1696455.349000] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:15 2019][1696455.356318] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:15 2019][1696455.362091] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.369424] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:15 2019][1696455.375971] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.383227] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.390467] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.397792] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.405566] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.412804] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.420570] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.428250] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:15 2019][1696455.434699] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:15 2019][1696455.440717] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:15 2019][1696455.447275] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.454073] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:15 2019][1696455.460518] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:15 2019][1696455.466206] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:15 2019][1696455.473781] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1696455.478843] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.485130] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1696455.491755] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.498038] osp-pre-30-0 S ffff9159ab265140 0 6341 2 0x00000080 [Wed Feb 6 09:39:15 2019][1696455.505311] Call Trace: [Wed Feb 6 09:39:15 2019][1696455.507955] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:15 2019][1696455.513365] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1696455.518529] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:15 2019][1696455.525496] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:15 2019][1696455.531268] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:15 2019][1696455.537976] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1696455.543049] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.549323] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1696455.555955] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.562226] osp-syn-34-0 S ffff9159ab266180 0 6342 2 0x00000080 [Wed Feb 6 09:39:15 2019][1696455.569528] Call Trace: [Wed Feb 6 09:39:15 2019][1696455.572165] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:15 2019][1696455.578269] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1696455.583433] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:15 2019][1696455.590748] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:15 2019][1696455.596535] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.603857] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:15 2019][1696455.610425] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.617666] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.624924] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.632242] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Wed Feb 6 09:39:15 2019][1696455.639244] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.647009] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.654269] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.662030] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.669706] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:15 2019][1696455.676149] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:15 2019][1696455.682186] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:15 2019][1696455.688735] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:15 2019][1696455.695536] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:15 2019][1696455.702002] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:15 2019][1696455.707674] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:15 2019][1696455.715258] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1696455.720319] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.726591] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1696455.733231] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.739507] osp-pre-34-0 S ffff9159a27a8000 0 6343 2 0x00000080 [Wed Feb 6 09:39:15 2019][1696455.746797] Call Trace: [Wed Feb 6 09:39:15 2019][1696455.749440] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:15 2019][1696455.754844] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1696455.760010] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:15 2019][1696455.766982] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:15 2019][1696455.772740] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:15 2019][1696455.779458] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1696455.784523] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.790809] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1696455.797428] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.803714] lquota_wb_fir-M S ffff9159a27ae180 0 6415 2 0x00000080 [Wed Feb 6 09:39:15 2019][1696455.810993] Call Trace: [Wed Feb 6 09:39:15 2019][1696455.813636] [] schedule+0x29/0x70 [Wed Feb 6 09:39:15 2019][1696455.818781] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:15 2019][1696455.824814] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:15 2019][1696455.831362] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Wed Feb 6 09:39:15 2019][1696455.838166] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Wed Feb 6 09:39:15 2019][1696455.844797] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:15 2019][1696455.850561] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Wed Feb 6 09:39:15 2019][1696455.856938] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:15 2019][1696455.861993] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:15 2019][1696455.868271] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:15 2019][1696455.874905] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696455.881183] lquota_wb_fir-M S ffff9159a27a9040 0 6416 2 0x00000080 [Wed Feb 6 09:39:16 2019][1696455.888477] Call Trace: [Wed Feb 6 09:39:16 2019][1696455.891108] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1696455.896266] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:16 2019][1696455.902276] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:16 2019][1696455.908819] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Wed Feb 6 09:39:16 2019][1696455.915638] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Wed Feb 6 09:39:16 2019][1696455.922258] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:16 2019][1696455.928019] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Wed Feb 6 09:39:16 2019][1696455.934395] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1696455.939461] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696455.945748] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1696455.952368] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696455.958652] dist_txn-0 S ffff9159a27aa080 0 6417 2 0x00000080 [Wed Feb 6 09:39:16 2019][1696455.965940] Call Trace: [Wed Feb 6 09:39:16 2019][1696455.968572] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:16 2019][1696455.974676] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1696455.979895] [] distribute_txn_commit_thread+0x9ac/0xca0 [ptlrpc] [Wed Feb 6 09:39:16 2019][1696455.987737] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:16 2019][1696455.993541] [] ? sub_trans_commit_cb+0x20/0x20 [ptlrpc] [Wed Feb 6 09:39:16 2019][1696456.000595] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1696456.005668] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.011938] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1696456.018563] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.024852] osp_up0-2 S ffff9159a0eae180 0 6432 2 0x00000080 [Wed Feb 6 09:39:16 2019][1696456.032149] Call Trace: [Wed Feb 6 09:39:16 2019][1696456.034807] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Wed Feb 6 09:39:16 2019][1696456.041770] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1696456.046937] [] osp_send_update_thread+0x2f8/0x590 [osp] [Wed Feb 6 09:39:16 2019][1696456.053991] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:16 2019][1696456.059754] [] ? osp_invalidate_request+0x390/0x390 [osp] [Wed Feb 6 09:39:16 2019][1696456.066999] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1696456.072064] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.078354] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1696456.084976] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.091266] osp-pre-0-2 S ffff9159a0ea8000 0 6433 2 0x00000080 [Wed Feb 6 09:39:16 2019][1696456.098533] Call Trace: [Wed Feb 6 09:39:16 2019][1696456.101163] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:16 2019][1696456.106575] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1696456.111739] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:16 2019][1696456.118705] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:16 2019][1696456.124475] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:16 2019][1696456.131182] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1696456.136242] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.142530] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1696456.149148] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.155436] osp_up1-2 S ffff9159a27ad140 0 6434 2 0x00000080 [Wed Feb 6 09:39:16 2019][1696456.162720] Call Trace: [Wed Feb 6 09:39:16 2019][1696456.165387] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Wed Feb 6 09:39:16 2019][1696456.172465] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Wed Feb 6 09:39:16 2019][1696456.179444] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1696456.184601] [] osp_send_update_thread+0x2f8/0x590 [osp] [Wed Feb 6 09:39:16 2019][1696456.191673] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:16 2019][1696456.197435] [] ? osp_invalidate_request+0x390/0x390 [osp] [Wed Feb 6 09:39:16 2019][1696456.204680] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1696456.209735] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.216025] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1696456.222648] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.228926] osp-pre-1-2 S ffff9159a27ac100 0 6435 2 0x00000080 [Wed Feb 6 09:39:16 2019][1696456.236224] Call Trace: [Wed Feb 6 09:39:16 2019][1696456.238856] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:16 2019][1696456.244262] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1696456.249436] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:16 2019][1696456.256400] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:16 2019][1696456.262154] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:16 2019][1696456.268875] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1696456.273933] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.280210] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1696456.286844] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.293124] osp_up3-2 S ffff9159a27ab0c0 0 6436 2 0x00000080 [Wed Feb 6 09:39:16 2019][1696456.300411] Call Trace: [Wed Feb 6 09:39:16 2019][1696456.303073] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Wed Feb 6 09:39:16 2019][1696456.310129] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Wed Feb 6 09:39:16 2019][1696456.317107] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1696456.322264] [] osp_send_update_thread+0x2f8/0x590 [osp] [Wed Feb 6 09:39:16 2019][1696456.329322] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:16 2019][1696456.335097] [] ? osp_invalidate_request+0x390/0x390 [osp] [Wed Feb 6 09:39:16 2019][1696456.342324] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1696456.347395] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.353665] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1696456.360286] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.366578] osp-pre-3-2 S ffff9159b88b8000 0 6437 2 0x00000080 [Wed Feb 6 09:39:16 2019][1696456.373857] Call Trace: [Wed Feb 6 09:39:16 2019][1696456.376490] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:16 2019][1696456.381905] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1696456.387068] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:16 2019][1696456.394035] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:16 2019][1696456.399809] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:16 2019][1696456.406515] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1696456.411576] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.417862] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1696456.424486] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.430760] osp-syn-12-2 S ffff9159b88b9040 0 6438 2 0x00000080 [Wed Feb 6 09:39:16 2019][1696456.438061] Call Trace: [Wed Feb 6 09:39:16 2019][1696456.440689] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:16 2019][1696456.446789] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1696456.451947] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:16 2019][1696456.459263] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:16 2019][1696456.465043] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.472387] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:16 2019][1696456.478936] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.486184] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.493439] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.500767] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.508548] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.515786] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.523547] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.531224] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:16 2019][1696456.537676] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:16 2019][1696456.543709] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:16 2019][1696456.550253] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.557047] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:16 2019][1696456.563505] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:16 2019][1696456.569170] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:16 2019][1696456.576742] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1696456.581823] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.588100] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1696456.594722] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.601009] osp-pre-12-2 S ffff9159b88ba080 0 6439 2 0x00000080 [Wed Feb 6 09:39:16 2019][1696456.608285] Call Trace: [Wed Feb 6 09:39:16 2019][1696456.610930] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:16 2019][1696456.616335] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1696456.621499] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:16 2019][1696456.628469] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:16 2019][1696456.634228] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:16 2019][1696456.640955] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1696456.646020] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.652298] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1696456.658930] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.665221] osp-syn-13-2 S ffff9159b88bb0c0 0 6440 2 0x00000080 [Wed Feb 6 09:39:16 2019][1696456.672517] Call Trace: [Wed Feb 6 09:39:16 2019][1696456.675145] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:16 2019][1696456.681247] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1696456.686418] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:16 2019][1696456.693730] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:16 2019][1696456.699503] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.706837] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:16 2019][1696456.713385] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.720639] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.727881] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.735208] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.742986] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.750229] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.757994] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.765667] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:16 2019][1696456.772110] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:16 2019][1696456.778149] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:16 2019][1696456.784693] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:16 2019][1696456.791491] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:16 2019][1696456.797957] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:16 2019][1696456.803631] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:16 2019][1696456.811212] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:16 2019][1696456.816269] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.822558] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:16 2019][1696456.829172] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:16 2019][1696456.835444] osp-pre-13-2 S ffff9159b88bc100 0 6441 2 0x00000080 [Wed Feb 6 09:39:16 2019][1696456.842747] Call Trace: [Wed Feb 6 09:39:16 2019][1696456.845385] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:16 2019][1696456.850801] [] schedule+0x29/0x70 [Wed Feb 6 09:39:16 2019][1696456.855967] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:16 2019][1696456.862936] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:16 2019][1696456.868707] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:16 2019][1696456.875416] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1696456.880492] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696456.886770] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1696456.893409] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696456.899682] osp-syn-39-2 S ffff9159b88bd140 0 6442 2 0x00000080 [Wed Feb 6 09:39:17 2019][1696456.906973] Call Trace: [Wed Feb 6 09:39:17 2019][1696456.909603] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:17 2019][1696456.915704] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1696456.920872] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:17 2019][1696456.928188] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:17 2019][1696456.933959] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:17 2019][1696456.941294] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:17 2019][1696456.947841] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:17 2019][1696456.955102] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:17 2019][1696456.962341] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:17 2019][1696456.969682] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:17 2019][1696456.977445] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:17 2019][1696456.984704] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:17 2019][1696456.992467] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.000131] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:17 2019][1696457.006595] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:17 2019][1696457.012617] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:17 2019][1696457.019167] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.025979] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:17 2019][1696457.032421] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:17 2019][1696457.038097] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:17 2019][1696457.045689] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1696457.050748] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.057036] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1696457.063661] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.069949] osp-pre-39-2 S ffff9159b88be180 0 6443 2 0x00000080 [Wed Feb 6 09:39:17 2019][1696457.077234] Call Trace: [Wed Feb 6 09:39:17 2019][1696457.079876] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:17 2019][1696457.085280] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1696457.090451] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:17 2019][1696457.097417] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:17 2019][1696457.103177] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:17 2019][1696457.109901] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1696457.114959] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.121235] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1696457.127871] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.134144] osp-syn-46-2 S ffff9159a2f50000 0 6444 2 0x00000080 [Wed Feb 6 09:39:17 2019][1696457.141426] Call Trace: [Wed Feb 6 09:39:17 2019][1696457.144074] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:17 2019][1696457.150169] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1696457.155337] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:17 2019][1696457.162650] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:17 2019][1696457.168425] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.175759] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:17 2019][1696457.182306] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.189566] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.196807] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.204151] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.211910] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.219168] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.226934] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.234597] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:17 2019][1696457.241059] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:17 2019][1696457.247082] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:17 2019][1696457.253649] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.260446] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:17 2019][1696457.266903] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:17 2019][1696457.272577] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:17 2019][1696457.280164] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1696457.285225] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.291506] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1696457.298138] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.304417] osp-pre-46-2 S ffff9159a2f51040 0 6445 2 0x00000080 [Wed Feb 6 09:39:17 2019][1696457.311716] Call Trace: [Wed Feb 6 09:39:17 2019][1696457.314345] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:17 2019][1696457.319749] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1696457.324916] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:17 2019][1696457.331879] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:17 2019][1696457.337654] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:17 2019][1696457.344360] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1696457.349424] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.355712] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1696457.362338] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.368627] osp-syn-10-2 S ffff9159a2f52080 0 6446 2 0x00000080 [Wed Feb 6 09:39:17 2019][1696457.375914] Call Trace: [Wed Feb 6 09:39:17 2019][1696457.378554] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:17 2019][1696457.384653] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1696457.389818] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:17 2019][1696457.397136] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:17 2019][1696457.402908] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.410241] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:17 2019][1696457.416789] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.424045] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.431285] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.438623] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.446385] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.453634] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.461412] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.469079] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:17 2019][1696457.475537] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:17 2019][1696457.481552] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:17 2019][1696457.488110] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.494911] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:17 2019][1696457.501364] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:17 2019][1696457.507052] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:17 2019][1696457.514625] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1696457.519699] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.525975] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1696457.532610] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.538879] osp-pre-10-2 S ffff9159a2f530c0 0 6447 2 0x00000080 [Wed Feb 6 09:39:17 2019][1696457.546156] Call Trace: [Wed Feb 6 09:39:17 2019][1696457.548797] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:17 2019][1696457.554205] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1696457.559372] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:17 2019][1696457.566339] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:17 2019][1696457.572112] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:17 2019][1696457.578819] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1696457.583890] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.590161] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1696457.596800] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.603073] osp-syn-11-2 S ffff9159a2f54100 0 6448 2 0x00000080 [Wed Feb 6 09:39:17 2019][1696457.610345] Call Trace: [Wed Feb 6 09:39:17 2019][1696457.612989] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:17 2019][1696457.619096] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1696457.624260] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:17 2019][1696457.631575] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:17 2019][1696457.637359] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.644676] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:17 2019][1696457.651239] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.658482] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.665745] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.673077] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.680853] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.688096] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.695871] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.703538] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:17 2019][1696457.710000] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:17 2019][1696457.716020] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:17 2019][1696457.722594] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.729393] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:17 2019][1696457.735846] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:17 2019][1696457.741531] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:17 2019][1696457.749104] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1696457.754179] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.760461] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1696457.767082] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.773369] osp-pre-11-2 S ffff9159a2f55140 0 6449 2 0x00000080 [Wed Feb 6 09:39:17 2019][1696457.780650] Call Trace: [Wed Feb 6 09:39:17 2019][1696457.783280] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:17 2019][1696457.788697] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1696457.793846] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:17 2019][1696457.800810] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:17 2019][1696457.806585] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:17 2019][1696457.813293] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:17 2019][1696457.818357] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.824645] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:17 2019][1696457.831269] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:17 2019][1696457.837555] osp-syn-14-2 S ffff9159a2f56180 0 6450 2 0x00000080 [Wed Feb 6 09:39:17 2019][1696457.844824] Call Trace: [Wed Feb 6 09:39:17 2019][1696457.847469] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:17 2019][1696457.853565] [] schedule+0x29/0x70 [Wed Feb 6 09:39:17 2019][1696457.858733] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:17 2019][1696457.866049] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:17 2019][1696457.871821] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:17 2019][1696457.879155] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:18 2019][1696457.885698] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:18 2019][1696457.892952] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:18 2019][1696457.900190] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:18 2019][1696457.907518] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:18 2019][1696457.915295] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:18 2019][1696457.922536] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:18 2019][1696457.930317] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:18 2019][1696457.937976] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:18 2019][1696457.944436] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:18 2019][1696457.950447] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:18 2019][1696457.957007] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:18 2019][1696457.963803] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:18 2019][1696457.970251] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:18 2019][1696457.975935] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:18 2019][1696457.983513] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1696457.988588] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696457.994864] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1696458.001496] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696458.007769] osp-pre-14-2 S ffff9159a2b28000 0 6451 2 0x00000080 [Wed Feb 6 09:39:18 2019][1696458.015069] Call Trace: [Wed Feb 6 09:39:18 2019][1696458.017699] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:18 2019][1696458.023112] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1696458.028275] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:18 2019][1696458.035245] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:18 2019][1696458.041013] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:18 2019][1696458.047720] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1696458.052784] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696458.059071] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1696458.065690] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696458.071966] osp-syn-26-2 S ffff9159a2b29040 0 6452 2 0x00000080 [Wed Feb 6 09:39:18 2019][1696458.079263] Call Trace: [Wed Feb 6 09:39:18 2019][1696458.081894] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:18 2019][1696458.087999] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1696458.093162] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:18 2019][1696458.100478] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:18 2019][1696458.106265] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.113586] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:18 2019][1696458.120131] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.127392] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.134632] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.141979] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.149737] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.156986] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.164746] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.172417] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:18 2019][1696458.178869] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:18 2019][1696458.184906] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:18 2019][1696458.191452] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.198270] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:18 2019][1696458.204713] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:18 2019][1696458.210398] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:18 2019][1696458.217973] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1696458.223035] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696458.229327] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1696458.235953] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696458.242227] osp-pre-26-2 S ffff9159a2b2a080 0 6453 2 0x00000080 [Wed Feb 6 09:39:18 2019][1696458.249524] Call Trace: [Wed Feb 6 09:39:18 2019][1696458.252153] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:18 2019][1696458.257555] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1696458.262720] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:18 2019][1696458.269691] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:18 2019][1696458.275450] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:18 2019][1696458.282177] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1696458.287243] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696458.293519] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1696458.300154] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696458.306429] osp-syn-31-2 S ffff9159a2b2b0c0 0 6454 2 0x00000080 [Wed Feb 6 09:39:18 2019][1696458.313722] Call Trace: [Wed Feb 6 09:39:18 2019][1696458.316351] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:18 2019][1696458.322457] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1696458.327621] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:18 2019][1696458.334935] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:18 2019][1696458.340721] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.348038] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:18 2019][1696458.354586] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.361839] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.369080] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.376413] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.384194] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.391433] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.399211] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.406871] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:18 2019][1696458.413318] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:18 2019][1696458.419352] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:18 2019][1696458.425900] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.432699] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:18 2019][1696458.439163] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:18 2019][1696458.444840] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:18 2019][1696458.452417] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1696458.457488] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696458.463758] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1696458.470394] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696458.476662] osp-pre-31-2 S ffff9159a2b2c100 0 6455 2 0x00000080 [Wed Feb 6 09:39:18 2019][1696458.483951] Call Trace: [Wed Feb 6 09:39:18 2019][1696458.486604] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:18 2019][1696458.492014] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1696458.497180] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:18 2019][1696458.504148] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:18 2019][1696458.509918] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:18 2019][1696458.516624] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1696458.521681] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696458.527966] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1696458.534587] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696458.540862] osp-syn-42-2 S ffff9159a2b2d140 0 6456 2 0x00000080 [Wed Feb 6 09:39:18 2019][1696458.548160] Call Trace: [Wed Feb 6 09:39:18 2019][1696458.550791] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:18 2019][1696458.556892] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1696458.562062] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:18 2019][1696458.569372] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:18 2019][1696458.575143] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.582481] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:18 2019][1696458.589028] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.596273] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.603537] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.610866] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.618625] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.625888] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.633656] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.641334] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:18 2019][1696458.647785] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:18 2019][1696458.653816] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:18 2019][1696458.660363] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.667186] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:18 2019][1696458.673635] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:18 2019][1696458.679309] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:18 2019][1696458.686903] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1696458.691969] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696458.698242] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1696458.704881] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696458.711153] osp-pre-42-2 S ffff9159a2b2e180 0 6457 2 0x00000080 [Wed Feb 6 09:39:18 2019][1696458.718438] Call Trace: [Wed Feb 6 09:39:18 2019][1696458.721067] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:18 2019][1696458.726469] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1696458.731633] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:18 2019][1696458.738606] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:18 2019][1696458.744365] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:18 2019][1696458.751090] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:18 2019][1696458.756152] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696458.762446] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:18 2019][1696458.769064] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:18 2019][1696458.775332] osp-syn-9-2 S ffff9159a9f00000 0 6458 2 0x00000080 [Wed Feb 6 09:39:18 2019][1696458.782627] Call Trace: [Wed Feb 6 09:39:18 2019][1696458.785260] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:18 2019][1696458.791356] [] schedule+0x29/0x70 [Wed Feb 6 09:39:18 2019][1696458.796522] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:18 2019][1696458.803838] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:18 2019][1696458.809610] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.816933] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:18 2019][1696458.823488] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.830732] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.837979] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.845321] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.853084] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.860341] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.868103] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:18 2019][1696458.875782] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:19 2019][1696458.882234] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:19 2019][1696458.888265] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:19 2019][1696458.894814] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:19 2019][1696458.901614] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:19 2019][1696458.908071] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:19 2019][1696458.913737] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:19 2019][1696458.921312] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:19 2019][1696458.926388] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696458.932665] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:19 2019][1696458.939301] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696458.945576] osp-pre-9-2 S ffff9159a9f01040 0 6459 2 0x00000080 [Wed Feb 6 09:39:19 2019][1696458.952878] Call Trace: [Wed Feb 6 09:39:19 2019][1696458.955511] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:19 2019][1696458.960917] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1696458.966086] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:19 2019][1696458.973054] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:19 2019][1696458.978822] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:19 2019][1696458.985546] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:19 2019][1696458.990604] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696458.996888] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:19 2019][1696459.003504] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696459.009770] osp-syn-2-2 S ffff9159a9f02080 0 6460 2 0x00000080 [Wed Feb 6 09:39:19 2019][1696459.017059] Call Trace: [Wed Feb 6 09:39:19 2019][1696459.019689] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:19 2019][1696459.025791] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1696459.030958] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:19 2019][1696459.038273] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:19 2019][1696459.044057] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.051377] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:19 2019][1696459.057934] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.065177] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.072424] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.079771] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.087531] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.094790] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.102547] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.110220] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:19 2019][1696459.116673] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:19 2019][1696459.122690] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:19 2019][1696459.129261] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.136064] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:19 2019][1696459.142517] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:19 2019][1696459.148203] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:19 2019][1696459.155772] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:19 2019][1696459.160852] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696459.167130] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:19 2019][1696459.173752] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696459.180039] osp-pre-2-2 S ffff9159a9f030c0 0 6461 2 0x00000080 [Wed Feb 6 09:39:19 2019][1696459.187319] Call Trace: [Wed Feb 6 09:39:19 2019][1696459.189965] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:19 2019][1696459.195373] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1696459.200538] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:19 2019][1696459.207509] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:19 2019][1696459.213266] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:19 2019][1696459.219983] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:19 2019][1696459.225044] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696459.231318] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:19 2019][1696459.237952] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696459.244232] osp-syn-43-2 S ffff9159a9f04100 0 6462 2 0x00000080 [Wed Feb 6 09:39:19 2019][1696459.251530] Call Trace: [Wed Feb 6 09:39:19 2019][1696459.254160] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:19 2019][1696459.260263] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1696459.265430] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:19 2019][1696459.272744] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:19 2019][1696459.278516] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.285852] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:19 2019][1696459.292400] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.299656] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.306896] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.314242] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.322000] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.329238] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.337013] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.344671] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:19 2019][1696459.351119] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:19 2019][1696459.357151] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:19 2019][1696459.363698] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.370494] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:19 2019][1696459.376954] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:19 2019][1696459.382629] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:19 2019][1696459.390222] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:19 2019][1696459.395287] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696459.401573] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:19 2019][1696459.408191] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696459.414468] osp-pre-43-2 S ffff9159a9f05140 0 6463 2 0x00000080 [Wed Feb 6 09:39:19 2019][1696459.421765] Call Trace: [Wed Feb 6 09:39:19 2019][1696459.424396] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:19 2019][1696459.429804] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1696459.434974] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:19 2019][1696459.441939] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:19 2019][1696459.447696] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:19 2019][1696459.454416] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:19 2019][1696459.459482] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696459.465758] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:19 2019][1696459.472393] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696459.478671] osp-syn-19-2 S ffff9159a9f06180 0 6464 2 0x00000080 [Wed Feb 6 09:39:19 2019][1696459.485964] Call Trace: [Wed Feb 6 09:39:19 2019][1696459.488601] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:19 2019][1696459.494705] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1696459.499871] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:19 2019][1696459.507182] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:19 2019][1696459.512956] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.520286] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:19 2019][1696459.526839] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.534098] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.541338] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.548685] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.556439] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.563692] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.571458] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.579118] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:19 2019][1696459.585581] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:19 2019][1696459.591604] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:19 2019][1696459.598173] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.604964] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:19 2019][1696459.611427] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:19 2019][1696459.617100] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:19 2019][1696459.624673] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:19 2019][1696459.629748] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696459.636025] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:19 2019][1696459.642647] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696459.648934] osp-pre-19-2 S ffff9159a4fa0000 0 6465 2 0x00000080 [Wed Feb 6 09:39:19 2019][1696459.656204] Call Trace: [Wed Feb 6 09:39:19 2019][1696459.658847] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:19 2019][1696459.664252] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1696459.669419] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:19 2019][1696459.676388] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:19 2019][1696459.682146] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:19 2019][1696459.688865] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:19 2019][1696459.693927] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696459.700214] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:19 2019][1696459.706830] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696459.713102] osp-syn-18-2 S ffff9159a4fa1040 0 6466 2 0x00000080 [Wed Feb 6 09:39:19 2019][1696459.720391] Call Trace: [Wed Feb 6 09:39:19 2019][1696459.723019] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:19 2019][1696459.729120] [] schedule+0x29/0x70 [Wed Feb 6 09:39:19 2019][1696459.734284] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:19 2019][1696459.741598] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:19 2019][1696459.747384] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.754707] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:19 2019][1696459.761269] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.768511] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.775769] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.783096] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.790855] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.798116] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.805878] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.813542] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:19 2019][1696459.820005] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:19 2019][1696459.826021] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:19 2019][1696459.832583] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:19 2019][1696459.839378] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:19 2019][1696459.845828] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:19 2019][1696459.851511] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:19 2019][1696459.859087] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:19 2019][1696459.864160] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:19 2019][1696459.870428] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:19 2019][1696459.877065] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696459.883333] osp-pre-18-2 S ffff9159a4fa2080 0 6467 2 0x00000080 [Wed Feb 6 09:39:20 2019][1696459.890617] Call Trace: [Wed Feb 6 09:39:20 2019][1696459.893247] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:20 2019][1696459.898675] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1696459.903829] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:20 2019][1696459.910803] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:20 2019][1696459.916573] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:20 2019][1696459.923282] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:20 2019][1696459.928357] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696459.934633] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:20 2019][1696459.941266] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696459.947541] osp-syn-25-2 S ffff9159a4fa30c0 0 6468 2 0x00000080 [Wed Feb 6 09:39:20 2019][1696459.954820] Call Trace: [Wed Feb 6 09:39:20 2019][1696459.957451] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:20 2019][1696459.963567] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1696459.968715] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:20 2019][1696459.976028] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:20 2019][1696459.981811] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:20 2019][1696459.989130] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:20 2019][1696459.995695] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.002937] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.010192] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.017519] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.025294] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.032530] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.040285] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.047960] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:20 2019][1696460.054410] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:20 2019][1696460.060431] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:20 2019][1696460.066995] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.073791] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:20 2019][1696460.080249] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:20 2019][1696460.085917] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:20 2019][1696460.093491] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:20 2019][1696460.098566] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696460.104846] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:20 2019][1696460.111481] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696460.117759] osp-pre-25-2 S ffff9159a4fa4100 0 6469 2 0x00000080 [Wed Feb 6 09:39:20 2019][1696460.125059] Call Trace: [Wed Feb 6 09:39:20 2019][1696460.127686] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:20 2019][1696460.133095] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1696460.138261] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:20 2019][1696460.145231] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:20 2019][1696460.150992] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:20 2019][1696460.157716] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:20 2019][1696460.162782] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696460.169071] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:20 2019][1696460.175700] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696460.181983] osp-syn-17-2 S ffff9159a4fa5140 0 6470 2 0x00000080 [Wed Feb 6 09:39:20 2019][1696460.189251] Call Trace: [Wed Feb 6 09:39:20 2019][1696460.191897] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:20 2019][1696460.197996] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1696460.203161] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:20 2019][1696460.210479] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:20 2019][1696460.216270] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.223593] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:20 2019][1696460.230148] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.237408] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.244642] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.251984] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.259742] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.266992] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.274771] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.282428] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:20 2019][1696460.288890] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:20 2019][1696460.294911] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:20 2019][1696460.301474] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.308274] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:20 2019][1696460.314732] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:20 2019][1696460.320402] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:20 2019][1696460.327988] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:20 2019][1696460.333051] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696460.339343] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:20 2019][1696460.345967] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696460.352251] osp-pre-17-2 S ffff9159a4fa6180 0 6471 2 0x00000080 [Wed Feb 6 09:39:20 2019][1696460.359520] Call Trace: [Wed Feb 6 09:39:20 2019][1696460.362164] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:20 2019][1696460.367569] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1696460.372739] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:20 2019][1696460.379701] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:20 2019][1696460.385456] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:20 2019][1696460.392184] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:20 2019][1696460.397248] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696460.403554] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:20 2019][1696460.410178] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696460.416478] osp-syn-0-2 S ffff9159a4690000 0 6472 2 0x00000080 [Wed Feb 6 09:39:20 2019][1696460.423752] Call Trace: [Wed Feb 6 09:39:20 2019][1696460.426381] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:20 2019][1696460.432481] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1696460.437647] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:20 2019][1696460.444958] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:20 2019][1696460.450726] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.458062] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:20 2019][1696460.464607] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.471850] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.479105] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.486437] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.494213] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.501465] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.509243] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.516902] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:20 2019][1696460.523363] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:20 2019][1696460.529378] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:20 2019][1696460.535936] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.542733] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:20 2019][1696460.549192] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:20 2019][1696460.554865] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:20 2019][1696460.562450] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:20 2019][1696460.567506] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696460.573782] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:20 2019][1696460.580419] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696460.586696] osp-pre-0-2 S ffff9159a4691040 0 6473 2 0x00000080 [Wed Feb 6 09:39:20 2019][1696460.593978] Call Trace: [Wed Feb 6 09:39:20 2019][1696460.596623] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:20 2019][1696460.602026] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1696460.607192] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:20 2019][1696460.614160] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:20 2019][1696460.619920] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:20 2019][1696460.626640] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:20 2019][1696460.631705] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696460.637993] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:20 2019][1696460.644615] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696460.650904] osp-syn-5-2 S ffff9159a4692080 0 6474 2 0x00000080 [Wed Feb 6 09:39:20 2019][1696460.658176] Call Trace: [Wed Feb 6 09:39:20 2019][1696460.660803] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:20 2019][1696460.666900] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1696460.672067] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:20 2019][1696460.679381] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:20 2019][1696460.685165] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.692481] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:20 2019][1696460.699042] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.706284] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.713521] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.720865] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.728627] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.735869] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.743648] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.751307] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:20 2019][1696460.757769] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:20 2019][1696460.763791] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:20 2019][1696460.770349] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:20 2019][1696460.777144] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:20 2019][1696460.783604] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:20 2019][1696460.789273] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:20 2019][1696460.796853] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:20 2019][1696460.801925] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696460.808198] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:20 2019][1696460.814830] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:20 2019][1696460.821101] osp-pre-5-2 S ffff9159a46930c0 0 6475 2 0x00000080 [Wed Feb 6 09:39:20 2019][1696460.828393] Call Trace: [Wed Feb 6 09:39:20 2019][1696460.831021] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:20 2019][1696460.836421] [] schedule+0x29/0x70 [Wed Feb 6 09:39:20 2019][1696460.841588] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:20 2019][1696460.848555] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:20 2019][1696460.854315] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:20 2019][1696460.861036] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1696460.866101] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696460.872378] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1696460.879009] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696460.885280] osp-syn-44-2 S ffff9159a4694100 0 6476 2 0x00000080 [Wed Feb 6 09:39:21 2019][1696460.892577] Call Trace: [Wed Feb 6 09:39:21 2019][1696460.895208] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:21 2019][1696460.901316] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1696460.906480] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:21 2019][1696460.913796] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:21 2019][1696460.919580] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:21 2019][1696460.926905] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:21 2019][1696460.933455] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:21 2019][1696460.940702] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:21 2019][1696460.947962] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:21 2019][1696460.955284] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:21 2019][1696460.963059] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:21 2019][1696460.970300] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:21 2019][1696460.978065] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:21 2019][1696460.985744] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:21 2019][1696460.992194] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:21 2019][1696460.998224] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:21 2019][1696461.004776] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.011592] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:21 2019][1696461.018041] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:21 2019][1696461.023724] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:21 2019][1696461.031300] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1696461.036372] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.042641] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1696461.049263] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.055552] osp-pre-44-2 S ffff9159a4695140 0 6477 2 0x00000080 [Wed Feb 6 09:39:21 2019][1696461.062826] Call Trace: [Wed Feb 6 09:39:21 2019][1696461.065465] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:21 2019][1696461.070871] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1696461.076040] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:21 2019][1696461.083007] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:21 2019][1696461.088776] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:21 2019][1696461.095485] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1696461.100558] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.106833] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1696461.113461] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.119737] osp-syn-32-2 S ffff9159a4696180 0 6478 2 0x00000080 [Wed Feb 6 09:39:21 2019][1696461.127040] Call Trace: [Wed Feb 6 09:39:21 2019][1696461.129678] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:21 2019][1696461.135776] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1696461.140941] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:21 2019][1696461.148259] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:21 2019][1696461.154027] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.161365] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:21 2019][1696461.167914] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.175159] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.182417] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.189747] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.197527] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.204772] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.212549] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.220213] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:21 2019][1696461.226660] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:21 2019][1696461.232690] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:21 2019][1696461.239241] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.246056] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:21 2019][1696461.252504] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:21 2019][1696461.258172] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:21 2019][1696461.265763] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1696461.270827] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.277102] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1696461.283731] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.290007] osp-pre-32-2 S ffff9159a7bf8000 0 6479 2 0x00000080 [Wed Feb 6 09:39:21 2019][1696461.297304] Call Trace: [Wed Feb 6 09:39:21 2019][1696461.299934] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:21 2019][1696461.305334] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1696461.310512] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:21 2019][1696461.317480] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:21 2019][1696461.323241] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:21 2019][1696461.329966] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1696461.335031] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.341317] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1696461.347935] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.354220] osp-syn-21-2 S ffff9159a7bf9040 0 6480 2 0x00000080 [Wed Feb 6 09:39:21 2019][1696461.361490] Call Trace: [Wed Feb 6 09:39:21 2019][1696461.364139] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:21 2019][1696461.370245] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1696461.375409] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:21 2019][1696461.382723] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:21 2019][1696461.388510] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.395834] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:21 2019][1696461.402401] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.409642] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.416898] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.424228] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.432007] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.439249] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.447021] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.454684] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:21 2019][1696461.461146] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:21 2019][1696461.467163] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:21 2019][1696461.473715] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.480533] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:21 2019][1696461.486980] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:21 2019][1696461.492660] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:21 2019][1696461.500236] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1696461.505296] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.511585] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1696461.518205] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.524481] osp-pre-21-2 S ffff9159a7bfa080 0 6481 2 0x00000080 [Wed Feb 6 09:39:21 2019][1696461.531777] Call Trace: [Wed Feb 6 09:39:21 2019][1696461.534413] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:21 2019][1696461.539818] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1696461.544987] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:21 2019][1696461.551952] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:21 2019][1696461.557713] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:21 2019][1696461.564430] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1696461.569487] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.575771] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1696461.582389] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.588677] osp-syn-16-2 S ffff9159a7bfb0c0 0 6482 2 0x00000080 [Wed Feb 6 09:39:21 2019][1696461.595962] Call Trace: [Wed Feb 6 09:39:21 2019][1696461.598600] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:21 2019][1696461.604697] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1696461.609866] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:21 2019][1696461.617177] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:21 2019][1696461.622938] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.630267] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:21 2019][1696461.636814] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.644050] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.651301] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.658635] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.666397] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.673657] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.681415] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.689092] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:21 2019][1696461.695536] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:21 2019][1696461.701554] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:21 2019][1696461.708113] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:21 2019][1696461.714912] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:21 2019][1696461.721368] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:21 2019][1696461.727052] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:21 2019][1696461.727530] LustreError: 15221:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1549474499, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff91280f308900/0x21ed9351bc56335e lrc: 3/1,0 mode: --/PR res: [0x2c0001541:0x82f7:0x0].0x0 bits 0x13/0x0 rrc: 25 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 15221 timeout: 0 lvb_type: 0 [Wed Feb 6 09:39:21 2019][1696461.774367] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1696461.779426] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.785702] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1696461.792330] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.798601] osp-pre-16-2 S ffff9159a7bfc100 0 6483 2 0x00000080 [Wed Feb 6 09:39:21 2019][1696461.805898] Call Trace: [Wed Feb 6 09:39:21 2019][1696461.808524] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:21 2019][1696461.813938] [] schedule+0x29/0x70 [Wed Feb 6 09:39:21 2019][1696461.819099] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:21 2019][1696461.826072] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:21 2019][1696461.831843] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:21 2019][1696461.838552] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:21 2019][1696461.843625] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.849902] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:21 2019][1696461.856536] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:21 2019][1696461.862806] osp-syn-41-2 S ffff9159a7bfd140 0 6484 2 0x00000080 [Wed Feb 6 09:39:22 2019][1696461.870086] Call Trace: [Wed Feb 6 09:39:22 2019][1696461.872738] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:22 2019][1696461.878843] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1696461.884011] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:22 2019][1696461.891324] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:22 2019][1696461.897094] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:22 2019][1696461.904428] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:22 2019][1696461.910978] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:22 2019][1696461.918236] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:22 2019][1696461.925474] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:22 2019][1696461.932816] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:22 2019][1696461.940585] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:22 2019][1696461.947828] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:22 2019][1696461.955608] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:22 2019][1696461.963265] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:22 2019][1696461.969726] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:22 2019][1696461.975738] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:22 2019][1696461.982289] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:22 2019][1696461.989104] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:22 2019][1696461.995553] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:22 2019][1696462.001236] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:22 2019][1696462.008808] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:22 2019][1696462.013885] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.020153] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:22 2019][1696462.026775] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.033066] osp-pre-41-2 S ffff9159a7bfe180 0 6485 2 0x00000080 [Wed Feb 6 09:39:22 2019][1696462.040357] Call Trace: [Wed Feb 6 09:39:22 2019][1696462.042992] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:22 2019][1696462.048399] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1696462.053567] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:22 2019][1696462.060535] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:22 2019][1696462.066295] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:22 2019][1696462.073021] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:22 2019][1696462.078087] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.084373] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:22 2019][1696462.090993] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.097265] osp-syn-4-2 S ffff91599ff68000 0 6486 2 0x00000080 [Wed Feb 6 09:39:22 2019][1696462.104547] Call Trace: [Wed Feb 6 09:39:22 2019][1696462.107191] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:22 2019][1696462.113287] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1696462.118454] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:22 2019][1696462.125769] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:22 2019][1696462.131547] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.138878] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:22 2019][1696462.145425] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.152686] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.159922] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.167260] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.175024] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.182268] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.190048] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.197706] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:22 2019][1696462.204169] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:22 2019][1696462.210190] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:22 2019][1696462.216747] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.223544] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:22 2019][1696462.229987] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:22 2019][1696462.235673] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:22 2019][1696462.243250] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:22 2019][1696462.248313] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.254603] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:22 2019][1696462.261225] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.267501] osp-pre-4-2 S ffff91599ff69040 0 6487 2 0x00000080 [Wed Feb 6 09:39:22 2019][1696462.274805] Call Trace: [Wed Feb 6 09:39:22 2019][1696462.277443] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:22 2019][1696462.282850] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1696462.288017] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:22 2019][1696462.294984] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:22 2019][1696462.300743] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:22 2019][1696462.307470] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:22 2019][1696462.312535] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.318809] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:22 2019][1696462.325444] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.331722] osp-syn-23-2 S ffff91599ff6a080 0 6488 2 0x00000080 [Wed Feb 6 09:39:22 2019][1696462.339034] Call Trace: [Wed Feb 6 09:39:22 2019][1696462.341670] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:22 2019][1696462.347774] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1696462.352942] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:22 2019][1696462.360253] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:22 2019][1696462.366041] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.373362] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:22 2019][1696462.379918] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.387176] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.394426] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.401756] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.409539] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.416784] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.424566] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.432232] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:22 2019][1696462.438693] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:22 2019][1696462.444718] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:22 2019][1696462.451284] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.458091] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:22 2019][1696462.464557] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:22 2019][1696462.470231] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:22 2019][1696462.477814] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:22 2019][1696462.482885] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.489159] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:22 2019][1696462.495793] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.502070] osp-pre-23-2 S ffff91599ff6b0c0 0 6489 2 0x00000080 [Wed Feb 6 09:39:22 2019][1696462.509372] Call Trace: [Wed Feb 6 09:39:22 2019][1696462.511999] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:22 2019][1696462.517411] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1696462.522572] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:22 2019][1696462.529541] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:22 2019][1696462.535314] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:22 2019][1696462.542021] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:22 2019][1696462.547098] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.553369] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:22 2019][1696462.559991] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.566279] osp-syn-33-2 S ffff91599ff6c100 0 6490 2 0x00000080 [Wed Feb 6 09:39:22 2019][1696462.573556] Call Trace: [Wed Feb 6 09:39:22 2019][1696462.576199] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:22 2019][1696462.582295] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1696462.587464] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:22 2019][1696462.594779] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:22 2019][1696462.600550] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.607888] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:22 2019][1696462.614435] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.621680] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.628934] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.636261] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.644036] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.651275] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.659042] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.666737] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:22 2019][1696462.673189] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:22 2019][1696462.679220] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:22 2019][1696462.685770] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.692582] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:22 2019][1696462.699033] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:22 2019][1696462.704705] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:22 2019][1696462.712286] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:22 2019][1696462.717343] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.723636] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:22 2019][1696462.730258] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.736540] osp-pre-33-2 S ffff91599ff6d140 0 6491 2 0x00000080 [Wed Feb 6 09:39:22 2019][1696462.743835] Call Trace: [Wed Feb 6 09:39:22 2019][1696462.746463] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:22 2019][1696462.751866] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1696462.757036] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:22 2019][1696462.763996] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:22 2019][1696462.769771] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:22 2019][1696462.776479] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:22 2019][1696462.781545] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.787829] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:22 2019][1696462.794447] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:22 2019][1696462.800732] osp-syn-20-2 S ffff91599ff6e180 0 6492 2 0x00000080 [Wed Feb 6 09:39:22 2019][1696462.808006] Call Trace: [Wed Feb 6 09:39:22 2019][1696462.810650] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:22 2019][1696462.816755] [] schedule+0x29/0x70 [Wed Feb 6 09:39:22 2019][1696462.821920] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:22 2019][1696462.829234] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:22 2019][1696462.835019] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.842332] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:22 2019][1696462.848876] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.856129] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:22 2019][1696462.863369] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:23 2019][1696462.870697] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:23 2019][1696462.878472] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:23 2019][1696462.885716] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:23 2019][1696462.893494] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:23 2019][1696462.901154] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:23 2019][1696462.907616] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:23 2019][1696462.913637] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:23 2019][1696462.920195] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:23 2019][1696462.926994] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:23 2019][1696462.933448] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:23 2019][1696462.939133] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:23 2019][1696462.946705] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:23 2019][1696462.951778] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696462.958051] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:23 2019][1696462.964674] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696462.970961] osp-pre-20-2 S ffff9159fc268000 0 6493 2 0x00000080 [Wed Feb 6 09:39:23 2019][1696462.978242] Call Trace: [Wed Feb 6 09:39:23 2019][1696462.980896] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:23 2019][1696462.986307] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1696462.991472] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:23 2019][1696462.998438] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:23 2019][1696463.004210] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:23 2019][1696463.010918] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:23 2019][1696463.015980] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696463.022270] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:23 2019][1696463.028896] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696463.035169] osp-syn-7-2 S ffff9159fc269040 0 6494 2 0x00000080 [Wed Feb 6 09:39:23 2019][1696463.042468] Call Trace: [Wed Feb 6 09:39:23 2019][1696463.045099] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:23 2019][1696463.051203] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1696463.056372] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:23 2019][1696463.063685] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:23 2019][1696463.069470] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.076795] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:23 2019][1696463.083357] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.090600] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.097861] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.105183] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.112955] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.120194] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.127952] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.135625] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:23 2019][1696463.142079] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:23 2019][1696463.148106] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:23 2019][1696463.154670] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.161470] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:23 2019][1696463.167930] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:23 2019][1696463.173598] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:23 2019][1696463.181183] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:23 2019][1696463.186238] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696463.192511] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:23 2019][1696463.199143] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696463.205412] osp-pre-7-2 S ffff9159fc26a080 0 6495 2 0x00000080 [Wed Feb 6 09:39:23 2019][1696463.212711] Call Trace: [Wed Feb 6 09:39:23 2019][1696463.215342] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:23 2019][1696463.220752] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1696463.225917] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:23 2019][1696463.232885] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:23 2019][1696463.238643] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:23 2019][1696463.245359] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:23 2019][1696463.250438] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696463.256726] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:23 2019][1696463.263352] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696463.269638] osp-syn-40-2 S ffff9159fc26b0c0 0 6496 2 0x00000080 [Wed Feb 6 09:39:23 2019][1696463.276909] Call Trace: [Wed Feb 6 09:39:23 2019][1696463.279537] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:23 2019][1696463.285650] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1696463.290802] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:23 2019][1696463.298113] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:23 2019][1696463.303900] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.311224] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:23 2019][1696463.317790] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.325030] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.332277] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.339625] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.347387] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.354642] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.362404] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.370068] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:23 2019][1696463.376529] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:23 2019][1696463.382559] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:23 2019][1696463.389131] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.395930] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:23 2019][1696463.402376] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:23 2019][1696463.408065] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:23 2019][1696463.415641] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:23 2019][1696463.420707] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696463.426997] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:23 2019][1696463.433620] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696463.439897] osp-pre-40-2 S ffff9159fc26c100 0 6497 2 0x00000080 [Wed Feb 6 09:39:23 2019][1696463.447187] Call Trace: [Wed Feb 6 09:39:23 2019][1696463.449816] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:23 2019][1696463.455226] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1696463.460390] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:23 2019][1696463.467359] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:23 2019][1696463.473121] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:23 2019][1696463.479839] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:23 2019][1696463.484904] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696463.491189] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:23 2019][1696463.497805] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696463.504094] osp-syn-45-2 S ffff9159fc26d140 0 6498 2 0x00000080 [Wed Feb 6 09:39:23 2019][1696463.511366] Call Trace: [Wed Feb 6 09:39:23 2019][1696463.513996] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:23 2019][1696463.520095] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1696463.525258] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:23 2019][1696463.532572] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:23 2019][1696463.538338] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.545669] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:23 2019][1696463.552218] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.559461] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.566720] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.574045] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.581814] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.589058] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.596822] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.604506] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:23 2019][1696463.610955] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:23 2019][1696463.616976] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:23 2019][1696463.623541] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.630338] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:23 2019][1696463.636795] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:23 2019][1696463.642458] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:23 2019][1696463.650043] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:23 2019][1696463.655105] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696463.661381] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:23 2019][1696463.668009] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696463.674281] osp-pre-45-2 S ffff9159fc26e180 0 6499 2 0x00000080 [Wed Feb 6 09:39:23 2019][1696463.681583] Call Trace: [Wed Feb 6 09:39:23 2019][1696463.684212] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:23 2019][1696463.689617] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1696463.694782] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:23 2019][1696463.701752] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:23 2019][1696463.707518] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:23 2019][1696463.714226] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:23 2019][1696463.719301] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696463.725572] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:23 2019][1696463.732197] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:23 2019][1696463.738485] osp-syn-6-2 S ffff9159fa610000 0 6500 2 0x00000080 [Wed Feb 6 09:39:23 2019][1696463.745768] Call Trace: [Wed Feb 6 09:39:23 2019][1696463.748402] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:23 2019][1696463.754519] [] schedule+0x29/0x70 [Wed Feb 6 09:39:23 2019][1696463.759675] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:23 2019][1696463.766994] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:23 2019][1696463.772777] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.780093] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:23 2019][1696463.786649] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.793891] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.801150] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.808475] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.816236] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.823495] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.831251] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.838927] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:23 2019][1696463.845373] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:23 2019][1696463.851392] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:23 2019][1696463.857951] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:23 2019][1696463.864750] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:24 2019][1696463.871205] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:24 2019][1696463.876889] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:24 2019][1696463.884461] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1696463.889535] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696463.895807] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1696463.902430] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696463.908719] osp-pre-6-2 S ffff9159fa611040 0 6501 2 0x00000080 [Wed Feb 6 09:39:24 2019][1696463.915990] Call Trace: [Wed Feb 6 09:39:24 2019][1696463.918635] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:24 2019][1696463.924048] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1696463.929209] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:24 2019][1696463.936178] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:24 2019][1696463.941953] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:24 2019][1696463.948660] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1696463.953721] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696463.960009] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1696463.966645] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696463.972932] osp-syn-24-2 S ffff9159fa612080 0 6502 2 0x00000080 [Wed Feb 6 09:39:24 2019][1696463.980200] Call Trace: [Wed Feb 6 09:39:24 2019][1696463.982844] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:24 2019][1696463.988941] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1696463.994109] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:24 2019][1696464.001423] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:24 2019][1696464.007195] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.014531] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:24 2019][1696464.021078] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.028338] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.035575] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.042905] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.050676] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.057919] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.065700] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.073360] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:24 2019][1696464.079808] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:24 2019][1696464.085840] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:24 2019][1696464.092390] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.099204] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:24 2019][1696464.105653] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:24 2019][1696464.111336] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:24 2019][1696464.118913] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1696464.123989] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696464.130267] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1696464.136900] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696464.143172] osp-pre-24-2 S ffff9159fa6130c0 0 6503 2 0x00000080 [Wed Feb 6 09:39:24 2019][1696464.150469] Call Trace: [Wed Feb 6 09:39:24 2019][1696464.153103] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:24 2019][1696464.158507] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1696464.163670] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:24 2019][1696464.170636] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:24 2019][1696464.176397] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:24 2019][1696464.183121] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1696464.188188] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696464.194474] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1696464.201092] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696464.207380] osp-syn-47-2 S ffff9159fa614100 0 6504 2 0x00000080 [Wed Feb 6 09:39:24 2019][1696464.214656] Call Trace: [Wed Feb 6 09:39:24 2019][1696464.217285] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:24 2019][1696464.223381] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1696464.228551] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:24 2019][1696464.235863] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:24 2019][1696464.241635] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.248970] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:24 2019][1696464.255518] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.262776] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.270017] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.277360] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.285123] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.292384] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.300143] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.307818] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:24 2019][1696464.314260] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:24 2019][1696464.320275] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:24 2019][1696464.326843] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.333640] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:24 2019][1696464.340104] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:24 2019][1696464.345772] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:24 2019][1696464.353356] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1696464.358417] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696464.364704] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1696464.371324] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696464.377601] osp-pre-47-2 S ffff9159fa615140 0 6505 2 0x00000080 [Wed Feb 6 09:39:24 2019][1696464.384891] Call Trace: [Wed Feb 6 09:39:24 2019][1696464.387521] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:24 2019][1696464.392930] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1696464.398097] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:24 2019][1696464.405062] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:24 2019][1696464.410826] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:24 2019][1696464.417543] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1696464.422600] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696464.428887] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1696464.435504] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696464.441794] osp-syn-38-2 S ffff9159fa616180 0 6506 2 0x00000080 [Wed Feb 6 09:39:24 2019][1696464.449083] Call Trace: [Wed Feb 6 09:39:24 2019][1696464.451730] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:24 2019][1696464.457827] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1696464.462985] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:24 2019][1696464.470304] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:24 2019][1696464.476083] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.483418] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:24 2019][1696464.489966] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.497225] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.504464] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.511797] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.519578] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.526823] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.534604] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.542262] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:24 2019][1696464.548726] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:24 2019][1696464.554745] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:24 2019][1696464.561296] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.568109] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:24 2019][1696464.574558] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:24 2019][1696464.580241] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:24 2019][1696464.587812] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1696464.592885] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696464.599163] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1696464.605783] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696464.612072] osp-pre-38-2 S ffff9159f9648000 0 6507 2 0x00000080 [Wed Feb 6 09:39:24 2019][1696464.619347] Call Trace: [Wed Feb 6 09:39:24 2019][1696464.621991] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:24 2019][1696464.627400] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1696464.632561] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:24 2019][1696464.639529] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:24 2019][1696464.645306] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:24 2019][1696464.652015] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1696464.657075] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696464.663361] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1696464.669979] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696464.676268] osp-syn-15-2 S ffff9159f9649040 0 6508 2 0x00000080 [Wed Feb 6 09:39:24 2019][1696464.683561] Call Trace: [Wed Feb 6 09:39:24 2019][1696464.686190] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:24 2019][1696464.692283] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1696464.697443] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:24 2019][1696464.704760] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:24 2019][1696464.710530] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.717866] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:24 2019][1696464.724411] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.731674] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.738909] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.746236] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.754013] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.761257] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.769034] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.776696] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:24 2019][1696464.783152] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:24 2019][1696464.789167] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:24 2019][1696464.795724] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:24 2019][1696464.802522] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:24 2019][1696464.808972] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:24 2019][1696464.814664] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:24 2019][1696464.822238] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:24 2019][1696464.827311] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696464.833582] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:24 2019][1696464.840217] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:24 2019][1696464.846488] osp-pre-15-2 S ffff9159f964a080 0 6509 2 0x00000080 [Wed Feb 6 09:39:24 2019][1696464.853766] Call Trace: [Wed Feb 6 09:39:24 2019][1696464.856404] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:24 2019][1696464.861826] [] schedule+0x29/0x70 [Wed Feb 6 09:39:24 2019][1696464.866983] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:25 2019][1696464.873954] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:25 2019][1696464.879724] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:25 2019][1696464.886431] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1696464.891493] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696464.897785] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1696464.904411] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696464.910684] osp-syn-29-2 S ffff9159f964b0c0 0 6510 2 0x00000080 [Wed Feb 6 09:39:25 2019][1696464.917985] Call Trace: [Wed Feb 6 09:39:25 2019][1696464.920612] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:25 2019][1696464.926705] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1696464.931878] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:25 2019][1696464.939190] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:25 2019][1696464.944957] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:25 2019][1696464.952297] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:25 2019][1696464.958844] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:25 2019][1696464.966087] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:25 2019][1696464.973346] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:25 2019][1696464.980672] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:25 2019][1696464.988429] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:25 2019][1696464.995684] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.003443] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.011110] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:25 2019][1696465.017571] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:25 2019][1696465.023594] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:25 2019][1696465.030163] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.036962] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:25 2019][1696465.043411] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:25 2019][1696465.049093] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:25 2019][1696465.056671] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1696465.061745] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696465.068024] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1696465.074642] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696465.080930] osp-pre-29-2 S ffff9159f964c100 0 6511 2 0x00000080 [Wed Feb 6 09:39:25 2019][1696465.088210] Call Trace: [Wed Feb 6 09:39:25 2019][1696465.090840] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:25 2019][1696465.096249] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1696465.101419] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:25 2019][1696465.108385] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:25 2019][1696465.114143] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:25 2019][1696465.120871] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1696465.125936] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696465.132208] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1696465.138841] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696465.145114] osp-syn-1-2 S ffff9159f964d140 0 6512 2 0x00000080 [Wed Feb 6 09:39:25 2019][1696465.152400] Call Trace: [Wed Feb 6 09:39:25 2019][1696465.155042] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:25 2019][1696465.161140] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1696465.166303] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:25 2019][1696465.173618] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:25 2019][1696465.179389] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.186728] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:25 2019][1696465.193272] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.200521] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.207781] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.215107] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.222868] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.230121] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.237880] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.245544] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:25 2019][1696465.252011] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:25 2019][1696465.258031] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:25 2019][1696465.264584] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.271399] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:25 2019][1696465.277844] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:25 2019][1696465.283524] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:25 2019][1696465.291114] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1696465.296176] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696465.302468] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1696465.309093] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696465.315366] osp-pre-1-2 S ffff9159f964e180 0 6513 2 0x00000080 [Wed Feb 6 09:39:25 2019][1696465.322667] Call Trace: [Wed Feb 6 09:39:25 2019][1696465.325297] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:25 2019][1696465.330709] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1696465.335874] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:25 2019][1696465.342840] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:25 2019][1696465.348620] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:25 2019][1696465.355330] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1696465.360408] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696465.366681] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1696465.373303] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696465.379592] osp-syn-22-2 S ffff9159a7238000 0 6514 2 0x00000080 [Wed Feb 6 09:39:25 2019][1696465.386871] Call Trace: [Wed Feb 6 09:39:25 2019][1696465.389499] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:25 2019][1696465.395592] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1696465.400751] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:25 2019][1696465.408066] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:25 2019][1696465.413853] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.421187] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:25 2019][1696465.427739] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.434987] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.442242] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.449572] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.457352] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.464592] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.472370] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.480031] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:25 2019][1696465.486488] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:25 2019][1696465.492504] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:25 2019][1696465.499050] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.505865] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:25 2019][1696465.512315] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:25 2019][1696465.517999] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:25 2019][1696465.525573] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1696465.530646] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696465.536914] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1696465.543531] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696465.549854] osp-pre-22-2 S ffff9159a7239040 0 6515 2 0x00000080 [Wed Feb 6 09:39:25 2019][1696465.557140] Call Trace: [Wed Feb 6 09:39:25 2019][1696465.559787] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:25 2019][1696465.565196] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1696465.570367] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:25 2019][1696465.577333] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:25 2019][1696465.583092] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:25 2019][1696465.589819] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1696465.594884] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696465.601158] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1696465.607790] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696465.614059] osp-syn-3-2 S ffff9159a723a080 0 6516 2 0x00000080 [Wed Feb 6 09:39:25 2019][1696465.621344] Call Trace: [Wed Feb 6 09:39:25 2019][1696465.623988] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:25 2019][1696465.630086] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1696465.635254] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:25 2019][1696465.642567] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:25 2019][1696465.648340] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.655675] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:25 2019][1696465.662221] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.669479] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.676719] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.684094] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.691963] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.699291] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.707104] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.714763] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:25 2019][1696465.721226] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:25 2019][1696465.727250] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:25 2019][1696465.733814] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:25 2019][1696465.740613] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:25 2019][1696465.747073] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:25 2019][1696465.752742] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:25 2019][1696465.760316] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:25 2019][1696465.765392] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696465.771671] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:25 2019][1696465.778305] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:25 2019][1696465.784582] osp-pre-3-2 S ffff9159a723b0c0 0 6517 2 0x00000080 [Wed Feb 6 09:39:25 2019][1696465.791883] Call Trace: [Wed Feb 6 09:39:25 2019][1696465.794512] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:25 2019][1696465.799912] [] schedule+0x29/0x70 [Wed Feb 6 09:39:25 2019][1696465.805080] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:25 2019][1696465.812049] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1696465.817808] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:26 2019][1696465.824528] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1696465.829592] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696465.835877] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1696465.842497] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696465.848784] osp-syn-27-2 S ffff9159a723c100 0 6518 2 0x00000080 [Wed Feb 6 09:39:26 2019][1696465.856080] Call Trace: [Wed Feb 6 09:39:26 2019][1696465.858721] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1696465.864817] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1696465.869980] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:26 2019][1696465.877292] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1696465.883065] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:26 2019][1696465.890401] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:26 2019][1696465.896948] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:26 2019][1696465.904205] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:26 2019][1696465.911447] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:26 2019][1696465.918788] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:26 2019][1696465.926550] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:26 2019][1696465.933801] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:26 2019][1696465.941565] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:26 2019][1696465.949231] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:26 2019][1696465.955690] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:26 2019][1696465.961715] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:26 2019][1696465.968280] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:26 2019][1696465.975076] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:26 2019][1696465.981534] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:26 2019][1696465.987206] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:26 2019][1696465.994801] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1696465.999872] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696466.006142] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1696466.012780] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696466.019054] osp-pre-27-2 S ffff9159a723d140 0 6519 2 0x00000080 [Wed Feb 6 09:39:26 2019][1696466.026330] Call Trace: [Wed Feb 6 09:39:26 2019][1696466.028977] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:26 2019][1696466.034381] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1696466.039548] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:26 2019][1696466.046515] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1696466.052283] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:26 2019][1696466.058992] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1696466.064066] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696466.070340] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1696466.076957] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696466.083245] osp-syn-37-2 S ffff9159a723e180 0 6520 2 0x00000080 [Wed Feb 6 09:39:26 2019][1696466.090538] Call Trace: [Wed Feb 6 09:39:26 2019][1696466.093169] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1696466.099268] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1696466.104434] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:26 2019][1696466.111748] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1696466.117518] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.124861] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:26 2019][1696466.131410] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.138651] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.145911] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.153245] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.161023] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.168262] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.176041] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.183699] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:26 2019][1696466.190147] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:26 2019][1696466.196187] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:26 2019][1696466.202735] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.209533] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:26 2019][1696466.215989] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:26 2019][1696466.221656] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:26 2019][1696466.229242] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1696466.234301] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696466.240576] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1696466.247208] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696466.253507] osp-pre-37-2 S ffff9159a1e48000 0 6521 2 0x00000080 [Wed Feb 6 09:39:26 2019][1696466.260805] Call Trace: [Wed Feb 6 09:39:26 2019][1696466.263434] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:26 2019][1696466.268835] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1696466.274003] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:26 2019][1696466.280967] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1696466.286717] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:26 2019][1696466.293441] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1696466.298507] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696466.304780] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1696466.311414] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696466.317693] osp-syn-36-2 S ffff9159a1e49040 0 6522 2 0x00000080 [Wed Feb 6 09:39:26 2019][1696466.324983] Call Trace: [Wed Feb 6 09:39:26 2019][1696466.327613] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1696466.333716] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1696466.338880] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:26 2019][1696466.346198] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1696466.351969] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.359305] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:26 2019][1696466.365850] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.373112] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.380357] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.387702] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.395463] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.402716] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.410474] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.418130] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:26 2019][1696466.424595] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:26 2019][1696466.430619] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:26 2019][1696466.437173] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.443988] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:26 2019][1696466.450435] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:26 2019][1696466.456120] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:26 2019][1696466.463694] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1696466.468771] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696466.475049] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1696466.481683] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696466.487949] osp-pre-36-2 S ffff9159a1e4a080 0 6523 2 0x00000080 [Wed Feb 6 09:39:26 2019][1696466.495235] Call Trace: [Wed Feb 6 09:39:26 2019][1696466.497865] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:26 2019][1696466.503277] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1696466.508443] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:26 2019][1696466.515435] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1696466.521209] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:26 2019][1696466.527912] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1696466.532992] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696466.539266] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1696466.545889] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696466.552177] osp-syn-8-2 S ffff9159a1e4b0c0 0 6524 2 0x00000080 [Wed Feb 6 09:39:26 2019][1696466.559448] Call Trace: [Wed Feb 6 09:39:26 2019][1696466.562077] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:26 2019][1696466.568175] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1696466.573343] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:26 2019][1696466.580656] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1696466.586436] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.593751] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:26 2019][1696466.600317] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.607558] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.614797] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.622145] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.629902] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.637144] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.644947] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.652609] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:26 2019][1696466.659070] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:26 2019][1696466.665092] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:26 2019][1696466.671666] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:26 2019][1696466.678466] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:26 2019][1696466.684932] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:26 2019][1696466.690608] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:26 2019][1696466.698199] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1696466.703255] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696466.709545] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:26 2019][1696466.716171] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696466.722458] osp-pre-8-2 S ffff9159a1e4c100 0 6525 2 0x00000080 [Wed Feb 6 09:39:26 2019][1696466.729736] Call Trace: [Wed Feb 6 09:39:26 2019][1696466.732366] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:26 2019][1696466.737788] [] schedule+0x29/0x70 [Wed Feb 6 09:39:26 2019][1696466.742934] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:26 2019][1696466.749897] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:26 2019][1696466.755663] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:26 2019][1696466.762373] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:26 2019][1696466.767434] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:26 2019][1696466.773736] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1696466.780365] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696466.786657] osp-syn-35-2 S ffff9159a1e4d140 0 6526 2 0x00000080 [Wed Feb 6 09:39:27 2019][1696466.793939] Call Trace: [Wed Feb 6 09:39:27 2019][1696466.796570] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:27 2019][1696466.802688] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1696466.807845] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:27 2019][1696466.815165] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1696466.820946] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:27 2019][1696466.828264] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:27 2019][1696466.834832] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:27 2019][1696466.842070] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:27 2019][1696466.849330] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:27 2019][1696466.856659] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:27 2019][1696466.864436] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:27 2019][1696466.871679] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:27 2019][1696466.879456] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:27 2019][1696466.887116] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:27 2019][1696466.893578] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:27 2019][1696466.899600] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:27 2019][1696466.906195] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:27 2019][1696466.912996] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:27 2019][1696466.919454] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:27 2019][1696466.925122] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:27 2019][1696466.932693] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1696466.937767] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696466.944041] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1696466.950662] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696466.956949] osp-pre-35-2 S ffff9159a1e4e180 0 6527 2 0x00000080 [Wed Feb 6 09:39:27 2019][1696466.964226] Call Trace: [Wed Feb 6 09:39:27 2019][1696466.966874] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:27 2019][1696466.972282] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1696466.977449] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:27 2019][1696466.984419] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1696466.990181] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:27 2019][1696466.996906] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1696467.001969] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.008258] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1696467.014883] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.021167] osp-syn-28-2 S ffff9159ac040000 0 6528 2 0x00000080 [Wed Feb 6 09:39:27 2019][1696467.028438] Call Trace: [Wed Feb 6 09:39:27 2019][1696467.031083] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:27 2019][1696467.037181] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1696467.042359] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:27 2019][1696467.049769] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1696467.055539] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.062874] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:27 2019][1696467.069417] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.076678] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.083917] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.091242] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.099021] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.106263] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.114029] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.121707] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:27 2019][1696467.128155] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:27 2019][1696467.134189] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:27 2019][1696467.140739] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.147540] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:27 2019][1696467.154003] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:27 2019][1696467.159676] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:27 2019][1696467.167257] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1696467.172331] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.178622] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1696467.185261] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.191528] osp-pre-28-2 S ffff9159ac041040 0 6529 2 0x00000080 [Wed Feb 6 09:39:27 2019][1696467.198803] Call Trace: [Wed Feb 6 09:39:27 2019][1696467.201435] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:27 2019][1696467.206858] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1696467.212007] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:27 2019][1696467.218979] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1696467.224751] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:27 2019][1696467.231462] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1696467.236535] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.242810] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1696467.249445] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.255715] osp-syn-30-2 S ffff9159ac042080 0 6530 2 0x00000080 [Wed Feb 6 09:39:27 2019][1696467.262995] Call Trace: [Wed Feb 6 09:39:27 2019][1696467.265632] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:27 2019][1696467.271746] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1696467.276892] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:27 2019][1696467.284205] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1696467.289988] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.297303] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:27 2019][1696467.303853] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.311138] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.318405] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.325747] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.333548] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.340897] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.348792] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.356497] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:27 2019][1696467.362960] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:27 2019][1696467.368980] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:27 2019][1696467.375538] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.382338] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:27 2019][1696467.388788] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:27 2019][1696467.394474] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:27 2019][1696467.402051] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1696467.407111] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.413399] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1696467.420019] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.426302] osp-pre-30-2 S ffff9159ac0430c0 0 6531 2 0x00000080 [Wed Feb 6 09:39:27 2019][1696467.433576] Call Trace: [Wed Feb 6 09:39:27 2019][1696467.436218] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:27 2019][1696467.441620] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1696467.446790] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:27 2019][1696467.453756] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1696467.459517] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:27 2019][1696467.466241] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1696467.471299] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.477587] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1696467.484242] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.490623] osp-syn-34-2 S ffff9159ac044100 0 6532 2 0x00000080 [Wed Feb 6 09:39:27 2019][1696467.498023] Call Trace: [Wed Feb 6 09:39:27 2019][1696467.500682] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:27 2019][1696467.506873] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1696467.512160] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Wed Feb 6 09:39:27 2019][1696467.519566] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1696467.525454] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.532941] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:27 2019][1696467.539576] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.546822] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.554079] [] llog_process_thread+0x892/0x15a0 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.561416] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.569189] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.576434] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.584212] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.591872] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:27 2019][1696467.598321] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:27 2019][1696467.604352] [] ? osp_sync_force+0x300/0x300 [osp] [Wed Feb 6 09:39:27 2019][1696467.610900] [] llog_cat_process+0x2e/0x30 [obdclass] [Wed Feb 6 09:39:27 2019][1696467.617719] [] osp_sync_thread+0x21f/0xa70 [osp] [Wed Feb 6 09:39:27 2019][1696467.624167] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:27 2019][1696467.629838] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Wed Feb 6 09:39:27 2019][1696467.637422] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1696467.642487] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.648778] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1696467.655401] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.661686] osp-pre-34-2 S ffff9159ac045140 0 6533 2 0x00000080 [Wed Feb 6 09:39:27 2019][1696467.668966] Call Trace: [Wed Feb 6 09:39:27 2019][1696467.671595] [] ? ktime_get+0x52/0xe0 [Wed Feb 6 09:39:27 2019][1696467.677021] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1696467.682173] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Wed Feb 6 09:39:27 2019][1696467.689142] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1696467.694912] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Wed Feb 6 09:39:27 2019][1696467.701620] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1696467.706693] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.712964] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1696467.719597] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.725867] mdt01_003 S ffff91599eb35140 0 6591 2 0x00000080 [Wed Feb 6 09:39:27 2019][1696467.733152] Call Trace: [Wed Feb 6 09:39:27 2019][1696467.735799] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1696467.740997] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:27 2019][1696467.747892] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1696467.753684] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:27 2019][1696467.760180] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:27 2019][1696467.767756] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1696467.772816] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.779105] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1696467.785729] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.792001] lquota_wb_fir-M S ffff91599eb30000 0 6593 2 0x00000080 [Wed Feb 6 09:39:27 2019][1696467.799292] Call Trace: [Wed Feb 6 09:39:27 2019][1696467.801924] [] schedule+0x29/0x70 [Wed Feb 6 09:39:27 2019][1696467.807070] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:27 2019][1696467.813098] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:27 2019][1696467.819633] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Wed Feb 6 09:39:27 2019][1696467.826435] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Wed Feb 6 09:39:27 2019][1696467.833068] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:27 2019][1696467.838835] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Wed Feb 6 09:39:27 2019][1696467.845197] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:27 2019][1696467.850272] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.856551] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:27 2019][1696467.863183] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:27 2019][1696467.869450] lquota_wb_fir-M S ffff91599eb36180 0 6595 2 0x00000080 [Wed Feb 6 09:39:27 2019][1696467.876736] Call Trace: [Wed Feb 6 09:39:28 2019][1696467.879368] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1696467.884517] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:28 2019][1696467.890547] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:28 2019][1696467.897092] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Wed Feb 6 09:39:28 2019][1696467.903893] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Wed Feb 6 09:39:28 2019][1696467.910525] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1696467.916288] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Wed Feb 6 09:39:28 2019][1696467.922669] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1696467.927734] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696467.934017] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1696467.940637] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696467.946921] dist_txn-2 S ffff91599eb31040 0 6596 2 0x00000080 [Wed Feb 6 09:39:28 2019][1696467.954205] Call Trace: [Wed Feb 6 09:39:28 2019][1696467.956855] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:39:28 2019][1696467.962955] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1696467.968165] [] distribute_txn_commit_thread+0x9ac/0xca0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696467.976002] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1696467.981807] [] ? sub_trans_commit_cb+0x20/0x20 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696467.988856] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1696467.993922] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.000210] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1696468.006834] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.013122] mdt_out01_002 S ffff9159f828c100 0 6610 2 0x00000080 [Wed Feb 6 09:39:28 2019][1696468.020412] Call Trace: [Wed Feb 6 09:39:28 2019][1696468.023062] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1696468.028239] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.035135] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1696468.040915] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.047407] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.054979] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1696468.060042] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.066329] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1696468.072955] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.079242] mdt_out01_003 S ffff9159f828d140 0 6611 2 0x00000080 [Wed Feb 6 09:39:28 2019][1696468.086531] Call Trace: [Wed Feb 6 09:39:28 2019][1696468.089181] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1696468.094359] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.101260] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1696468.107047] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.113539] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.121116] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1696468.126192] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.132470] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1696468.139104] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.145381] mdt_out01_004 S ffff9159f828e180 0 6612 2 0x00000080 [Wed Feb 6 09:39:28 2019][1696468.152679] Call Trace: [Wed Feb 6 09:39:28 2019][1696468.155315] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1696468.160496] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.167395] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1696468.173178] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.179665] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.187245] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1696468.192305] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.198595] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1696468.205220] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.211506] mdt00_003 S ffff91599eb330c0 0 6613 2 0x00000080 [Wed Feb 6 09:39:28 2019][1696468.218788] Call Trace: [Wed Feb 6 09:39:28 2019][1696468.221427] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1696468.226617] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.233501] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1696468.239283] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.245771] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.253347] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1696468.258422] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.264700] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1696468.271334] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.277614] mdt00_004 R running task 0 6615 2 0x00000080 [Wed Feb 6 09:39:28 2019][1696468.284918] Call Trace: [Wed Feb 6 09:39:28 2019][1696468.287583] [] ? ptlrpc_expire_one_request+0xd0/0x520 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.295252] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:28 2019][1696468.302522] [] ? ptlrpc_check_set.part.23+0x935/0x1df0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.310300] [] ? ptlrpc_check_set+0x5b/0xe0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.317135] [] ? ptlrpc_set_wait+0x57d/0x8d0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.324017] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1696468.329810] [] ? ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.336811] [] ? ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.343898] [] ? mdt_do_glimpse+0x1e9/0x4c0 [mdt] [Wed Feb 6 09:39:28 2019][1696468.350461] [] ? ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.358331] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.366263] [] ? mdt_glimpse_enqueue+0x3d3/0x4f0 [mdt] [Wed Feb 6 09:39:28 2019][1696468.373259] [] ? mdt_intent_glimpse+0x1f/0x30 [mdt] [Wed Feb 6 09:39:28 2019][1696468.379978] [] ? mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:28 2019][1696468.386793] [] ? mdt_intent_brw+0x30/0x30 [mdt] [Wed Feb 6 09:39:28 2019][1696468.393178] [] ? ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.400254] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:28 2019][1696468.407572] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:28 2019][1696468.414149] [] ? ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.421583] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.429277] [] ? tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.435758] [] ? tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.443033] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.450791] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:28 2019][1696468.458067] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.466100] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.473084] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:28 2019][1696468.479534] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:28 2019][1696468.485581] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.492247] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.499826] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1696468.505070] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.511342] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1696468.518150] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.524422] mdt00_005 S ffff91599eb32080 0 6616 2 0x00000080 [Wed Feb 6 09:39:28 2019][1696468.531722] Call Trace: [Wed Feb 6 09:39:28 2019][1696468.534350] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1696468.539534] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.546432] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1696468.552220] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.558716] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.566291] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1696468.571364] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.577635] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1696468.584269] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.590537] mdt02_003 S ffff913ac58159a0 0 6617 2 0x00000080 [Wed Feb 6 09:39:28 2019][1696468.597840] Call Trace: [Wed Feb 6 09:39:28 2019][1696468.600467] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1696468.605639] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.612533] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1696468.618321] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.624805] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.632390] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1696468.637455] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.643741] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1696468.650358] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.656646] mdt01_004 S ffff914aa94b1040 0 6618 2 0x00000080 [Wed Feb 6 09:39:28 2019][1696468.663933] Call Trace: [Wed Feb 6 09:39:28 2019][1696468.666564] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1696468.671759] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.678639] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1696468.684425] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.690916] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.698492] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1696468.703571] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.709848] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1696468.716470] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.722758] mdt03_003 S ffff914aa94b2080 0 6682 2 0x00000080 [Wed Feb 6 09:39:28 2019][1696468.730036] Call Trace: [Wed Feb 6 09:39:28 2019][1696468.732668] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1696468.737859] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.744739] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1696468.750522] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.757009] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.764582] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1696468.769640] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.775931] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1696468.782555] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.788840] mdt01_005 S ffff914aa94b30c0 0 6683 2 0x00000080 [Wed Feb 6 09:39:28 2019][1696468.796110] Call Trace: [Wed Feb 6 09:39:28 2019][1696468.798757] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1696468.803934] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.810833] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:28 2019][1696468.816620] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.823106] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.830690] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:28 2019][1696468.835762] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.842032] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:28 2019][1696468.848668] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:28 2019][1696468.854942] mdt03_004 S ffff914aa94b4100 0 6684 2 0x00000080 [Wed Feb 6 09:39:28 2019][1696468.862221] Call Trace: [Wed Feb 6 09:39:28 2019][1696468.864867] [] schedule+0x29/0x70 [Wed Feb 6 09:39:28 2019][1696468.870044] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:28 2019][1696468.876943] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1696468.882732] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696468.889223] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696468.896803] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1696468.901873] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1696468.908143] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1696468.914784] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1696468.921059] mdt02_004 S ffff914aa94b5140 0 6687 2 0x00000080 [Wed Feb 6 09:39:29 2019][1696468.928354] Call Trace: [Wed Feb 6 09:39:29 2019][1696468.931007] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1696468.936187] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696468.943085] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1696468.948865] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696468.955343] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696468.962938] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1696468.968001] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1696468.974290] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1696468.980916] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1696468.987203] mdt02_005 S ffff914aa94b6180 0 6688 2 0x00000080 [Wed Feb 6 09:39:29 2019][1696468.994481] Call Trace: [Wed Feb 6 09:39:29 2019][1696468.997122] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1696469.002296] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.009189] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1696469.014972] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.021451] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.029041] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1696469.034104] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1696469.040392] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1696469.047018] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1696469.053301] mdt00_006 D ffff9159b442a080 0 6690 2 0x00000080 [Wed Feb 6 09:39:29 2019][1696469.060573] Call Trace: [Wed Feb 6 09:39:29 2019][1696469.063216] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1696469.068360] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 09:39:29 2019][1696469.075003] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 09:39:29 2019][1696469.081883] [] down_write+0x2d/0x3d [Wed Feb 6 09:39:29 2019][1696469.087218] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 09:39:29 2019][1696469.094124] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 09:39:29 2019][1696469.101006] [] ? ___slab_alloc+0x24e/0x4f0 [Wed Feb 6 09:39:29 2019][1696469.106971] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:29 2019][1696469.114137] [] ? lu_buf_alloc+0x48/0x320 [obdclass] [Wed Feb 6 09:39:29 2019][1696469.120885] [] ? ldlm_cli_enqueue_local+0x27d/0x870 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.128379] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:39:29 2019][1696469.136411] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:39:29 2019][1696469.143818] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:39:29 2019][1696469.151066] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:39:29 2019][1696469.157786] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:29 2019][1696469.165128] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:39:29 2019][1696469.171761] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:39:29 2019][1696469.178425] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.185056] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:29 2019][1696469.191702] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:39:29 2019][1696469.198346] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.205250] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:29 2019][1696469.212569] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:29 2019][1696469.219148] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.226407] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.234116] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.240426] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.247527] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.255277] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:29 2019][1696469.262551] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.270413] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.277394] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:29 2019][1696469.283841] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:29 2019][1696469.289886] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.296380] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.303955] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1696469.309027] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1696469.315296] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1696469.321935] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1696469.328214] mdt02_006 S ffff9159b442e180 0 6694 2 0x00000080 [Wed Feb 6 09:39:29 2019][1696469.335508] Call Trace: [Wed Feb 6 09:39:29 2019][1696469.338159] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1696469.343343] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.350267] [] ? ptlrpc_at_set_timer+0x5b/0x110 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.357410] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1696469.363208] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.369683] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.377258] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1696469.382332] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1696469.388600] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1696469.395218] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1696469.401508] mdt03_005 D ffff9159b4429040 0 6758 2 0x00000080 [Wed Feb 6 09:39:29 2019][1696469.408799] Call Trace: [Wed Feb 6 09:39:29 2019][1696469.411441] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1696469.416587] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 09:39:29 2019][1696469.423224] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 09:39:29 2019][1696469.430108] [] down_write+0x2d/0x3d [Wed Feb 6 09:39:29 2019][1696469.435438] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 09:39:29 2019][1696469.442340] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 09:39:29 2019][1696469.449246] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:29 2019][1696469.456389] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1696469.462187] [] ? ldlm_cli_enqueue_local+0x27d/0x870 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.469682] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:39:29 2019][1696469.477716] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:39:29 2019][1696469.485128] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:39:29 2019][1696469.492372] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:39:29 2019][1696469.499088] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:29 2019][1696469.506425] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:39:29 2019][1696469.513053] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:39:29 2019][1696469.519709] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.526361] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:29 2019][1696469.532995] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:39:29 2019][1696469.539642] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.546544] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:29 2019][1696469.553859] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:29 2019][1696469.560430] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.567684] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.575385] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.581710] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.588796] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.596558] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:29 2019][1696469.603816] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.611687] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.618654] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:29 2019][1696469.625117] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:29 2019][1696469.631158] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.637641] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.645226] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:29 2019][1696469.650283] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1696469.656574] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:29 2019][1696469.663197] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:29 2019][1696469.669481] mdt03_006 D ffff9159b442d140 0 6759 2 0x00000080 [Wed Feb 6 09:39:29 2019][1696469.676754] Call Trace: [Wed Feb 6 09:39:29 2019][1696469.679397] [] schedule+0x29/0x70 [Wed Feb 6 09:39:29 2019][1696469.684541] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 09:39:29 2019][1696469.691182] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 09:39:29 2019][1696469.698066] [] down_write+0x2d/0x3d [Wed Feb 6 09:39:29 2019][1696469.703394] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 09:39:29 2019][1696469.710295] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 09:39:29 2019][1696469.717192] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:29 2019][1696469.724344] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:29 2019][1696469.730126] [] ? ldlm_cli_enqueue_local+0x27d/0x870 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.737623] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:39:29 2019][1696469.745658] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:39:29 2019][1696469.753059] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:39:29 2019][1696469.760298] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:39:29 2019][1696469.767035] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:29 2019][1696469.774358] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:39:29 2019][1696469.781004] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:39:29 2019][1696469.787657] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.794305] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:29 2019][1696469.800928] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:39:29 2019][1696469.807578] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.814486] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:29 2019][1696469.821807] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:29 2019][1696469.828389] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.835644] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.843351] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.849651] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:29 2019][1696469.856743] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696469.864510] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:30 2019][1696469.871770] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696469.879639] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696469.886619] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:30 2019][1696469.893070] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:30 2019][1696469.899124] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696469.905601] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696469.913190] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1696469.918253] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696469.924529] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1696469.931164] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696469.937435] mdt01_006 S ffff9159b4428000 0 6760 2 0x00000080 [Wed Feb 6 09:39:30 2019][1696469.944727] Call Trace: [Wed Feb 6 09:39:30 2019][1696469.947374] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1696469.952555] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696469.959451] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1696469.965230] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696469.971721] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696469.979295] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1696469.984371] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696469.990650] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1696469.997284] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.003556] mdt02_007 S ffff9159b442c100 0 6761 2 0x00000080 [Wed Feb 6 09:39:30 2019][1696470.010844] Call Trace: [Wed Feb 6 09:39:30 2019][1696470.013475] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1696470.018656] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.025547] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1696470.031337] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.037834] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.045409] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1696470.050481] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.056750] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1696470.063385] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.069654] ldlm_cn00_003 S ffff9159b442b0c0 0 6762 2 0x00000080 [Wed Feb 6 09:39:30 2019][1696470.076938] Call Trace: [Wed Feb 6 09:39:30 2019][1696470.079569] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1696470.084769] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.091653] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1696470.097437] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.103927] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.111506] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1696470.116582] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.122852] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1696470.129479] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.135761] ldlm_cn03_003 S ffff9159f33c8000 0 6763 2 0x00000080 [Wed Feb 6 09:39:30 2019][1696470.143054] Call Trace: [Wed Feb 6 09:39:30 2019][1696470.145690] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1696470.150886] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.157760] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1696470.163545] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.170032] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.177601] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1696470.182661] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.188950] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1696470.195570] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.201847] ldlm_cn02_004 S ffff9159f33c9040 0 6764 2 0x00000080 [Wed Feb 6 09:39:30 2019][1696470.209144] Call Trace: [Wed Feb 6 09:39:30 2019][1696470.211773] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1696470.216950] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.223845] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1696470.229626] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.236105] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.243691] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1696470.248749] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.255024] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1696470.261659] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.267936] ldlm_cn00_004 S ffff9159f33ca080 0 6765 2 0x00000080 [Wed Feb 6 09:39:30 2019][1696470.275232] Call Trace: [Wed Feb 6 09:39:30 2019][1696470.277869] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1696470.283050] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.289945] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1696470.295730] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.302222] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.309801] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1696470.314870] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.321159] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1696470.327780] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.334073] ldlm_cn01_004 S ffff9159f33cb0c0 0 6766 2 0x00000080 [Wed Feb 6 09:39:30 2019][1696470.341351] Call Trace: [Wed Feb 6 09:39:30 2019][1696470.343980] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1696470.349155] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.356048] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1696470.361827] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.368306] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.375899] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1696470.380961] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.387253] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1696470.393877] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.400152] ldlm_cn02_005 S ffff9159f33cc100 0 6767 2 0x00000080 [Wed Feb 6 09:39:30 2019][1696470.407458] Call Trace: [Wed Feb 6 09:39:30 2019][1696470.410091] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1696470.415271] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.422168] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1696470.427945] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.434435] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.442009] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1696470.447074] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.453362] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1696470.459987] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.466278] ldlm_cn01_005 S ffff9159f33cd140 0 6768 2 0x00000080 [Wed Feb 6 09:39:30 2019][1696470.473574] Call Trace: [Wed Feb 6 09:39:30 2019][1696470.476226] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1696470.481408] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.488307] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1696470.494096] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.500586] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.508168] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1696470.513243] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.519521] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1696470.526158] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.532434] ldlm_cn00_005 S ffff9159f33ce180 0 6769 2 0x00000080 [Wed Feb 6 09:39:30 2019][1696470.539737] Call Trace: [Wed Feb 6 09:39:30 2019][1696470.542374] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1696470.547554] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.554456] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1696470.560242] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.566740] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.574320] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1696470.579396] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.585676] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1696470.592309] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.598583] ldlm_cn03_004 S ffff9159a2760000 0 6770 2 0x00000080 [Wed Feb 6 09:39:30 2019][1696470.605879] Call Trace: [Wed Feb 6 09:39:30 2019][1696470.608511] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1696470.613692] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.620587] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1696470.626371] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.632864] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.640440] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1696470.645517] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.651794] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1696470.658431] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.664709] ldlm_cn03_005 S ffff9159a2761040 0 6771 2 0x00000080 [Wed Feb 6 09:39:30 2019][1696470.672010] Call Trace: [Wed Feb 6 09:39:30 2019][1696470.674640] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1696470.679820] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.686714] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1696470.692493] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.698983] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.706560] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1696470.711620] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.717909] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1696470.724528] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.730799] ldlm_cn01_006 S ffff9159a2762080 0 6772 2 0x00000080 [Wed Feb 6 09:39:30 2019][1696470.738095] Call Trace: [Wed Feb 6 09:39:30 2019][1696470.740729] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1696470.745904] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.752802] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1696470.758579] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.765067] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.772642] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1696470.777706] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.783992] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1696470.790608] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.796899] mdt_io00_003 S ffff9159a27630c0 0 6773 2 0x00000080 [Wed Feb 6 09:39:30 2019][1696470.804180] Call Trace: [Wed Feb 6 09:39:30 2019][1696470.806810] [] schedule+0x29/0x70 [Wed Feb 6 09:39:30 2019][1696470.811990] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.818889] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:30 2019][1696470.824674] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.831161] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:30 2019][1696470.838738] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:30 2019][1696470.843803] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.850086] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:30 2019][1696470.856706] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:30 2019][1696470.862995] mdt_io01_003 S ffff9159a2764100 0 6774 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696470.870282] Call Trace: [Wed Feb 6 09:39:31 2019][1696470.872925] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1696470.878096] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696470.884987] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1696470.890768] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696470.897246] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696470.904838] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1696470.909903] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696470.916192] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1696470.922817] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696470.929101] ldlm_cn02_006 S ffff9159a2766180 0 6776 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696470.936378] Call Trace: [Wed Feb 6 09:39:31 2019][1696470.939024] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1696470.944202] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696470.951100] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1696470.956880] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696470.963371] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696470.970945] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1696470.976018] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696470.982300] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1696470.988933] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696470.995209] ldlm_cn00_006 S ffff9159a2765140 0 6777 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696471.002501] Call Trace: [Wed Feb 6 09:39:31 2019][1696471.005131] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1696471.010307] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.017204] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1696471.022991] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.029479] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.037057] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1696471.042131] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.048405] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1696471.055045] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.061317] mdt_io02_003 S ffff9159f774c100 0 6779 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696471.068598] Call Trace: [Wed Feb 6 09:39:31 2019][1696471.071241] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1696471.076415] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.083307] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1696471.089091] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.095581] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.103153] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1696471.108207] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.114498] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1696471.121125] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.127400] mdt_rdpg00_002 S ffff9159f774d140 0 6781 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696471.134697] Call Trace: [Wed Feb 6 09:39:31 2019][1696471.137331] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1696471.142505] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.149402] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1696471.155185] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.161663] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.169259] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1696471.174320] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.180609] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1696471.187226] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.193514] mdt_io02_004 S ffff9159f774e180 0 6782 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696471.200782] Call Trace: [Wed Feb 6 09:39:31 2019][1696471.203424] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1696471.208604] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.215502] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1696471.221287] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.227781] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.235354] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1696471.240415] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.246703] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1696471.253328] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.259615] mdt_io01_004 S ffff9159ed231040 0 6783 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696471.266884] Call Trace: [Wed Feb 6 09:39:31 2019][1696471.269513] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1696471.274705] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.281590] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1696471.287375] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.293863] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.301439] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1696471.306510] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.312781] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1696471.319415] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.325684] mdt_out00_002 S ffff9159ed2330c0 0 6784 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696471.332962] Call Trace: [Wed Feb 6 09:39:31 2019][1696471.335590] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1696471.340787] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.347668] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1696471.353452] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.359939] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.367514] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1696471.372590] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.378865] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1696471.385504] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.391780] mdt_io01_005 S ffff9159ed234100 0 6785 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696471.399076] Call Trace: [Wed Feb 6 09:39:31 2019][1696471.401713] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1696471.406907] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.413782] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1696471.419568] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.426059] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.433630] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1696471.438689] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.444978] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1696471.451599] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.457888] mdt_io00_004 S ffff9159ed236180 0 6786 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696471.465165] Call Trace: [Wed Feb 6 09:39:31 2019][1696471.467812] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1696471.472985] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.479878] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1696471.485665] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.492152] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.499729] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1696471.504802] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.511073] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1696471.517709] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.523988] mdt_rdpg00_003 S ffff9159ed235140 0 6787 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696471.531276] Call Trace: [Wed Feb 6 09:39:31 2019][1696471.533907] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1696471.539091] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.545988] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1696471.551779] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.558273] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.565849] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1696471.570907] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.577194] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1696471.583811] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.590086] mdt_io01_006 S ffff9159ed232080 0 6788 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696471.597382] Call Trace: [Wed Feb 6 09:39:31 2019][1696471.600012] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1696471.605189] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.612079] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1696471.617856] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.624351] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.631924] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1696471.636984] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.643273] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1696471.649890] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.656180] mdt00_007 S ffff9159f1f60000 0 6795 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696471.663464] Call Trace: [Wed Feb 6 09:39:31 2019][1696471.666109] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1696471.671283] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.678172] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1696471.683952] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.690448] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.698028] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1696471.703089] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.709378] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1696471.716002] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.722291] mdt02_008 S ffff9139f6232080 0 6804 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696471.729576] Call Trace: [Wed Feb 6 09:39:31 2019][1696471.732223] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1696471.737404] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.744301] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1696471.750083] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.756571] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.764147] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1696471.769218] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.775490] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1696471.782125] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.788396] mdt_io00_005 S ffff9139f6235140 0 6805 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696471.795695] Call Trace: [Wed Feb 6 09:39:31 2019][1696471.798328] [] schedule+0x29/0x70 [Wed Feb 6 09:39:31 2019][1696471.803507] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.810408] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:39:31 2019][1696471.817377] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:31 2019][1696471.823175] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.829649] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:31 2019][1696471.837237] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:31 2019][1696471.842295] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.848570] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:31 2019][1696471.855205] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:31 2019][1696471.861476] mdt00_008 S ffff9139f62330c0 0 6806 2 0x00000080 [Wed Feb 6 09:39:31 2019][1696471.868776] Call Trace: [Wed Feb 6 09:39:32 2019][1696471.871404] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696471.876582] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696471.883479] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1696471.889260] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696471.895738] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696471.903324] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1696471.908388] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696471.914677] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1696471.921304] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696471.927591] mdt01_007 S ffff9139f6236180 0 6813 2 0x00000080 [Wed Feb 6 09:39:32 2019][1696471.934858] Call Trace: [Wed Feb 6 09:39:32 2019][1696471.937490] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696471.942686] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696471.949562] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1696471.955349] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696471.961837] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696471.969415] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1696471.974486] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696471.980755] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1696471.987394] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696471.993666] mdt03_007 S ffff9139f6234100 0 6814 2 0x00000080 [Wed Feb 6 09:39:32 2019][1696472.000936] Call Trace: [Wed Feb 6 09:39:32 2019][1696472.003565] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696472.008761] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.015644] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1696472.021435] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.027922] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.035495] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1696472.040575] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.046848] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1696472.053471] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.059762] mdt02_009 S ffff9159f1f66180 0 6827 2 0x00000080 [Wed Feb 6 09:39:32 2019][1696472.067048] Call Trace: [Wed Feb 6 09:39:32 2019][1696472.069678] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696472.074854] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.081752] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1696472.087533] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.094009] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.101593] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1696472.106649] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.112928] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1696472.119562] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.125832] mdt03_008 S ffff9159f1f64100 0 6845 2 0x00000080 [Wed Feb 6 09:39:32 2019][1696472.133122] Call Trace: [Wed Feb 6 09:39:32 2019][1696472.135754] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696472.140932] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.147827] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1696472.153608] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.160087] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.167674] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1696472.172737] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.179026] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1696472.185651] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.191936] mdt_rdpg02_002 S ffff9159f1f61040 0 6854 2 0x00000080 [Wed Feb 6 09:39:32 2019][1696472.199221] Call Trace: [Wed Feb 6 09:39:32 2019][1696472.201872] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696472.207049] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.213942] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1696472.219722] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.226200] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.233794] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1696472.238857] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.245142] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1696472.251757] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.258044] mdt_rdpg03_002 S ffff9159f1f630c0 0 6861 2 0x00000080 [Wed Feb 6 09:39:32 2019][1696472.265320] Call Trace: [Wed Feb 6 09:39:32 2019][1696472.267949] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696472.273133] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.280031] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1696472.285819] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.292314] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.299891] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1696472.304968] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.311244] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1696472.317878] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.324157] mdt_io03_003 S ffff9159f1f65140 0 6862 2 0x00000080 [Wed Feb 6 09:39:32 2019][1696472.331459] Call Trace: [Wed Feb 6 09:39:32 2019][1696472.334098] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696472.339277] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.346179] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1696472.351962] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.358452] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.366024] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1696472.371098] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.377373] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1696472.383993] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.390281] mdt_io00_006 S ffff9159f72d8000 0 6863 2 0x00000080 [Wed Feb 6 09:39:32 2019][1696472.397568] Call Trace: [Wed Feb 6 09:39:32 2019][1696472.400212] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696472.405384] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.412277] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1696472.418055] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.424528] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.432114] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1696472.437174] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.443450] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1696472.450079] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.456354] mdt_io03_004 S ffff9159f72d9040 0 6864 2 0x00000080 [Wed Feb 6 09:39:32 2019][1696472.463652] Call Trace: [Wed Feb 6 09:39:32 2019][1696472.466281] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696472.471462] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.478353] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1696472.484134] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.490617] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.498212] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1696472.503275] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.509571] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1696472.516185] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.522469] mdt_io01_007 S ffff9159f72da080 0 6865 2 0x00000080 [Wed Feb 6 09:39:32 2019][1696472.529748] Call Trace: [Wed Feb 6 09:39:32 2019][1696472.532378] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696472.537558] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.544453] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1696472.550235] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.556723] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.564299] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1696472.569374] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.575652] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1696472.582288] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.588566] mdt_io01_008 S ffff9159f72db0c0 0 6866 2 0x00000080 [Wed Feb 6 09:39:32 2019][1696472.595855] Call Trace: [Wed Feb 6 09:39:32 2019][1696472.598489] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696472.603670] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.610569] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1696472.616355] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.622843] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.630418] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1696472.635491] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.641759] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1696472.648396] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.654671] mdt_io01_009 S ffff9159f72dc100 0 6867 2 0x00000080 [Wed Feb 6 09:39:32 2019][1696472.661951] Call Trace: [Wed Feb 6 09:39:32 2019][1696472.664580] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696472.669778] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.676662] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1696472.682459] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.688938] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.696529] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1696472.701591] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.707867] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1696472.714500] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.720776] mdt_io02_005 S ffff9159f72dd140 0 6868 2 0x00000080 [Wed Feb 6 09:39:32 2019][1696472.728070] Call Trace: [Wed Feb 6 09:39:32 2019][1696472.730699] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696472.735872] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.742769] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1696472.748552] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.755033] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.762621] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1696472.767682] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.773976] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1696472.780600] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.786870] mdt_io03_005 S ffff9159f72de180 0 6869 2 0x00000080 [Wed Feb 6 09:39:32 2019][1696472.794167] Call Trace: [Wed Feb 6 09:39:32 2019][1696472.796806] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696472.801984] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.808873] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:32 2019][1696472.814651] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.821140] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:32 2019][1696472.828715] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:32 2019][1696472.833776] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.840074] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:32 2019][1696472.846694] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:32 2019][1696472.852965] mdt_rdpg01_002 S ffff9159fc2c8000 0 6870 2 0x00000080 [Wed Feb 6 09:39:32 2019][1696472.860256] Call Trace: [Wed Feb 6 09:39:32 2019][1696472.862888] [] schedule+0x29/0x70 [Wed Feb 6 09:39:32 2019][1696472.868070] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696472.874967] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1696472.880745] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696472.887237] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696472.894811] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1696472.899872] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696472.906161] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1696472.912786] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696472.919090] mdt00_009 S ffff9159fc2c9040 0 6883 2 0x00000080 [Wed Feb 6 09:39:33 2019][1696472.926363] Call Trace: [Wed Feb 6 09:39:33 2019][1696472.929001] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1696472.934177] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696472.941073] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1696472.946857] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696472.953334] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696472.960920] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1696472.965983] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696472.972268] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1696472.978889] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696472.985175] mdt_io00_007 S ffff9159fc2cb0c0 0 6909 2 0x00000080 [Wed Feb 6 09:39:33 2019][1696472.992466] Call Trace: [Wed Feb 6 09:39:33 2019][1696472.995117] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1696473.000292] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.007188] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1696473.012965] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.019454] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.027029] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1696473.032093] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696473.038384] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1696473.045006] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696473.051293] mdt03_009 S ffff9159ff3a2080 0 6951 2 0x00000080 [Wed Feb 6 09:39:33 2019][1696473.058564] Call Trace: [Wed Feb 6 09:39:33 2019][1696473.061207] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1696473.066387] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.073286] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1696473.079072] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.085560] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.093134] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1696473.098207] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696473.104480] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1696473.111117] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696473.117392] mdt01_008 S ffff9159ff3a0000 0 6952 2 0x00000080 [Wed Feb 6 09:39:33 2019][1696473.124694] Call Trace: [Wed Feb 6 09:39:33 2019][1696473.127332] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:39:33 2019][1696473.134244] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.142003] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1696473.147173] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.154264] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.161348] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:33 2019][1696473.168508] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1696473.174293] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.181630] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:39:33 2019][1696473.188975] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.196994] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:33 2019][1696473.203717] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:39:33 2019][1696473.210703] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:33 2019][1696473.217433] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.225477] [] ? htable_lookup+0x8b/0x130 [obdclass] [Wed Feb 6 09:39:33 2019][1696473.232290] [] ? lu_object_find_at+0x1b3/0x280 [obdclass] [Wed Feb 6 09:39:33 2019][1696473.239530] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:33 2019][1696473.246693] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:39:33 2019][1696473.253780] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.260456] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.267539] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.274705] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:39:33 2019][1696473.281421] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:33 2019][1696473.288070] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.294976] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:39:33 2019][1696473.301801] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.308712] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:33 2019][1696473.316036] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:33 2019][1696473.322601] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.329870] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.337574] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.343899] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.350990] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.358760] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:33 2019][1696473.366012] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.373890] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.380854] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:33 2019][1696473.387298] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:33 2019][1696473.393357] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.399835] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.407429] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1696473.412492] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696473.418765] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1696473.425398] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696473.431667] mdt01_009 S ffff9159ff3a5140 0 6985 2 0x00000080 [Wed Feb 6 09:39:33 2019][1696473.438935] Call Trace: [Wed Feb 6 09:39:33 2019][1696473.441570] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1696473.446759] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.453640] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1696473.459423] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.465918] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.473490] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1696473.478550] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696473.484837] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1696473.491454] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696473.497745] mdt02_010 S ffff9159ff3a4100 0 7029 2 0x00000080 [Wed Feb 6 09:39:33 2019][1696473.505014] Call Trace: [Wed Feb 6 09:39:33 2019][1696473.507641] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1696473.512819] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.519715] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1696473.525503] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.531985] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.539572] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1696473.544637] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696473.550924] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1696473.557541] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696473.563829] mdt01_010 D ffff9149f3e30000 0 7081 2 0x00000080 [Wed Feb 6 09:39:33 2019][1696473.571097] Call Trace: [Wed Feb 6 09:39:33 2019][1696473.573728] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1696473.578893] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 09:39:33 2019][1696473.585521] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 09:39:33 2019][1696473.592410] [] down_write+0x2d/0x3d [Wed Feb 6 09:39:33 2019][1696473.597753] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 09:39:33 2019][1696473.604645] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 09:39:33 2019][1696473.611566] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:33 2019][1696473.618714] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1696473.624506] [] ? ldlm_cli_enqueue_local+0x27d/0x870 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.632007] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:39:33 2019][1696473.640038] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:39:33 2019][1696473.647452] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:39:33 2019][1696473.654700] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:39:33 2019][1696473.661414] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:33 2019][1696473.668750] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:39:33 2019][1696473.675374] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:39:33 2019][1696473.682030] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.688678] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:33 2019][1696473.695328] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.702231] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:39:33 2019][1696473.708879] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.715784] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:33 2019][1696473.723098] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:33 2019][1696473.729656] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.736927] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.744621] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.750933] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.758035] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.765795] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:33 2019][1696473.773069] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.780933] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.787914] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:33 2019][1696473.794361] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:33 2019][1696473.800421] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.806900] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.814479] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:33 2019][1696473.819556] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696473.825831] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:33 2019][1696473.832467] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:33 2019][1696473.838735] mdt_fld_0002 S ffff9149f3e330c0 0 7088 2 0x00000080 [Wed Feb 6 09:39:33 2019][1696473.846021] Call Trace: [Wed Feb 6 09:39:33 2019][1696473.848650] [] schedule+0x29/0x70 [Wed Feb 6 09:39:33 2019][1696473.853850] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.860734] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:33 2019][1696473.866519] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:33 2019][1696473.873010] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696473.880583] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1696473.885657] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696473.891936] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1696473.898573] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696473.904849] ldlm_bl_03 R running task 0 7089 2 0x00000080 [Wed Feb 6 09:39:34 2019][1696473.912151] Call Trace: [Wed Feb 6 09:39:34 2019][1696473.914806] [] ? ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696473.923444] [] ? ldlm_cancel_locks_for_export_cb+0x3f/0x50 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696473.931561] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [Wed Feb 6 09:39:34 2019][1696473.939157] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696473.947798] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696473.956442] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [Wed Feb 6 09:39:34 2019][1696473.963958] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696473.971569] [] ? ldlm_bl_thread_main+0x4c8/0x700 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696473.978810] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1696473.984595] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696473.992175] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1696473.997424] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.003691] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1696474.010478] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.016768] mdt_out02_002 S ffff9149f3e36180 0 7090 2 0x00000080 [Wed Feb 6 09:39:34 2019][1696474.024059] Call Trace: [Wed Feb 6 09:39:34 2019][1696474.026699] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1696474.031876] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.038774] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1696474.044553] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.051041] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.058615] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1696474.063674] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.069968] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1696474.076592] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.082867] ldlm_cb01_002 S ffff9149f3e34100 0 7091 2 0x00000080 [Wed Feb 6 09:39:34 2019][1696474.090170] Call Trace: [Wed Feb 6 09:39:34 2019][1696474.092808] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1696474.097985] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.104884] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1696474.110662] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.117136] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.124729] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1696474.129788] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.136066] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1696474.142699] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.148978] mdt_out00_003 S ffff9159f6ad8000 0 7092 2 0x00000080 [Wed Feb 6 09:39:34 2019][1696474.156260] Call Trace: [Wed Feb 6 09:39:34 2019][1696474.158906] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1696474.164082] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.170981] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1696474.176766] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.183244] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.190841] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1696474.195902] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.202187] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1696474.208802] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.215090] ldlm_cb00_002 S ffff9159f6ad9040 0 7093 2 0x00000080 [Wed Feb 6 09:39:34 2019][1696474.222371] Call Trace: [Wed Feb 6 09:39:34 2019][1696474.225004] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1696474.230183] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.237084] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1696474.242870] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.249357] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.256934] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1696474.262009] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.268284] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1696474.274923] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.281201] mdt_io00_008 S ffff9159f6ada080 0 7094 2 0x00000080 [Wed Feb 6 09:39:34 2019][1696474.288486] Call Trace: [Wed Feb 6 09:39:34 2019][1696474.291136] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1696474.296308] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.303200] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1696474.308977] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.315452] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.323039] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1696474.328097] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.334373] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1696474.341012] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.347284] mdt_io01_010 S ffff9159f6adb0c0 0 7095 2 0x00000080 [Wed Feb 6 09:39:34 2019][1696474.354579] Call Trace: [Wed Feb 6 09:39:34 2019][1696474.357214] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1696474.362395] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.369298] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1696474.375081] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.381557] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.389145] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1696474.394207] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.400492] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1696474.407108] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.413394] mdt_io02_006 S ffff9159f6adc100 0 7096 2 0x00000080 [Wed Feb 6 09:39:34 2019][1696474.420669] Call Trace: [Wed Feb 6 09:39:34 2019][1696474.423302] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1696474.428479] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.435373] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1696474.441159] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.447634] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.455219] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1696474.460286] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.466570] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1696474.473186] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.479475] mdt_io00_009 S ffff9159f6add140 0 7097 2 0x00000080 [Wed Feb 6 09:39:34 2019][1696474.486744] Call Trace: [Wed Feb 6 09:39:34 2019][1696474.489379] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1696474.494558] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.501458] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1696474.507244] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.513733] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.521308] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1696474.526382] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.532661] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1696474.539283] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.545574] mdt_io02_007 S ffff9159f6ade180 0 7098 2 0x00000080 [Wed Feb 6 09:39:34 2019][1696474.552856] Call Trace: [Wed Feb 6 09:39:34 2019][1696474.555500] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1696474.560680] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.567579] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1696474.573360] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.579853] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.587434] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1696474.592511] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.598789] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1696474.605426] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.611699] mdt_io02_008 S ffff9159f7c00000 0 7099 2 0x00000080 [Wed Feb 6 09:39:34 2019][1696474.618967] Call Trace: [Wed Feb 6 09:39:34 2019][1696474.621598] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1696474.626792] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.633670] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1696474.639454] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.645946] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.653522] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1696474.658582] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.664871] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1696474.671496] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.677782] mdt_io00_010 S ffff9159f7c01040 0 7100 2 0x00000080 [Wed Feb 6 09:39:34 2019][1696474.685052] Call Trace: [Wed Feb 6 09:39:34 2019][1696474.687681] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1696474.692876] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.699758] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1696474.705543] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.712030] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.719605] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1696474.724680] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.730949] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1696474.737584] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.743859] mdt_io02_009 S ffff9159f7c02080 0 7101 2 0x00000080 [Wed Feb 6 09:39:34 2019][1696474.751148] Call Trace: [Wed Feb 6 09:39:34 2019][1696474.753780] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1696474.758969] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.765849] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1696474.771633] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.778123] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.785695] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1696474.790754] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.797042] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1696474.803665] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.809953] ldlm_cb02_002 S ffff9159f7c030c0 0 7108 2 0x00000080 [Wed Feb 6 09:39:34 2019][1696474.817223] Call Trace: [Wed Feb 6 09:39:34 2019][1696474.819852] [] schedule+0x29/0x70 [Wed Feb 6 09:39:34 2019][1696474.825035] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.831936] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:34 2019][1696474.837722] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.844210] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:34 2019][1696474.851786] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:34 2019][1696474.856864] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.863136] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:34 2019][1696474.869770] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:34 2019][1696474.876039] mdt_io02_010 S ffff9159f7c04100 0 7109 2 0x00000080 [Wed Feb 6 09:39:35 2019][1696474.883306] Call Trace: [Wed Feb 6 09:39:35 2019][1696474.885951] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1696474.891129] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696474.898029] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1696474.903816] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696474.910303] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696474.917879] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1696474.922955] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696474.929234] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1696474.935867] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696474.942148] mdt_io02_011 S ffff9159f7c05140 0 7110 2 0x00000080 [Wed Feb 6 09:39:35 2019][1696474.949448] Call Trace: [Wed Feb 6 09:39:35 2019][1696474.952085] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1696474.957261] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696474.964154] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1696474.969933] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696474.976408] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696474.983996] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1696474.989060] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696474.995346] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1696475.001966] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.008254] mdt_rdpg02_003 S ffff9159f7c06180 0 7111 2 0x00000080 [Wed Feb 6 09:39:35 2019][1696475.015538] Call Trace: [Wed Feb 6 09:39:35 2019][1696475.018181] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1696475.023356] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.030253] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1696475.036032] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.042512] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.050098] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1696475.055163] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.061450] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1696475.068076] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.074361] mdt_rdpg01_003 S ffff91599d3c0000 0 7112 2 0x00000080 [Wed Feb 6 09:39:35 2019][1696475.081635] Call Trace: [Wed Feb 6 09:39:35 2019][1696475.084277] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1696475.089452] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.096342] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1696475.102122] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.108610] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.116186] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1696475.121257] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.127531] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1696475.134162] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.140435] mdt_io01_011 S ffff9149f3d94100 0 7120 2 0x00000080 [Wed Feb 6 09:39:35 2019][1696475.147736] Call Trace: [Wed Feb 6 09:39:35 2019][1696475.150364] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1696475.155547] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.162445] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1696475.168234] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.174717] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.182293] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1696475.187371] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.193640] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1696475.200260] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.206549] mdt03_010 S ffff9149f3d90000 0 7135 2 0x00000080 [Wed Feb 6 09:39:35 2019][1696475.213818] Call Trace: [Wed Feb 6 09:39:35 2019][1696475.216450] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1696475.221630] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.228524] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1696475.234309] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.240791] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.248379] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1696475.253439] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.259729] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1696475.266344] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.272619] mdt00_010 S ffff9129fb99c100 0 7191 2 0x00000080 [Wed Feb 6 09:39:35 2019][1696475.279912] Call Trace: [Wed Feb 6 09:39:35 2019][1696475.282548] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1696475.287727] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.294626] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1696475.300411] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.306889] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.314467] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1696475.319528] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.325799] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1696475.332422] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.338697] ldlm_cb01_003 S ffff9129fb999040 0 7214 2 0x00000080 [Wed Feb 6 09:39:35 2019][1696475.345998] Call Trace: [Wed Feb 6 09:39:35 2019][1696475.348630] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1696475.353810] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.360711] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1696475.366495] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.372980] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.380561] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1696475.385622] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.391897] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1696475.398530] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.404802] mdt_io00_011 S ffff9129fb99b0c0 0 7215 2 0x00000080 [Wed Feb 6 09:39:35 2019][1696475.412103] Call Trace: [Wed Feb 6 09:39:35 2019][1696475.414732] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1696475.419913] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.426813] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1696475.432595] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.439088] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.446662] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1696475.451724] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.458011] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1696475.464625] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.470905] mdt_io02_012 S ffff9129fb998000 0 7222 2 0x00000080 [Wed Feb 6 09:39:35 2019][1696475.478193] Call Trace: [Wed Feb 6 09:39:35 2019][1696475.480826] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1696475.486004] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.492901] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1696475.498682] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.505157] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.512746] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1696475.517808] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.524085] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1696475.530718] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.536986] mdt_io01_012 S ffff91599d3c6180 0 7223 2 0x00000080 [Wed Feb 6 09:39:35 2019][1696475.544270] Call Trace: [Wed Feb 6 09:39:35 2019][1696475.546917] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1696475.552100] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.559000] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1696475.564783] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.571257] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.578843] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1696475.583898] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.590170] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1696475.596805] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.603084] mdt_io03_006 S ffff9149ba354100 0 7248 2 0x00000080 [Wed Feb 6 09:39:35 2019][1696475.610365] Call Trace: [Wed Feb 6 09:39:35 2019][1696475.613006] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1696475.618181] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.625076] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1696475.630865] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.637351] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.644941] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1696475.650006] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.656293] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1696475.662920] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.669194] mdt01_011 S ffff91599d3c4100 0 7415 2 0x00000080 [Wed Feb 6 09:39:35 2019][1696475.676493] Call Trace: [Wed Feb 6 09:39:35 2019][1696475.679122] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1696475.684302] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.691193] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1696475.696973] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.698559] Lustre: fir-MDT0002: Client 54802b78-b8a3-6ef2-1c4c-1f299e55d7f1 (at 10.9.103.14@o2ib4) reconnecting [Wed Feb 6 09:39:35 2019][1696475.698561] Lustre: Skipped 20 previous similar messages [Wed Feb 6 09:39:35 2019][1696475.719308] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.726882] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1696475.731943] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.738226] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1696475.744845] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.751133] mdt_rdpg01_004 S ffff91599d3c1040 0 7470 2 0x00000080 [Wed Feb 6 09:39:35 2019][1696475.758414] Call Trace: [Wed Feb 6 09:39:35 2019][1696475.761045] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1696475.766222] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.773117] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1696475.778902] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.785380] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.792972] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1696475.798034] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.804311] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1696475.810944] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.817215] mdt_rdpg01_005 S ffff91599d3c30c0 0 7471 2 0x00000080 [Wed Feb 6 09:39:35 2019][1696475.824493] Call Trace: [Wed Feb 6 09:39:35 2019][1696475.827143] [] schedule+0x29/0x70 [Wed Feb 6 09:39:35 2019][1696475.832314] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.839207] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:35 2019][1696475.844983] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.851462] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:35 2019][1696475.859044] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:35 2019][1696475.864101] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:35 2019][1696475.870388] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:35 2019][1696475.877004] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1696475.883283] mdt00_011 D ffff9159ef230000 0 7500 2 0x00000080 [Wed Feb 6 09:39:36 2019][1696475.890595] Call Trace: [Wed Feb 6 09:39:36 2019][1696475.893257] [] ? keys_fini+0xb1/0x1d0 [obdclass] [Wed Feb 6 09:39:36 2019][1696475.899705] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1696475.904876] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 09:39:36 2019][1696475.911520] [] ? ptlrpc_request_cache_free+0x90/0x1e0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696475.919179] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 09:39:36 2019][1696475.926076] [] down_write+0x2d/0x3d [Wed Feb 6 09:39:36 2019][1696475.931414] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 09:39:36 2019][1696475.938307] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 09:39:36 2019][1696475.945202] [] ? ___slab_alloc+0x209/0x4f0 [Wed Feb 6 09:39:36 2019][1696475.951147] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:36 2019][1696475.958324] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696475.966275] [] ? lu_buf_alloc+0x48/0x320 [obdclass] [Wed Feb 6 09:39:36 2019][1696475.973013] [] ? ldlm_cli_enqueue_local+0x27d/0x870 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696475.980521] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:39:36 2019][1696475.988531] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:39:36 2019][1696475.995946] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:39:36 2019][1696476.003190] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:39:36 2019][1696476.009910] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:36 2019][1696476.017249] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:39:36 2019][1696476.023882] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:39:36 2019][1696476.030538] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.037188] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:36 2019][1696476.043820] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:39:36 2019][1696476.050463] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.057367] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:36 2019][1696476.064683] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:36 2019][1696476.071244] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.078516] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.086212] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.092541] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.099633] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.107387] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:36 2019][1696476.114664] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.122532] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.129503] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:36 2019][1696476.135959] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:36 2019][1696476.141998] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.148486] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.156060] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1696476.161117] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1696476.167407] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1696476.174029] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1696476.180306] ldlm_cn00_007 S ffff9159efebc100 0 7739 2 0x00000080 [Wed Feb 6 09:39:36 2019][1696476.187606] Call Trace: [Wed Feb 6 09:39:36 2019][1696476.190235] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1696476.195414] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.202309] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1696476.208091] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.214568] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.222157] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1696476.227219] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1696476.233507] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1696476.240123] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1696476.246397] mdt03_011 S ffff9159fc2ca080 0 8573 2 0x00000080 [Wed Feb 6 09:39:36 2019][1696476.253702] Call Trace: [Wed Feb 6 09:39:36 2019][1696476.256346] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:39:36 2019][1696476.263258] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.271020] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1696476.276196] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.283291] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.290368] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:36 2019][1696476.297523] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1696476.303306] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.310628] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:39:36 2019][1696476.317985] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.326009] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:36 2019][1696476.332731] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:39:36 2019][1696476.339722] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:36 2019][1696476.346461] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.354479] [] ? lod_attr_get+0xab/0x130 [lod] [Wed Feb 6 09:39:36 2019][1696476.360777] [] ? mdd_la_get+0x84/0x1a0 [mdd] [Wed Feb 6 09:39:36 2019][1696476.366894] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:36 2019][1696476.374046] [] ? mdt_attr_get_complex+0x4ec/0x850 [mdt] [Wed Feb 6 09:39:36 2019][1696476.381121] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:39:36 2019][1696476.388200] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.394849] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.401944] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.409096] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:39:36 2019][1696476.415813] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:36 2019][1696476.422453] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:39:36 2019][1696476.429267] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.436156] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:36 2019][1696476.443497] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:36 2019][1696476.450057] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.457316] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.465043] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.471356] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.478443] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.486215] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:36 2019][1696476.493466] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.501327] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.508307] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:36 2019][1696476.514749] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:36 2019][1696476.520794] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.527287] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.534872] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1696476.539931] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1696476.546215] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1696476.552834] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1696476.559105] ldlm_cn01_007 S ffff9159fc2cc100 0 8616 2 0x00000080 [Wed Feb 6 09:39:36 2019][1696476.566394] Call Trace: [Wed Feb 6 09:39:36 2019][1696476.569021] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1696476.574198] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.581095] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1696476.586884] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.593362] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.600950] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1696476.606010] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1696476.612291] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1696476.618907] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1696476.625184] mdt02_011 S ffff9159ff280000 0 8829 2 0x00000080 [Wed Feb 6 09:39:36 2019][1696476.632477] Call Trace: [Wed Feb 6 09:39:36 2019][1696476.635109] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1696476.640287] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.647181] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1696476.652967] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.659456] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.667027] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1696476.672100] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1696476.678375] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1696476.684997] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1696476.691280] mdt00_012 S ffff9159ff2830c0 0 8929 2 0x00000080 [Wed Feb 6 09:39:36 2019][1696476.698563] Call Trace: [Wed Feb 6 09:39:36 2019][1696476.701194] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1696476.706372] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.713286] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1696476.719069] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.725550] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.733134] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1696476.738189] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1696476.744458] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1696476.751094] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1696476.757364] mdt01_012 S ffff9159ff286180 0 8946 2 0x00000080 [Wed Feb 6 09:39:36 2019][1696476.764636] Call Trace: [Wed Feb 6 09:39:36 2019][1696476.767282] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1696476.772455] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.779346] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1696476.785125] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.791604] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.799198] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:36 2019][1696476.804259] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1696476.810550] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:36 2019][1696476.817173] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:36 2019][1696476.823450] mdt01_013 S ffff9159ff282080 0 8947 2 0x00000080 [Wed Feb 6 09:39:36 2019][1696476.830740] Call Trace: [Wed Feb 6 09:39:36 2019][1696476.833370] [] schedule+0x29/0x70 [Wed Feb 6 09:39:36 2019][1696476.838548] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.845442] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:36 2019][1696476.851228] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:36 2019][1696476.857713] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696476.865304] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1696476.870364] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696476.876635] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1696476.883269] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696476.889544] mdt01_014 S ffff9159ff281040 0 8948 2 0x00000080 [Wed Feb 6 09:39:37 2019][1696476.896819] Call Trace: [Wed Feb 6 09:39:37 2019][1696476.899478] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1696476.904654] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696476.911553] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1696476.917340] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696476.923826] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696476.931419] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1696476.936480] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696476.942768] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1696476.949386] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696476.955663] mdt01_015 S ffff9159ff285140 0 8949 2 0x00000080 [Wed Feb 6 09:39:37 2019][1696476.962964] Call Trace: [Wed Feb 6 09:39:37 2019][1696476.965601] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:39:37 2019][1696476.972515] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696476.980285] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1696476.985461] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696476.992539] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696476.999633] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:37 2019][1696477.006772] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1696477.012552] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.019913] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:39:37 2019][1696477.027249] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.035267] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:37 2019][1696477.041999] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:39:37 2019][1696477.048976] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:37 2019][1696477.055717] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.063758] [] ? htable_lookup+0x8b/0x130 [obdclass] [Wed Feb 6 09:39:37 2019][1696477.070570] [] ? lu_object_find_at+0x1b3/0x280 [obdclass] [Wed Feb 6 09:39:37 2019][1696477.077816] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:37 2019][1696477.084961] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:39:37 2019][1696477.092046] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.098715] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.105803] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.112958] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:39:37 2019][1696477.119692] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:37 2019][1696477.126338] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.133231] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:39:37 2019][1696477.140065] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.146954] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:37 2019][1696477.154275] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:37 2019][1696477.160866] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.168126] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.175831] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.182161] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.189253] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.197019] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:37 2019][1696477.204275] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.212149] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.219120] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:37 2019][1696477.225568] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:37 2019][1696477.231641] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.238120] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.245696] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1696477.250772] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.257041] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1696477.263655] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.269948] mdt01_016 S ffff91599d3d2080 0 8950 2 0x00000080 [Wed Feb 6 09:39:37 2019][1696477.277235] Call Trace: [Wed Feb 6 09:39:37 2019][1696477.279874] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1696477.285047] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.291959] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1696477.297735] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.304210] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.311796] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1696477.316855] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.323132] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1696477.329766] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.336043] mdt_io01_013 S ffff91599d3d30c0 0 8951 2 0x00000080 [Wed Feb 6 09:39:37 2019][1696477.343334] Call Trace: [Wed Feb 6 09:39:37 2019][1696477.345964] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1696477.351142] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.358034] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1696477.363813] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.370291] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.377884] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1696477.382949] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.389234] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1696477.395855] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.402129] mdt_io01_014 S ffff91599d3d1040 0 8952 2 0x00000080 [Wed Feb 6 09:39:37 2019][1696477.409418] Call Trace: [Wed Feb 6 09:39:37 2019][1696477.412048] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1696477.417224] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.424124] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1696477.429911] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.436403] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.443977] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1696477.449030] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.455326] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1696477.461945] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.468222] mdt_io01_015 S ffff91599d3d4100 0 8953 2 0x00000080 [Wed Feb 6 09:39:37 2019][1696477.475538] Call Trace: [Wed Feb 6 09:39:37 2019][1696477.478167] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1696477.483338] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.490230] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1696477.496008] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.502482] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.510074] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1696477.515145] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.521420] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1696477.528054] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.534345] mdt_io01_016 S ffff91599d3d0000 0 8954 2 0x00000080 [Wed Feb 6 09:39:37 2019][1696477.541616] Call Trace: [Wed Feb 6 09:39:37 2019][1696477.544260] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1696477.549432] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.556323] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1696477.562106] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.568587] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.576174] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1696477.581236] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.587525] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1696477.594156] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.600423] mdt_io01_017 S ffff91599d3d5140 0 8955 2 0x00000080 [Wed Feb 6 09:39:37 2019][1696477.607719] Call Trace: [Wed Feb 6 09:39:37 2019][1696477.610351] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1696477.615529] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.622429] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1696477.628216] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.634699] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.642285] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1696477.647347] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.653620] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1696477.660256] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.666529] mdt_io01_018 S ffff91599d3d6180 0 8956 2 0x00000080 [Wed Feb 6 09:39:37 2019][1696477.673819] Call Trace: [Wed Feb 6 09:39:37 2019][1696477.676462] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1696477.681637] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.688529] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1696477.694305] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.700784] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.708370] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1696477.713434] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.719724] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1696477.726348] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.732619] mdt_io01_019 S ffff9159efebe180 0 8957 2 0x00000080 [Wed Feb 6 09:39:37 2019][1696477.739905] Call Trace: [Wed Feb 6 09:39:37 2019][1696477.742535] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1696477.747713] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.754604] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1696477.760383] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.766861] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.774460] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1696477.779518] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.785805] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1696477.792420] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.798690] mdt_io01_020 S ffff9159efebb0c0 0 8958 2 0x00000080 [Wed Feb 6 09:39:37 2019][1696477.805981] Call Trace: [Wed Feb 6 09:39:37 2019][1696477.808611] [] schedule+0x29/0x70 [Wed Feb 6 09:39:37 2019][1696477.813792] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.820683] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:37 2019][1696477.826471] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.832984] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:37 2019][1696477.840557] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:37 2019][1696477.845617] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.851900] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:37 2019][1696477.858514] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:37 2019][1696477.864793] mdt_rdpg00_004 S ffff9159efeb8000 0 8959 2 0x00000080 [Wed Feb 6 09:39:38 2019][1696477.872093] Call Trace: [Wed Feb 6 09:39:38 2019][1696477.874721] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1696477.879891] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696477.886786] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1696477.892584] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696477.899063] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696477.906653] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1696477.911716] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696477.917989] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1696477.924626] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696477.930904] mdt_io00_012 S ffff9159efebd140 0 8960 2 0x00000080 [Wed Feb 6 09:39:38 2019][1696477.938186] Call Trace: [Wed Feb 6 09:39:38 2019][1696477.940815] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1696477.946011] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696477.952904] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1696477.958683] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696477.965173] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696477.972748] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1696477.977809] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696477.984096] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1696477.990714] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696477.996988] mdt_io00_013 S ffff9159efeba080 0 8961 2 0x00000080 [Wed Feb 6 09:39:38 2019][1696478.004291] Call Trace: [Wed Feb 6 09:39:38 2019][1696478.006920] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1696478.012095] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.018991] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1696478.024780] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.031261] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.038849] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1696478.043913] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.050193] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1696478.056827] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.063097] mdt_io02_013 S ffff9159efeb9040 0 8962 2 0x00000080 [Wed Feb 6 09:39:38 2019][1696478.070401] Call Trace: [Wed Feb 6 09:39:38 2019][1696478.073033] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1696478.078207] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.085102] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1696478.090890] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.097373] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.104959] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1696478.110024] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.116308] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1696478.122927] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.129217] mdt_io02_014 S ffff912a3f25d140 0 8963 2 0x00000080 [Wed Feb 6 09:39:38 2019][1696478.136521] Call Trace: [Wed Feb 6 09:39:38 2019][1696478.139151] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1696478.144328] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.151219] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1696478.156997] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.163472] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.171059] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1696478.176116] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.182392] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1696478.189026] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.195293] mdt_out02_003 S ffff912a3f25c100 0 8966 2 0x00000080 [Wed Feb 6 09:39:38 2019][1696478.202585] Call Trace: [Wed Feb 6 09:39:38 2019][1696478.205218] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1696478.210395] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.217288] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1696478.223068] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.229553] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.237146] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1696478.242212] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.248499] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1696478.255123] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.261416] mdt00_013 S ffff912a3f25b0c0 0 8967 2 0x00000080 [Wed Feb 6 09:39:38 2019][1696478.268698] Call Trace: [Wed Feb 6 09:39:38 2019][1696478.271339] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1696478.276517] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.283408] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1696478.289187] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.295678] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.303252] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1696478.308323] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.314597] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1696478.321232] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.327509] mdt00_014 S ffff912a3f259040 0 8968 2 0x00000080 [Wed Feb 6 09:39:38 2019][1696478.334809] Call Trace: [Wed Feb 6 09:39:38 2019][1696478.337439] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1696478.342620] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.349513] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1696478.355299] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.361786] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.369362] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1696478.374453] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.380726] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1696478.387357] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.393630] mdt00_015 S ffff912a3f258000 0 8969 2 0x00000080 [Wed Feb 6 09:39:38 2019][1696478.400919] Call Trace: [Wed Feb 6 09:39:38 2019][1696478.403549] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1696478.408728] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.415623] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1696478.421411] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.427893] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.435479] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1696478.440544] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.446834] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1696478.453455] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.459741] mdt01_017 S ffff912a3f25a080 0 8970 2 0x00000080 [Wed Feb 6 09:39:38 2019][1696478.467014] Call Trace: [Wed Feb 6 09:39:38 2019][1696478.469660] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1696478.474840] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.481733] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1696478.487518] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.493992] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.501592] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1696478.506654] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.512935] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1696478.519574] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.525851] mdt01_018 D ffff9129fb166180 0 8971 2 0x00000080 [Wed Feb 6 09:39:38 2019][1696478.533154] Call Trace: [Wed Feb 6 09:39:38 2019][1696478.535789] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1696478.540943] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 09:39:38 2019][1696478.547578] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 09:39:38 2019][1696478.554461] [] down_write+0x2d/0x3d [Wed Feb 6 09:39:38 2019][1696478.559806] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 09:39:38 2019][1696478.566697] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 09:39:38 2019][1696478.573595] [] ? ___slab_alloc+0x209/0x4f0 [Wed Feb 6 09:39:38 2019][1696478.579552] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:38 2019][1696478.586703] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:38 2019][1696478.592476] [] ? lu_buf_alloc+0x48/0x320 [obdclass] [Wed Feb 6 09:39:38 2019][1696478.599220] [] ? ldlm_cli_enqueue_local+0x27d/0x870 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.606712] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:39:38 2019][1696478.614743] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:39:38 2019][1696478.622148] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:39:38 2019][1696478.629398] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:39:38 2019][1696478.636116] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:38 2019][1696478.643446] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:39:38 2019][1696478.650084] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:39:38 2019][1696478.656735] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.663379] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:38 2019][1696478.670023] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.676916] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:39:38 2019][1696478.683575] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.690457] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:38 2019][1696478.697778] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:38 2019][1696478.704339] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.711608] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.719307] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.725629] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.732718] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.740487] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:38 2019][1696478.747744] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.755613] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.762606] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:38 2019][1696478.769054] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:38 2019][1696478.775099] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.781591] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.789170] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:38 2019][1696478.794244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.800523] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:38 2019][1696478.807158] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:38 2019][1696478.813435] mdt01_019 S ffff9129fb1630c0 0 8972 2 0x00000080 [Wed Feb 6 09:39:38 2019][1696478.820729] Call Trace: [Wed Feb 6 09:39:38 2019][1696478.823363] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:39:38 2019][1696478.830278] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.838020] [] schedule+0x29/0x70 [Wed Feb 6 09:39:38 2019][1696478.843196] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.850292] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:39:38 2019][1696478.857373] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:38 2019][1696478.864516] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:39 2019][1696478.870312] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696478.877632] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:39:39 2019][1696478.884977] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696478.893028] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:39 2019][1696478.899749] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:39:39 2019][1696478.906740] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:39 2019][1696478.913479] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696478.921502] [] ? lod_attr_get+0xab/0x130 [lod] [Wed Feb 6 09:39:39 2019][1696478.927800] [] ? mdd_la_get+0x84/0x1a0 [mdd] [Wed Feb 6 09:39:39 2019][1696478.933911] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:39 2019][1696478.941067] [] ? mdt_attr_get_complex+0x4ec/0x850 [mdt] [Wed Feb 6 09:39:39 2019][1696478.948144] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:39:39 2019][1696478.955231] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696478.961886] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696478.968991] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696478.976139] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:39:39 2019][1696478.982857] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:39 2019][1696478.989499] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:39:39 2019][1696478.996316] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.003203] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:39 2019][1696479.010539] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:39 2019][1696479.017102] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.024361] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.032089] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.038400] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.045485] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.053248] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:39 2019][1696479.060504] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.068366] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.075351] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:39 2019][1696479.081797] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:39 2019][1696479.087856] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.094332] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.101931] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1696479.106994] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1696479.113280] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1696479.119892] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1696479.126180] mdt01_020 D ffff9129fb165140 0 8973 2 0x00000080 [Wed Feb 6 09:39:39 2019][1696479.133452] Call Trace: [Wed Feb 6 09:39:39 2019][1696479.136083] [] schedule+0x29/0x70 [Wed Feb 6 09:39:39 2019][1696479.141234] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 09:39:39 2019][1696479.147872] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 09:39:39 2019][1696479.154755] [] down_write+0x2d/0x3d [Wed Feb 6 09:39:39 2019][1696479.160116] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 09:39:39 2019][1696479.167012] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 09:39:39 2019][1696479.173893] [] ? ___slab_alloc+0x209/0x4f0 [Wed Feb 6 09:39:39 2019][1696479.179855] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:39 2019][1696479.186991] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:39 2019][1696479.192762] [] ? lu_buf_alloc+0x48/0x320 [obdclass] [Wed Feb 6 09:39:39 2019][1696479.199509] [] ? ldlm_cli_enqueue_local+0x27d/0x870 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.207005] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:39:39 2019][1696479.215035] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:39:39 2019][1696479.222451] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:39:39 2019][1696479.229714] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:39:39 2019][1696479.236433] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:39 2019][1696479.243757] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:39:39 2019][1696479.250398] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:39:39 2019][1696479.257043] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.263673] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:39 2019][1696479.270312] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:39:39 2019][1696479.276956] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.283842] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:39 2019][1696479.291175] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:39 2019][1696479.297735] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.305005] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.312700] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.319017] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.326105] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.333877] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:39 2019][1696479.341132] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.349013] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.355979] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:39 2019][1696479.362433] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:39 2019][1696479.368505] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.374988] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.382579] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1696479.387640] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1696479.393916] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1696479.400552] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1696479.406820] mdt_io00_014 S ffff9159a7b90000 0 8974 2 0x00000080 [Wed Feb 6 09:39:39 2019][1696479.414100] Call Trace: [Wed Feb 6 09:39:39 2019][1696479.416736] [] schedule+0x29/0x70 [Wed Feb 6 09:39:39 2019][1696479.421928] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.428827] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:39 2019][1696479.434611] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.441102] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.448676] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1696479.453737] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1696479.460023] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1696479.466639] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1696479.472912] mdt_io00_015 S ffff9159a7b91040 0 8975 2 0x00000080 [Wed Feb 6 09:39:39 2019][1696479.480197] Call Trace: [Wed Feb 6 09:39:39 2019][1696479.482849] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Wed Feb 6 09:39:39 2019][1696479.490167] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Wed Feb 6 09:39:39 2019][1696479.497071] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Wed Feb 6 09:39:39 2019][1696479.504126] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Wed Feb 6 09:39:39 2019][1696479.510842] [] ? lnet_ni_send+0x44/0xd0 [lnet] [Wed Feb 6 09:39:39 2019][1696479.517140] [] ? lnet_send+0x82/0x1c0 [lnet] [Wed Feb 6 09:39:39 2019][1696479.523250] [] ? LNetPut+0x2cc/0xb50 [lnet] [Wed Feb 6 09:39:39 2019][1696479.529300] [] ? ptl_send_buf+0x146/0x530 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.535945] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:39 2019][1696479.543201] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.550289] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.556939] [] ? update_curr+0x14c/0x1e0 [Wed Feb 6 09:39:39 2019][1696479.562698] [] ? account_entity_dequeue+0xae/0xd0 [Wed Feb 6 09:39:39 2019][1696479.569247] [] ? dequeue_entity+0x11c/0x5e0 [Wed Feb 6 09:39:39 2019][1696479.575259] [] ? dequeue_task_fair+0x41e/0x660 [Wed Feb 6 09:39:39 2019][1696479.581526] [] ? sched_clock_cpu+0x85/0xc0 [Wed Feb 6 09:39:39 2019][1696479.587470] [] ? tracing_is_on+0x15/0x30 [Wed Feb 6 09:39:39 2019][1696479.593223] [] ? tracing_record_cmdline+0x1d/0x120 [Wed Feb 6 09:39:39 2019][1696479.599843] [] ? probe_sched_switch+0x2d/0xa0 [Wed Feb 6 09:39:39 2019][1696479.606045] [] ? schedule+0x29/0x70 [Wed Feb 6 09:39:39 2019][1696479.611403] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.618456] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:39 2019][1696479.624258] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.630908] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.638486] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1696479.643737] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1696479.650013] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1696479.656807] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1696479.663099] mdt_io00_016 S ffff9159a7b92080 0 8976 2 0x00000080 [Wed Feb 6 09:39:39 2019][1696479.670389] Call Trace: [Wed Feb 6 09:39:39 2019][1696479.673025] [] schedule+0x29/0x70 [Wed Feb 6 09:39:39 2019][1696479.678202] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.685094] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:39 2019][1696479.690870] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.697344] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.704931] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1696479.709986] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1696479.716256] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1696479.722890] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1696479.729161] ldlm_cb03_002 S ffff9159a7b930c0 0 8977 2 0x00000080 [Wed Feb 6 09:39:39 2019][1696479.736441] Call Trace: [Wed Feb 6 09:39:39 2019][1696479.739086] [] schedule+0x29/0x70 [Wed Feb 6 09:39:39 2019][1696479.744264] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.751161] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:39 2019][1696479.756939] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.763425] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.770997] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1696479.776058] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1696479.782347] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1696479.788969] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1696479.795244] mdt02_012 S ffff9159a7b94100 0 8978 2 0x00000080 [Wed Feb 6 09:39:39 2019][1696479.802553] Call Trace: [Wed Feb 6 09:39:39 2019][1696479.805184] [] schedule+0x29/0x70 [Wed Feb 6 09:39:39 2019][1696479.810362] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.817260] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:39:39 2019][1696479.824229] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:39 2019][1696479.830013] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.836510] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:39 2019][1696479.844081] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:39 2019][1696479.849147] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1696479.855431] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:39 2019][1696479.862048] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:39 2019][1696479.868336] mdt_io02_015 S ffff9159a7b95140 0 8979 2 0x00000080 [Wed Feb 6 09:39:40 2019][1696479.875608] Call Trace: [Wed Feb 6 09:39:40 2019][1696479.878239] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1696479.883420] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696479.890321] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1696479.896107] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696479.902588] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696479.910174] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1696479.915236] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696479.921510] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1696479.928141] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696479.934416] mdt_io02_016 S ffff9159a7b96180 0 8980 2 0x00000080 [Wed Feb 6 09:39:40 2019][1696479.941699] Call Trace: [Wed Feb 6 09:39:40 2019][1696479.944351] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1696479.949531] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696479.956434] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1696479.962218] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696479.968695] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696479.976293] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1696479.981359] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696479.987630] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1696479.994259] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.000530] mdt02_013 S ffff9159e7e28000 0 8981 2 0x00000080 [Wed Feb 6 09:39:40 2019][1696480.007815] Call Trace: [Wed Feb 6 09:39:40 2019][1696480.010455] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1696480.015632] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.022533] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1696480.028318] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.034788] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.042395] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1696480.047453] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.053728] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1696480.060362] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.066634] mdt02_014 S ffff9159e7e29040 0 8982 2 0x00000080 [Wed Feb 6 09:39:40 2019][1696480.073906] Call Trace: [Wed Feb 6 09:39:40 2019][1696480.076553] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1696480.081732] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.088626] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1696480.094410] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.100887] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.108474] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1696480.113538] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.119827] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1696480.126451] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.132739] mdt02_015 S ffff9159e7e2a080 0 8983 2 0x00000080 [Wed Feb 6 09:39:40 2019][1696480.140020] Call Trace: [Wed Feb 6 09:39:40 2019][1696480.142660] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1696480.147839] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.154737] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1696480.160525] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.167018] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.174596] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1696480.179658] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.185947] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1696480.192568] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.198860] mdt02_016 S ffff9159e7e2b0c0 0 8984 2 0x00000080 [Wed Feb 6 09:39:40 2019][1696480.206153] Call Trace: [Wed Feb 6 09:39:40 2019][1696480.208783] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1696480.213956] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.220870] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1696480.226650] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.233124] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.240712] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1696480.245777] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.252051] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1696480.258687] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.264957] mdt00_016 S ffff9159e7e2c100 0 8985 2 0x00000080 [Wed Feb 6 09:39:40 2019][1696480.272247] Call Trace: [Wed Feb 6 09:39:40 2019][1696480.274877] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1696480.280071] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.286953] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1696480.292737] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.299226] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.306796] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1696480.311850] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.318140] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1696480.324755] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.331034] mdt_io00_017 S ffff9159e7e2d140 0 8986 2 0x00000080 [Wed Feb 6 09:39:40 2019][1696480.338332] Call Trace: [Wed Feb 6 09:39:40 2019][1696480.340962] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1696480.346142] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.353032] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1696480.358809] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.365284] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.372875] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1696480.377940] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.384214] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1696480.390849] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.397123] mdt_io00_018 S ffff9159e7e2e180 0 8993 2 0x00000080 [Wed Feb 6 09:39:40 2019][1696480.404418] Call Trace: [Wed Feb 6 09:39:40 2019][1696480.407052] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1696480.412229] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.419125] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1696480.424908] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.431396] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.438972] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1696480.444042] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.450312] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1696480.456938] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.463240] mdt00_017 S ffff9159e7f78000 0 8994 2 0x00000080 [Wed Feb 6 09:39:40 2019][1696480.470532] Call Trace: [Wed Feb 6 09:39:40 2019][1696480.473183] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1696480.478359] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.485259] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1696480.491047] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.497542] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.505117] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1696480.510190] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.516461] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1696480.523109] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.529384] mdt00_018 S ffff9159e7f79040 0 8995 2 0x00000080 [Wed Feb 6 09:39:40 2019][1696480.536683] Call Trace: [Wed Feb 6 09:39:40 2019][1696480.539313] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1696480.544494] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.551386] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1696480.557174] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.563660] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.571237] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1696480.576310] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.582595] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1696480.589217] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.595505] mdt01_021 S ffff9159e7f7a080 0 8996 2 0x00000080 [Wed Feb 6 09:39:40 2019][1696480.602776] Call Trace: [Wed Feb 6 09:39:40 2019][1696480.605405] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1696480.610582] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.617480] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1696480.623265] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.629741] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.637328] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1696480.642388] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.648680] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1696480.655305] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.661579] mdt01_022 S ffff9159e7f7b0c0 0 8997 2 0x00000080 [Wed Feb 6 09:39:40 2019][1696480.668882] Call Trace: [Wed Feb 6 09:39:40 2019][1696480.671518] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1696480.676697] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.683592] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:40 2019][1696480.689378] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.695858] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.703461] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:40 2019][1696480.708520] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.714796] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:40 2019][1696480.721430] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:40 2019][1696480.727707] mdt01_023 D ffff9159e7f7c100 0 8998 2 0x00000080 [Wed Feb 6 09:39:40 2019][1696480.734990] Call Trace: [Wed Feb 6 09:39:40 2019][1696480.737620] [] schedule+0x29/0x70 [Wed Feb 6 09:39:40 2019][1696480.742783] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 09:39:40 2019][1696480.749401] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 09:39:40 2019][1696480.756285] [] down_write+0x2d/0x3d [Wed Feb 6 09:39:40 2019][1696480.761642] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 09:39:40 2019][1696480.768538] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 09:39:40 2019][1696480.775438] [] ? ___slab_alloc+0x24e/0x4f0 [Wed Feb 6 09:39:40 2019][1696480.781391] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:40 2019][1696480.788564] [] ? lu_buf_alloc+0x48/0x320 [obdclass] [Wed Feb 6 09:39:40 2019][1696480.795300] [] ? ldlm_cli_enqueue_local+0x27d/0x870 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.802795] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:39:40 2019][1696480.810829] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:39:40 2019][1696480.818241] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:39:40 2019][1696480.825492] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:39:40 2019][1696480.832223] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:40 2019][1696480.839543] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:39:40 2019][1696480.846187] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:39:40 2019][1696480.852834] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:40 2019][1696480.859460] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:40 2019][1696480.866108] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:39:40 2019][1696480.872756] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696480.879644] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:41 2019][1696480.886979] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:41 2019][1696480.893538] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696480.900795] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696480.908508] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696480.914818] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696480.921905] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696480.929674] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:41 2019][1696480.936927] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696480.944787] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696480.951766] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:41 2019][1696480.958216] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:41 2019][1696480.964256] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696480.970758] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696480.978325] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1696480.983402] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696480.989670] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1696480.996295] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.002580] mdt00_019 S ffff9159e7f7d140 0 8999 2 0x00000080 [Wed Feb 6 09:39:41 2019][1696481.009872] Call Trace: [Wed Feb 6 09:39:41 2019][1696481.012503] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1696481.017681] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.024581] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1696481.030382] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.036856] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.044442] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1696481.049497] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.055777] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1696481.062410] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.068679] mdt_io03_007 S ffff9159e7f7e180 0 9000 2 0x00000080 [Wed Feb 6 09:39:41 2019][1696481.075953] Call Trace: [Wed Feb 6 09:39:41 2019][1696481.078602] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1696481.083785] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.090693] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1696481.096475] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.102968] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.110541] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1696481.115604] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.121893] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1696481.128514] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.134790] mdt_io01_021 S ffff9159a7660000 0 9001 2 0x00000080 [Wed Feb 6 09:39:41 2019][1696481.142081] Call Trace: [Wed Feb 6 09:39:41 2019][1696481.144714] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1696481.149892] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.156787] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1696481.162572] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.169051] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.176638] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1696481.181695] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.187971] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1696481.194607] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.200886] mdt_io00_019 S ffff9159a7661040 0 9002 2 0x00000080 [Wed Feb 6 09:39:41 2019][1696481.208187] Call Trace: [Wed Feb 6 09:39:41 2019][1696481.210817] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1696481.215999] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.222893] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1696481.228669] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.235150] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.242743] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1696481.247808] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.254097] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1696481.260722] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.266996] mdt_io00_020 S ffff9159a7662080 0 9003 2 0x00000080 [Wed Feb 6 09:39:41 2019][1696481.274296] Call Trace: [Wed Feb 6 09:39:41 2019][1696481.276925] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1696481.282110] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.289014] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1696481.294793] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.301283] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.308857] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1696481.313918] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.320209] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1696481.326828] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.333143] mdt01_024 S ffff9159a76630c0 0 9004 2 0x00000080 [Wed Feb 6 09:39:41 2019][1696481.340424] Call Trace: [Wed Feb 6 09:39:41 2019][1696481.343053] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1696481.348222] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.355115] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1696481.360894] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.367372] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.374959] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1696481.380020] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.386310] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1696481.392950] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.399224] mdt01_025 S ffff9159a7664100 0 9005 2 0x00000080 [Wed Feb 6 09:39:41 2019][1696481.406511] Call Trace: [Wed Feb 6 09:39:41 2019][1696481.409139] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1696481.414317] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.421213] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1696481.426998] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.433480] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.441065] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1696481.446120] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.452414] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1696481.459033] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.465303] mdt02_017 S ffff9159a7665140 0 9006 2 0x00000080 [Wed Feb 6 09:39:41 2019][1696481.472605] Call Trace: [Wed Feb 6 09:39:41 2019][1696481.475236] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1696481.480414] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.487303] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1696481.493082] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.499570] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.507148] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1696481.512207] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.518496] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1696481.525114] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.531399] mdt_io02_017 S ffff9159a7666180 0 9007 2 0x00000080 [Wed Feb 6 09:39:41 2019][1696481.538670] Call Trace: [Wed Feb 6 09:39:41 2019][1696481.541314] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1696481.546489] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.553383] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1696481.559169] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.565653] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.573252] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1696481.578308] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.584596] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1696481.591212] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.597487] mdt01_026 S ffff9159a4ea8000 0 9008 2 0x00000080 [Wed Feb 6 09:39:41 2019][1696481.604784] Call Trace: [Wed Feb 6 09:39:41 2019][1696481.607413] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1696481.612592] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.619486] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1696481.625273] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.631769] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.639341] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:41 2019][1696481.644415] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.650690] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:41 2019][1696481.657310] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:41 2019][1696481.663600] mdt01_027 S ffff9159a4ea9040 0 9009 2 0x00000080 [Wed Feb 6 09:39:41 2019][1696481.670887] Call Trace: [Wed Feb 6 09:39:41 2019][1696481.673537] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:39:41 2019][1696481.680444] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.688189] [] schedule+0x29/0x70 [Wed Feb 6 09:39:41 2019][1696481.693362] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.700448] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.707531] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:41 2019][1696481.714680] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:41 2019][1696481.720464] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.727787] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:39:41 2019][1696481.735137] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.743161] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:41 2019][1696481.749886] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:39:41 2019][1696481.756868] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:41 2019][1696481.763611] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.771634] [] ? lod_attr_get+0xab/0x130 [lod] [Wed Feb 6 09:39:41 2019][1696481.777920] [] ? mdd_la_get+0x84/0x1a0 [mdd] [Wed Feb 6 09:39:41 2019][1696481.784033] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:41 2019][1696481.791191] [] ? mdt_attr_get_complex+0x4ec/0x850 [mdt] [Wed Feb 6 09:39:41 2019][1696481.798253] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:39:41 2019][1696481.805352] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.812012] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.819100] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.826258] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:39:41 2019][1696481.832984] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:41 2019][1696481.839620] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:39:41 2019][1696481.846459] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.853351] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:41 2019][1696481.860675] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:41 2019][1696481.867248] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:41 2019][1696481.874513] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696481.882221] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696481.888547] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696481.895646] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696481.903409] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:42 2019][1696481.910672] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696481.918546] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696481.925521] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:42 2019][1696481.931975] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:42 2019][1696481.938040] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696481.944528] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696481.952111] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:42 2019][1696481.957173] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1696481.963453] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:42 2019][1696481.970073] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1696481.976348] mdt01_028 S ffff9159a4eaa080 0 9010 2 0x00000080 [Wed Feb 6 09:39:42 2019][1696481.983638] Call Trace: [Wed Feb 6 09:39:42 2019][1696481.986271] [] schedule+0x29/0x70 [Wed Feb 6 09:39:42 2019][1696481.991463] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696481.998354] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:42 2019][1696482.004144] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.010630] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.018212] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:42 2019][1696482.023279] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1696482.029556] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:42 2019][1696482.036183] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1696482.042454] mdt01_029 S ffff9159a4eab0c0 0 9013 2 0x00000080 [Wed Feb 6 09:39:42 2019][1696482.049743] Call Trace: [Wed Feb 6 09:39:42 2019][1696482.052380] [] schedule+0x29/0x70 [Wed Feb 6 09:39:42 2019][1696482.057567] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.064450] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:42 2019][1696482.070250] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.076738] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.084316] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:42 2019][1696482.089377] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1696482.095652] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:42 2019][1696482.102274] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1696482.108552] mdt01_030 D ffff9159a4eac100 0 9014 2 0x00000080 [Wed Feb 6 09:39:42 2019][1696482.115828] Call Trace: [Wed Feb 6 09:39:42 2019][1696482.118472] [] schedule+0x29/0x70 [Wed Feb 6 09:39:42 2019][1696482.123624] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 09:39:42 2019][1696482.130248] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 09:39:42 2019][1696482.137131] [] down_write+0x2d/0x3d [Wed Feb 6 09:39:42 2019][1696482.142470] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 09:39:42 2019][1696482.149365] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 09:39:42 2019][1696482.156245] [] ? ___slab_alloc+0x209/0x4f0 [Wed Feb 6 09:39:42 2019][1696482.162200] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:42 2019][1696482.169344] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:42 2019][1696482.175124] [] ? lu_buf_alloc+0x48/0x320 [obdclass] [Wed Feb 6 09:39:42 2019][1696482.181879] [] ? ldlm_cli_enqueue_local+0x27d/0x870 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.189382] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:39:42 2019][1696482.197400] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:39:42 2019][1696482.204812] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:39:42 2019][1696482.212050] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:39:42 2019][1696482.218774] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:42 2019][1696482.226104] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:39:42 2019][1696482.232734] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:39:42 2019][1696482.239403] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.246039] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:42 2019][1696482.252676] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:39:42 2019][1696482.259328] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.266222] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:42 2019][1696482.273541] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:42 2019][1696482.280116] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.287390] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.295104] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.301427] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.308522] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.316281] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:42 2019][1696482.323550] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.331428] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.338405] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:42 2019][1696482.344855] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:42 2019][1696482.350910] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.357397] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.364969] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:42 2019][1696482.370033] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1696482.376308] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:42 2019][1696482.382928] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1696482.389202] mdt01_031 D ffff9159a4ead140 0 9015 2 0x00000080 [Wed Feb 6 09:39:42 2019][1696482.396483] Call Trace: [Wed Feb 6 09:39:42 2019][1696482.399129] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [Wed Feb 6 09:39:42 2019][1696482.406267] [] schedule+0x29/0x70 [Wed Feb 6 09:39:42 2019][1696482.411416] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 09:39:42 2019][1696482.418040] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 09:39:42 2019][1696482.424919] [] down_write+0x2d/0x3d [Wed Feb 6 09:39:42 2019][1696482.430248] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 09:39:42 2019][1696482.437137] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 09:39:42 2019][1696482.444048] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:42 2019][1696482.451186] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:42 2019][1696482.456951] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:39:42 2019][1696482.464968] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:39:42 2019][1696482.472380] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:39:42 2019][1696482.479624] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:39:42 2019][1696482.486347] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:42 2019][1696482.493672] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:39:42 2019][1696482.500302] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:39:42 2019][1696482.506970] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.513606] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:42 2019][1696482.520266] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.527160] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:39:42 2019][1696482.533818] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.540710] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:42 2019][1696482.548042] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:42 2019][1696482.554615] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.561887] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.569599] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.575915] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.583013] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.590771] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:42 2019][1696482.598048] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.605919] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.612886] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:42 2019][1696482.619336] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:42 2019][1696482.625390] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.631879] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.639451] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:42 2019][1696482.644508] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1696482.650785] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:42 2019][1696482.657407] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1696482.663687] mdt01_032 S ffff9159a4eae180 0 9016 2 0x00000080 [Wed Feb 6 09:39:42 2019][1696482.670967] Call Trace: [Wed Feb 6 09:39:42 2019][1696482.673599] [] schedule+0x29/0x70 [Wed Feb 6 09:39:42 2019][1696482.678786] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.685675] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:42 2019][1696482.691471] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.697953] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.705527] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:42 2019][1696482.710592] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1696482.716875] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:42 2019][1696482.723501] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:42 2019][1696482.729774] mdt01_033 D ffff9159a62f0000 0 9017 2 0x00000080 [Wed Feb 6 09:39:42 2019][1696482.737048] Call Trace: [Wed Feb 6 09:39:42 2019][1696482.739679] [] schedule+0x29/0x70 [Wed Feb 6 09:39:42 2019][1696482.744827] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 09:39:42 2019][1696482.751456] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 09:39:42 2019][1696482.758335] [] down_write+0x2d/0x3d [Wed Feb 6 09:39:42 2019][1696482.763669] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 09:39:42 2019][1696482.770562] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 09:39:42 2019][1696482.777471] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:42 2019][1696482.784612] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:42 2019][1696482.790405] [] ? ldlm_cli_enqueue_local+0x27d/0x870 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.797900] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:39:42 2019][1696482.805926] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:39:42 2019][1696482.813336] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:39:42 2019][1696482.820569] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:39:42 2019][1696482.827293] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:42 2019][1696482.834622] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:39:42 2019][1696482.841253] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:39:42 2019][1696482.847924] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.854559] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:42 2019][1696482.861194] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:39:42 2019][1696482.867849] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:42 2019][1696482.874739] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:43 2019][1696482.882055] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:43 2019][1696482.888622] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696482.895890] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696482.903605] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696482.909928] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696482.917024] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696482.924781] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:43 2019][1696482.932053] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696482.939924] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696482.946898] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:43 2019][1696482.953350] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:43 2019][1696482.959402] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696482.965887] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696482.973468] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1696482.978534] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696482.984808] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1696482.991431] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696482.997705] mdt01_034 S ffff9159a62f1040 0 9018 2 0x00000080 [Wed Feb 6 09:39:43 2019][1696483.004985] Call Trace: [Wed Feb 6 09:39:43 2019][1696483.007618] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1696483.012810] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.019696] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1696483.025495] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.031975] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.039552] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1696483.044609] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.050886] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1696483.057503] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.063781] mdt01_035 S ffff9159a62f2080 0 9019 2 0x00000080 [Wed Feb 6 09:39:43 2019][1696483.071053] Call Trace: [Wed Feb 6 09:39:43 2019][1696483.073685] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1696483.078872] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.085752] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1696483.091539] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.098027] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.105608] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1696483.110669] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.116946] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1696483.123563] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.129840] mdt02_018 S ffff9159a62f30c0 0 9020 2 0x00000080 [Wed Feb 6 09:39:43 2019][1696483.137114] Call Trace: [Wed Feb 6 09:39:43 2019][1696483.139755] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:39:43 2019][1696483.146669] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.154420] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1696483.159600] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.166685] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.173768] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:43 2019][1696483.180909] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1696483.186707] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.194031] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:39:43 2019][1696483.201401] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.209424] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:43 2019][1696483.216145] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:39:43 2019][1696483.223119] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:43 2019][1696483.229862] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.237881] [] ? lod_attr_get+0xab/0x130 [lod] [Wed Feb 6 09:39:43 2019][1696483.244167] [] ? mdd_la_get+0x84/0x1a0 [mdd] [Wed Feb 6 09:39:43 2019][1696483.250280] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:43 2019][1696483.257438] [] ? mdt_attr_get_complex+0x4ec/0x850 [mdt] [Wed Feb 6 09:39:43 2019][1696483.264499] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:39:43 2019][1696483.271596] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.278261] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.285348] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.292508] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:39:43 2019][1696483.299232] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:43 2019][1696483.305865] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:39:43 2019][1696483.312706] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.319592] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:43 2019][1696483.326911] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:43 2019][1696483.333484] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.340751] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.348461] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.354781] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.361877] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.369634] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:43 2019][1696483.376901] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.384769] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.391741] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:43 2019][1696483.398194] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:43 2019][1696483.404246] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.410745] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.418328] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1696483.423383] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.429662] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1696483.436280] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.442560] mdt00_020 S ffff9159a62f4100 0 9027 2 0x00000080 [Wed Feb 6 09:39:43 2019][1696483.449849] Call Trace: [Wed Feb 6 09:39:43 2019][1696483.452481] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1696483.457673] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.464556] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1696483.470346] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.476826] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.484407] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1696483.489473] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.495748] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1696483.502381] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.508649] mdt00_021 S ffff9159a62f5140 0 9028 2 0x00000080 [Wed Feb 6 09:39:43 2019][1696483.515924] Call Trace: [Wed Feb 6 09:39:43 2019][1696483.518557] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1696483.523752] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.530634] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1696483.536439] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.542922] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.550497] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1696483.555558] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.561836] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1696483.568464] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.574738] mdt02_019 S ffff9159a62f6180 0 9029 2 0x00000080 [Wed Feb 6 09:39:43 2019][1696483.582019] Call Trace: [Wed Feb 6 09:39:43 2019][1696483.584650] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1696483.589841] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.596759] [] ? keys_fini+0xb1/0x1d0 [obdclass] [Wed Feb 6 09:39:43 2019][1696483.603208] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1696483.609001] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.615491] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.623071] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1696483.628133] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.634411] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1696483.641043] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.647324] mdt02_020 S ffff9159b7550000 0 9030 2 0x00000080 [Wed Feb 6 09:39:43 2019][1696483.654605] Call Trace: [Wed Feb 6 09:39:43 2019][1696483.657235] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1696483.662427] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.669310] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1696483.675104] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.681588] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.689166] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1696483.694229] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.700504] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1696483.707125] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.713400] mdt_io02_018 S ffff9159b7551040 0 9031 2 0x00000080 [Wed Feb 6 09:39:43 2019][1696483.720681] Call Trace: [Wed Feb 6 09:39:43 2019][1696483.723312] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1696483.728511] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.735406] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1696483.741203] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.747692] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.755273] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1696483.760340] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.766625] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1696483.773243] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.779516] ldlm_cn02_007 S ffff9159b7552080 0 9032 2 0x00000080 [Wed Feb 6 09:39:43 2019][1696483.786799] Call Trace: [Wed Feb 6 09:39:43 2019][1696483.789432] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1696483.794625] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.801508] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1696483.807296] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.813783] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.821367] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:43 2019][1696483.826433] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.832709] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:43 2019][1696483.839335] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:43 2019][1696483.845608] mdt00_022 S ffff9159b75530c0 0 9033 2 0x00000080 [Wed Feb 6 09:39:43 2019][1696483.852886] Call Trace: [Wed Feb 6 09:39:43 2019][1696483.855516] [] schedule+0x29/0x70 [Wed Feb 6 09:39:43 2019][1696483.860703] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.867587] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:43 2019][1696483.873396] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:43 2019][1696483.879888] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696483.887469] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1696483.892545] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696483.898818] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1696483.905436] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696483.911711] mdt_io02_019 S ffff9159b7554100 0 9036 2 0x00000080 [Wed Feb 6 09:39:44 2019][1696483.918986] Call Trace: [Wed Feb 6 09:39:44 2019][1696483.921620] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1696483.926809] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696483.933698] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1696483.939498] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696483.945990] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696483.953573] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1696483.958638] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696483.964911] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1696483.971531] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696483.977812] mdt_io01_022 S ffff9159b7555140 0 9037 2 0x00000080 [Wed Feb 6 09:39:44 2019][1696483.985101] Call Trace: [Wed Feb 6 09:39:44 2019][1696483.987749] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1696483.992941] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696483.999823] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1696484.005616] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.012109] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.019683] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1696484.024751] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.031028] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1696484.037652] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.043928] ldlm_cb02_003 S ffff9159b7556180 0 9046 2 0x00000080 [Wed Feb 6 09:39:44 2019][1696484.051211] Call Trace: [Wed Feb 6 09:39:44 2019][1696484.053843] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1696484.059034] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.065923] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1696484.071716] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.078200] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.085792] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1696484.090846] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.097129] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1696484.103756] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.110030] mdt_out01_005 S ffff91599d6a8000 0 9053 2 0x00000080 [Wed Feb 6 09:39:44 2019][1696484.117309] Call Trace: [Wed Feb 6 09:39:44 2019][1696484.119946] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1696484.125138] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.132023] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1696484.137819] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.144308] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.151881] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1696484.156940] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.163219] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1696484.169837] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.176112] mdt_io02_020 S ffff91599d6a9040 0 9062 2 0x00000080 [Wed Feb 6 09:39:44 2019][1696484.183401] Call Trace: [Wed Feb 6 09:39:44 2019][1696484.186041] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1696484.191238] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.198123] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1696484.203917] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.210399] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.217977] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1696484.223034] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.229311] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1696484.235935] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.242209] ldlm_cn01_008 S ffff91599d6ae180 0 9129 2 0x00000080 [Wed Feb 6 09:39:44 2019][1696484.249490] Call Trace: [Wed Feb 6 09:39:44 2019][1696484.252123] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1696484.257308] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.264197] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:39:44 2019][1696484.271167] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1696484.276966] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.283451] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.291033] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1696484.296098] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.302384] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1696484.309006] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.315289] ldlm_cn00_008 S ffff91599d6ac100 0 9139 2 0x00000080 [Wed Feb 6 09:39:44 2019][1696484.322580] Call Trace: [Wed Feb 6 09:39:44 2019][1696484.325212] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1696484.330402] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.337291] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1696484.343089] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.349579] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.357158] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1696484.362215] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.368494] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1696484.375114] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.381391] mdt_rdpg01_006 S ffff91599d6ad140 0 9146 2 0x00000080 [Wed Feb 6 09:39:44 2019][1696484.388675] Call Trace: [Wed Feb 6 09:39:44 2019][1696484.391318] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1696484.396516] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.403396] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1696484.409187] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.415666] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.423242] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1696484.428307] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.434580] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1696484.441204] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.447487] mdt_rdpg02_004 S ffff91599d6ab0c0 0 9147 2 0x00000080 [Wed Feb 6 09:39:44 2019][1696484.454779] Call Trace: [Wed Feb 6 09:39:44 2019][1696484.457419] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1696484.462620] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.469503] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1696484.475302] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.481792] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.489375] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1696484.494436] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.500709] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1696484.507327] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.513605] mdt_rdpg02_005 S ffff9159aa7b0000 0 9148 2 0x00000080 [Wed Feb 6 09:39:44 2019][1696484.520883] Call Trace: [Wed Feb 6 09:39:44 2019][1696484.523524] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1696484.528721] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.535608] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1696484.541416] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.547901] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.555489] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1696484.560554] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.566826] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1696484.573448] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.579719] mdt_rdpg01_007 S ffff9159aa7b1040 0 9149 2 0x00000080 [Wed Feb 6 09:39:44 2019][1696484.587007] Call Trace: [Wed Feb 6 09:39:44 2019][1696484.589638] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1696484.594827] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.601716] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1696484.607510] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.613991] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.621572] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1696484.626647] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.632924] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1696484.639542] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.645818] mdt_rdpg01_008 S ffff9159aa7b2080 0 9150 2 0x00000080 [Wed Feb 6 09:39:44 2019][1696484.653099] Call Trace: [Wed Feb 6 09:39:44 2019][1696484.655736] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1696484.660933] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.667815] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1696484.673609] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.680095] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.687666] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1696484.692735] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.699017] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1696484.705636] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.711908] mdt_rdpg01_009 S ffff9159aa7b30c0 0 9151 2 0x00000080 [Wed Feb 6 09:39:44 2019][1696484.719196] Call Trace: [Wed Feb 6 09:39:44 2019][1696484.721832] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1696484.727020] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.733898] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1696484.739693] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.746184] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.753767] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1696484.758835] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.765108] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1696484.771726] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.778001] ldlm_cn01_009 S ffff9159aa7b4100 0 9152 2 0x00000080 [Wed Feb 6 09:39:44 2019][1696484.785283] Call Trace: [Wed Feb 6 09:39:44 2019][1696484.787917] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1696484.793116] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.800004] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:44 2019][1696484.805795] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.812283] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:44 2019][1696484.819856] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:44 2019][1696484.824921] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.831194] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:44 2019][1696484.837814] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:44 2019][1696484.844093] ldlm_cn01_010 S ffff9159aa7b5140 0 9155 2 0x00000080 [Wed Feb 6 09:39:44 2019][1696484.851381] Call Trace: [Wed Feb 6 09:39:44 2019][1696484.854015] [] schedule+0x29/0x70 [Wed Feb 6 09:39:44 2019][1696484.859203] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696484.866090] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1696484.871884] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696484.878368] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696484.885948] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1696484.891012] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696484.897287] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1696484.903904] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696484.910180] ldlm_cb00_003 S ffff9159aa7b6180 0 9162 2 0x00000080 [Wed Feb 6 09:39:45 2019][1696484.917465] Call Trace: [Wed Feb 6 09:39:45 2019][1696484.920102] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1696484.925298] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696484.932187] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1696484.937988] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696484.944489] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696484.952071] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1696484.957129] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696484.963409] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1696484.970033] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696484.976315] mdt02_021 S ffff9159f0a18000 0 9171 2 0x00000080 [Wed Feb 6 09:39:45 2019][1696484.983601] Call Trace: [Wed Feb 6 09:39:45 2019][1696484.986242] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:39:45 2019][1696484.993162] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.000914] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1696485.006104] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.013194] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.020285] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:45 2019][1696485.027435] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1696485.033232] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.040571] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:39:45 2019][1696485.047919] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.055940] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:45 2019][1696485.062666] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:39:45 2019][1696485.069650] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:45 2019][1696485.076403] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.084426] [] ? lod_attr_get+0xab/0x130 [lod] [Wed Feb 6 09:39:45 2019][1696485.090717] [] ? mdd_la_get+0x84/0x1a0 [mdd] [Wed Feb 6 09:39:45 2019][1696485.096834] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:45 2019][1696485.103985] [] ? mdt_attr_get_complex+0x4ec/0x850 [mdt] [Wed Feb 6 09:39:45 2019][1696485.111060] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:39:45 2019][1696485.118169] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.124840] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.131954] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.139112] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:39:45 2019][1696485.145836] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:45 2019][1696485.152469] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:39:45 2019][1696485.159297] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.166189] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:45 2019][1696485.173508] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:45 2019][1696485.180083] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.187355] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.195066] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.201379] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.208476] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.216239] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:45 2019][1696485.223509] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.231398] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.238375] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:45 2019][1696485.244827] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:45 2019][1696485.250884] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.257368] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.264950] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1696485.270015] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.276291] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1696485.282907] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.289188] mdt00_023 S ffff9159f0a19040 0 9174 2 0x00000080 [Wed Feb 6 09:39:45 2019][1696485.296469] Call Trace: [Wed Feb 6 09:39:45 2019][1696485.299108] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1696485.304302] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.311189] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1696485.316995] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.323490] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.331075] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1696485.336131] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.342408] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1696485.349032] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.355303] mdt00_024 S ffff9159f0a1a080 0 9181 2 0x00000080 [Wed Feb 6 09:39:45 2019][1696485.362591] Call Trace: [Wed Feb 6 09:39:45 2019][1696485.365229] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1696485.370419] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.377303] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1696485.383092] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.389593] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.397172] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1696485.402239] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.408528] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1696485.415160] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.421431] mdt00_025 S ffff9159f0a1b0c0 0 9182 2 0x00000080 [Wed Feb 6 09:39:45 2019][1696485.428716] Call Trace: [Wed Feb 6 09:39:45 2019][1696485.431348] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1696485.436539] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.443427] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1696485.449227] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.455719] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.463300] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1696485.468362] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.474638] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1696485.481258] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.487539] mdt02_022 S ffff9159f0a1d140 0 9184 2 0x00000080 [Wed Feb 6 09:39:45 2019][1696485.494834] Call Trace: [Wed Feb 6 09:39:45 2019][1696485.497474] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1696485.502671] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.509550] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1696485.515342] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.521823] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.529398] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1696485.534456] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.540734] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1696485.547356] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.553632] mdt01_036 S ffff9159f0a1e180 0 9185 2 0x00000080 [Wed Feb 6 09:39:45 2019][1696485.560918] Call Trace: [Wed Feb 6 09:39:45 2019][1696485.563554] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1696485.568744] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.575637] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1696485.581425] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.587909] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.595483] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1696485.600545] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.606822] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1696485.613450] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.619726] mdt_io00_021 S ffff9159ad548000 0 9188 2 0x00000080 [Wed Feb 6 09:39:45 2019][1696485.627004] Call Trace: [Wed Feb 6 09:39:45 2019][1696485.629648] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1696485.634840] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.641727] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1696485.647520] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.654001] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.661581] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1696485.666648] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.672924] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1696485.679542] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.685821] mdt_io01_023 S ffff9159ad549040 0 9189 2 0x00000080 [Wed Feb 6 09:39:45 2019][1696485.693092] Call Trace: [Wed Feb 6 09:39:45 2019][1696485.695727] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1696485.700925] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.707812] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1696485.713609] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.720119] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.727698] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1696485.732759] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.739033] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1696485.745663] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.751937] mdt_io01_024 S ffff9159ad54a080 0 9190 2 0x00000080 [Wed Feb 6 09:39:45 2019][1696485.759230] Call Trace: [Wed Feb 6 09:39:45 2019][1696485.761867] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1696485.767059] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.773947] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1696485.779747] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.786251] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.793842] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:45 2019][1696485.798909] [] ? finish_task_switch+0x54/0x1c0 [Wed Feb 6 09:39:45 2019][1696485.805189] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.811481] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:45 2019][1696485.818097] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:45 2019][1696485.824371] ldlm_cn02_008 S ffff9159ad54b0c0 0 9191 2 0x00000080 [Wed Feb 6 09:39:45 2019][1696485.831648] Call Trace: [Wed Feb 6 09:39:45 2019][1696485.834279] [] schedule+0x29/0x70 [Wed Feb 6 09:39:45 2019][1696485.839473] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.846361] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:45 2019][1696485.852164] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.858647] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:45 2019][1696485.866219] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696485.871278] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696485.877554] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1696485.884173] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696485.890452] mdt02_023 S ffff9139ee33a080 0 9199 2 0x00000080 [Wed Feb 6 09:39:46 2019][1696485.897737] Call Trace: [Wed Feb 6 09:39:46 2019][1696485.900381] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1696485.905573] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696485.912461] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1696485.918262] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696485.924748] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696485.932335] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696485.937399] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696485.943676] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1696485.950293] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696485.956574] mdt_io02_021 S ffff9139ee33d140 0 9200 2 0x00000080 [Wed Feb 6 09:39:46 2019][1696485.963855] Call Trace: [Wed Feb 6 09:39:46 2019][1696485.966490] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1696485.971688] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696485.978579] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1696485.984378] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696485.990884] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696485.998469] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696486.003525] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.009801] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1696486.016422] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.022699] mdt_io02_022 S ffff9139ee33c100 0 9201 2 0x00000080 [Wed Feb 6 09:39:46 2019][1696486.029981] Call Trace: [Wed Feb 6 09:39:46 2019][1696486.032614] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1696486.037798] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.044679] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1696486.050478] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.056958] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.064540] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696486.069604] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.075880] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1696486.082520] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.088791] ldlm_cn00_009 S ffff9139ee338000 0 9226 2 0x00000080 [Wed Feb 6 09:39:46 2019][1696486.096066] Call Trace: [Wed Feb 6 09:39:46 2019][1696486.098700] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1696486.103892] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.110772] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1696486.116573] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.123061] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.130645] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696486.135705] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.141986] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1696486.148613] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.154887] ldlm_cn02_009 S ffff9159ad54c100 0 9373 2 0x00000080 [Wed Feb 6 09:39:46 2019][1696486.162172] Call Trace: [Wed Feb 6 09:39:46 2019][1696486.164816] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1696486.170011] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.176893] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1696486.182682] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.189166] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.196745] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696486.201808] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.208086] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1696486.214701] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.220979] mdt_io01_025 S ffff9159a4bee180 0 9649 2 0x00000080 [Wed Feb 6 09:39:46 2019][1696486.228252] Call Trace: [Wed Feb 6 09:39:46 2019][1696486.230883] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1696486.236076] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.242967] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1696486.248764] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.255268] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.262842] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696486.267900] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.274177] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1696486.280801] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.287076] mdt_io01_026 S ffff9149d62d4100 0 9759 2 0x00000080 [Wed Feb 6 09:39:46 2019][1696486.294365] Call Trace: [Wed Feb 6 09:39:46 2019][1696486.296994] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1696486.302186] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.309080] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1696486.314880] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.321373] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.328956] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696486.334024] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.340298] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1696486.346929] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.353198] mdt_io01_027 S ffff9149d62d5140 0 9838 2 0x00000080 [Wed Feb 6 09:39:46 2019][1696486.360466] Call Trace: [Wed Feb 6 09:39:46 2019][1696486.363099] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1696486.368292] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.375179] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1696486.380976] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.387467] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.395055] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696486.400117] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.406390] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1696486.413009] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.419285] mdt_io01_028 S ffff9149d62d0000 0 9839 2 0x00000080 [Wed Feb 6 09:39:46 2019][1696486.426563] Call Trace: [Wed Feb 6 09:39:46 2019][1696486.429201] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1696486.434394] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.441277] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1696486.447065] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.453550] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.461127] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696486.466192] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.472473] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1696486.479099] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.485379] mdt_io01_029 S ffff914aa9478000 0 9840 2 0x00000080 [Wed Feb 6 09:39:46 2019][1696486.492658] Call Trace: [Wed Feb 6 09:39:46 2019][1696486.495295] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1696486.500489] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.507376] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1696486.513175] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.519663] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.527247] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696486.532312] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.538590] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1696486.545222] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.551502] mdt_seqm_0002 S ffff9159f6d18000 0 9874 2 0x00000080 [Wed Feb 6 09:39:46 2019][1696486.558795] Call Trace: [Wed Feb 6 09:39:46 2019][1696486.561432] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1696486.566628] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.573521] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1696486.579325] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.585810] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.593390] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696486.598451] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.604726] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1696486.611357] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.617630] ldlm_cn02_010 S ffff9149d7abd140 0 9990 2 0x00000080 [Wed Feb 6 09:39:46 2019][1696486.624913] Call Trace: [Wed Feb 6 09:39:46 2019][1696486.627553] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1696486.632745] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.639628] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1696486.645424] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.651905] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.659482] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696486.664545] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.670819] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1696486.677440] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.683718] mdt_out03_002 S ffff9135b0d74100 0 10218 2 0x00000080 [Wed Feb 6 09:39:46 2019][1696486.691004] Call Trace: [Wed Feb 6 09:39:46 2019][1696486.693636] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1696486.698828] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.705724] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1696486.711525] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.718009] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.725589] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696486.730654] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.736932] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1696486.743564] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.749839] mdt_io00_022 S ffff914997efe180 0 10285 2 0x00000080 [Wed Feb 6 09:39:46 2019][1696486.757114] Call Trace: [Wed Feb 6 09:39:46 2019][1696486.759749] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1696486.764940] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.771822] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1696486.777616] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.784105] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.791682] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696486.796745] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.803021] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:46 2019][1696486.809643] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.815918] mdt03_012 S ffff9149f3d930c0 0 10287 2 0x00000080 [Wed Feb 6 09:39:46 2019][1696486.823192] Call Trace: [Wed Feb 6 09:39:46 2019][1696486.825822] [] schedule+0x29/0x70 [Wed Feb 6 09:39:46 2019][1696486.831014] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.837903] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:46 2019][1696486.843701] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.850205] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:46 2019][1696486.857783] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:46 2019][1696486.862847] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:46 2019][1696486.869125] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696486.875747] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696486.882020] mdt_io00_023 S ffff9149ca784100 0 10518 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696486.889318] Call Trace: [Wed Feb 6 09:39:47 2019][1696486.891959] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696486.897154] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696486.904047] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1696486.909839] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696486.916325] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696486.923904] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1696486.928972] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696486.935256] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696486.941888] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696486.948163] mdt_out00_004 S ffff9149ca7830c0 0 10654 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696486.955448] Call Trace: [Wed Feb 6 09:39:47 2019][1696486.958082] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696486.963272] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696486.970160] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1696486.975961] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696486.982446] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696486.990028] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1696486.995091] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.001364] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696487.007984] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.014256] kworker/13:2 S ffff9135b0d71040 0 10873 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696487.021543] Call Trace: [Wed Feb 6 09:39:47 2019][1696487.024174] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:47 2019][1696487.030367] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696487.035512] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:47 2019][1696487.041277] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:47 2019][1696487.047989] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1696487.053060] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.059337] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696487.065959] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.072234] mdt_io01_030 S ffff9129ee3ce180 0 11093 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696487.079516] Call Trace: [Wed Feb 6 09:39:47 2019][1696487.082152] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696487.087355] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.094245] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1696487.100042] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.106534] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.114115] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1696487.119174] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.125452] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696487.132066] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.138345] ldlm_cn00_010 S ffff9149d7ab9040 0 11333 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696487.145618] Call Trace: [Wed Feb 6 09:39:47 2019][1696487.148248] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696487.153439] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.160328] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1696487.166127] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.172617] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.180192] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1696487.185270] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.191549] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696487.198173] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.204443] mdt_io02_023 S ffff9149ca780000 0 11807 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696487.211728] Call Trace: [Wed Feb 6 09:39:47 2019][1696487.214361] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696487.219553] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.226440] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1696487.232238] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.238729] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.246312] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1696487.251385] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.257661] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696487.264285] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.270557] ldlm_cn00_011 S ffff9135b0d70000 0 12237 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696487.277831] Call Trace: [Wed Feb 6 09:39:47 2019][1696487.280466] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696487.285664] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.292550] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1696487.298360] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.304850] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.312431] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1696487.317499] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.323775] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696487.330395] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.336670] mdt_rdpg00_005 S ffff9129fb85c100 0 12704 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696487.343963] Call Trace: [Wed Feb 6 09:39:47 2019][1696487.346598] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696487.351790] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.358673] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1696487.364465] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.370943] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.378524] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1696487.383582] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.389860] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696487.396479] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.402751] mdt_io03_008 S ffff9139fbad0000 0 12779 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696487.410027] Call Trace: [Wed Feb 6 09:39:47 2019][1696487.412658] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696487.417844] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.424728] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1696487.430519] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.437008] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.444579] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1696487.449637] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.455913] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696487.462539] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.468820] kworker/42:2 S ffff913aa9802080 0 12966 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696487.476123] Call Trace: [Wed Feb 6 09:39:47 2019][1696487.478768] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:47 2019][1696487.484962] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696487.490107] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:47 2019][1696487.495858] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:47 2019][1696487.502571] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1696487.507635] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.513919] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696487.520537] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.526821] mdt_out01_006 S ffff9149ca742080 0 13924 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696487.534104] Call Trace: [Wed Feb 6 09:39:47 2019][1696487.536732] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696487.541931] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.548822] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1696487.554619] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.561114] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.568697] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1696487.573760] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.580036] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696487.586657] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.592936] ldlm_cn00_012 S ffff9130aa7830c0 0 14696 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696487.600230] Call Trace: [Wed Feb 6 09:39:47 2019][1696487.602868] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696487.608067] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.614966] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1696487.620773] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.627254] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.634833] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1696487.639898] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.646174] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696487.652795] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.659077] ldlm_cn00_013 S ffff912fe0b66180 0 14884 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696487.666362] Call Trace: [Wed Feb 6 09:39:47 2019][1696487.669002] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696487.674201] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.681094] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1696487.686886] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.693377] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.700959] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1696487.706026] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.712300] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696487.718927] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.725210] ldlm_cn00_014 S ffff912fe0b61040 0 14885 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696487.732496] Call Trace: [Wed Feb 6 09:39:47 2019][1696487.735131] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696487.740333] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.747223] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1696487.753020] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.759502] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.767075] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1696487.772137] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.778410] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696487.785028] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.791310] ldlm_cn00_015 S ffff912fe0b65140 0 14886 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696487.798585] Call Trace: [Wed Feb 6 09:39:47 2019][1696487.801224] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696487.806420] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.813311] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:47 2019][1696487.819101] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.825583] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:47 2019][1696487.833162] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:47 2019][1696487.838229] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.844506] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:47 2019][1696487.851126] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:47 2019][1696487.857402] ldlm_cn00_016 S ffff912fe0b64100 0 14889 2 0x00000080 [Wed Feb 6 09:39:47 2019][1696487.864675] Call Trace: [Wed Feb 6 09:39:47 2019][1696487.867306] [] schedule+0x29/0x70 [Wed Feb 6 09:39:47 2019][1696487.872492] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696487.879371] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1696487.885170] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696487.891664] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696487.899242] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1696487.904306] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696487.910584] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1696487.917211] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696487.923483] ldlm_cn00_017 S ffff9135b0d72080 0 14897 2 0x00000080 [Wed Feb 6 09:39:48 2019][1696487.930762] Call Trace: [Wed Feb 6 09:39:48 2019][1696487.933399] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1696487.938592] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696487.945477] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1696487.951274] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696487.957777] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696487.965366] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1696487.970423] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696487.976707] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1696487.983326] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696487.989602] ldlm_cn00_018 S ffff9159f6d1a080 0 14898 2 0x00000080 [Wed Feb 6 09:39:48 2019][1696487.996889] Call Trace: [Wed Feb 6 09:39:48 2019][1696487.999521] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1696488.004707] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.011593] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1696488.017388] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.023877] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.031455] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1696488.036520] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.042796] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1696488.049417] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.055687] mdt_io00_024 S ffff9159f6d1e180 0 14901 2 0x00000080 [Wed Feb 6 09:39:48 2019][1696488.062973] Call Trace: [Wed Feb 6 09:39:48 2019][1696488.065603] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1696488.070794] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.077691] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1696488.083497] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.089984] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.097565] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1696488.102627] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.108906] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1696488.115530] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.121804] mdt_io01_031 S ffff9159fc530000 0 14973 2 0x00000080 [Wed Feb 6 09:39:48 2019][1696488.129088] Call Trace: [Wed Feb 6 09:39:48 2019][1696488.131723] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1696488.136914] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.143803] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1696488.149612] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.156100] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.163687] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1696488.168750] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.175028] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1696488.181647] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.187918] mdt_io01_032 S ffff9129fb850000 0 15082 2 0x00000080 [Wed Feb 6 09:39:48 2019][1696488.195203] Call Trace: [Wed Feb 6 09:39:48 2019][1696488.197833] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1696488.203027] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.209917] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1696488.215723] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.222217] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.229800] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1696488.234862] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.241138] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1696488.247757] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.254034] mdt_io01_033 S ffff9129fb851040 0 15083 2 0x00000080 [Wed Feb 6 09:39:48 2019][1696488.261341] Call Trace: [Wed Feb 6 09:39:48 2019][1696488.263972] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1696488.269167] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.276052] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1696488.281846] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.288326] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.295911] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1696488.300975] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.307251] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1696488.313880] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.320161] mdt_io01_034 S ffff9129fb8530c0 0 15084 2 0x00000080 [Wed Feb 6 09:39:48 2019][1696488.327446] Call Trace: [Wed Feb 6 09:39:48 2019][1696488.330087] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1696488.335275] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.342162] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1696488.347963] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.354451] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.362034] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1696488.367091] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.373369] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1696488.379989] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.386266] mdt_io01_035 S ffff9143cc481040 0 15117 2 0x00000080 [Wed Feb 6 09:39:48 2019][1696488.393559] Call Trace: [Wed Feb 6 09:39:48 2019][1696488.396200] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1696488.401399] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.408282] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1696488.414084] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.420573] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.428157] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1696488.433221] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.439498] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1696488.446113] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.452392] mdt_io01_036 S ffff9143cc4830c0 0 15118 2 0x00000080 [Wed Feb 6 09:39:48 2019][1696488.459666] Call Trace: [Wed Feb 6 09:39:48 2019][1696488.462293] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1696488.467487] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.474374] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1696488.480173] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.486658] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.494235] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1696488.499293] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.505570] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1696488.512190] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.518471] mdt_io01_037 S ffff9143cc485140 0 15179 2 0x00000080 [Wed Feb 6 09:39:48 2019][1696488.525753] Call Trace: [Wed Feb 6 09:39:48 2019][1696488.528393] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1696488.533575] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.540455] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1696488.546244] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.552739] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.560320] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1696488.565383] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.571658] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1696488.578274] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.584549] mdt_io01_038 S ffff9143cc482080 0 15180 2 0x00000080 [Wed Feb 6 09:39:48 2019][1696488.591828] Call Trace: [Wed Feb 6 09:39:48 2019][1696488.594466] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1696488.599658] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.606541] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1696488.612336] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.618827] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.626411] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1696488.631474] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.637752] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1696488.644368] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.650648] mdt_out02_004 S ffff91432121d140 0 15751 2 0x00000080 [Wed Feb 6 09:39:48 2019][1696488.657930] Call Trace: [Wed Feb 6 09:39:48 2019][1696488.660562] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1696488.665747] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.672634] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1696488.678427] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.684910] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.692484] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1696488.697544] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.703820] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1696488.710441] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.716711] mdt03_013 S ffff91432121c100 0 16071 2 0x00000080 [Wed Feb 6 09:39:48 2019][1696488.723986] Call Trace: [Wed Feb 6 09:39:48 2019][1696488.726619] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1696488.731811] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.738700] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1696488.744496] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.750989] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.758578] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1696488.763637] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.769911] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1696488.776535] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.782811] kworker/29:0 S ffff9130e37ed140 0 17671 2 0x00000080 [Wed Feb 6 09:39:48 2019][1696488.790098] Call Trace: [Wed Feb 6 09:39:48 2019][1696488.792730] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:48 2019][1696488.798921] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1696488.804064] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:48 2019][1696488.809823] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:48 2019][1696488.816533] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:48 2019][1696488.821600] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.827883] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:48 2019][1696488.834508] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:48 2019][1696488.840781] ldlm_cn02_011 S ffff9130e37ea080 0 17691 2 0x00000080 [Wed Feb 6 09:39:48 2019][1696488.848054] Call Trace: [Wed Feb 6 09:39:48 2019][1696488.850688] [] schedule+0x29/0x70 [Wed Feb 6 09:39:48 2019][1696488.855882] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.862766] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:48 2019][1696488.868571] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:48 2019][1696488.875066] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696488.882647] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696488.887714] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696488.893995] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696488.900624] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696488.906901] kworker/21:0 S ffff9130aa780000 0 18390 2 0x00000080 [Wed Feb 6 09:39:49 2019][1696488.914206] Call Trace: [Wed Feb 6 09:39:49 2019][1696488.916845] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:49 2019][1696488.923041] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1696488.928200] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:49 2019][1696488.933958] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:49 2019][1696488.940668] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696488.945725] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696488.952003] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696488.958625] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696488.964900] kworker/45:2 S ffff9130aa785140 0 18925 2 0x00000080 [Wed Feb 6 09:39:49 2019][1696488.972185] Call Trace: [Wed Feb 6 09:39:49 2019][1696488.974820] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:49 2019][1696488.981009] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1696488.986156] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:49 2019][1696488.991910] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:49 2019][1696488.998619] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696489.003684] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.009961] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696489.016588] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.022860] ldlm_cn00_019 S ffff91432121b0c0 0 19851 2 0x00000080 [Wed Feb 6 09:39:49 2019][1696489.030144] Call Trace: [Wed Feb 6 09:39:49 2019][1696489.032775] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1696489.037974] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.044861] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1696489.050652] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.057146] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.064727] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696489.069795] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.076070] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696489.082687] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.088963] ldlm_cn00_020 S ffff914321218000 0 19871 2 0x00000080 [Wed Feb 6 09:39:49 2019][1696489.096239] Call Trace: [Wed Feb 6 09:39:49 2019][1696489.098870] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1696489.104059] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.110951] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1696489.116747] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.123245] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.130818] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696489.135876] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.142154] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696489.148773] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.155052] mdt_io03_009 S ffff9159fbfa1040 0 19893 2 0x00000080 [Wed Feb 6 09:39:49 2019][1696489.162339] Call Trace: [Wed Feb 6 09:39:49 2019][1696489.164976] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1696489.170185] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.177071] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1696489.182868] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.189358] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.196934] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696489.201999] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.208280] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696489.214905] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.221179] mdt_rdpg03_003 S ffff9159fbfa4100 0 19894 2 0x00000080 [Wed Feb 6 09:39:49 2019][1696489.228458] Call Trace: [Wed Feb 6 09:39:49 2019][1696489.231098] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1696489.236298] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.243189] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1696489.248990] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.255480] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.263060] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696489.268115] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.274397] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696489.281015] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.287291] mdt_io03_010 S ffff9159fbfa30c0 0 19906 2 0x00000080 [Wed Feb 6 09:39:49 2019][1696489.294576] Call Trace: [Wed Feb 6 09:39:49 2019][1696489.297210] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1696489.302406] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.309292] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1696489.315086] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.321566] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.329148] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696489.334216] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.340488] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696489.347107] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.353383] mdt_rdpg00_006 S ffff9159fbfa6180 0 19913 2 0x00000080 [Wed Feb 6 09:39:49 2019][1696489.360673] Call Trace: [Wed Feb 6 09:39:49 2019][1696489.363303] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1696489.368504] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.375398] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1696489.381193] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.387682] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.395266] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696489.400323] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.406601] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696489.413217] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.419491] ldlm_cb03_003 S ffff915927e330c0 0 19917 2 0x00000080 [Wed Feb 6 09:39:49 2019][1696489.426790] Call Trace: [Wed Feb 6 09:39:49 2019][1696489.429433] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1696489.434628] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.441515] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1696489.447318] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.453814] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.461396] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696489.466462] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.472737] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696489.479365] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.485641] mdt_rdpg03_004 S ffff915927e34100 0 19922 2 0x00000080 [Wed Feb 6 09:39:49 2019][1696489.492920] Call Trace: [Wed Feb 6 09:39:49 2019][1696489.495558] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1696489.500757] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.507641] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1696489.513439] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.519930] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.527503] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696489.532563] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.538848] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696489.545469] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.551744] mdt_io03_011 S ffff915927e35140 0 19923 2 0x00000080 [Wed Feb 6 09:39:49 2019][1696489.559036] Call Trace: [Wed Feb 6 09:39:49 2019][1696489.561673] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1696489.566866] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.573751] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1696489.579553] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.586041] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.593625] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696489.598685] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.604971] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696489.611592] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.617868] mdt_io03_012 S ffff915927e36180 0 19930 2 0x00000080 [Wed Feb 6 09:39:49 2019][1696489.625142] Call Trace: [Wed Feb 6 09:39:49 2019][1696489.627774] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1696489.632967] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.639856] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1696489.645652] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.652144] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.659729] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696489.664796] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.671076] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696489.677700] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.683983] ldlm_cn01_011 S ffff9129ee3cc100 0 19972 2 0x00000080 [Wed Feb 6 09:39:49 2019][1696489.691277] Call Trace: [Wed Feb 6 09:39:49 2019][1696489.693917] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1696489.699105] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.705991] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1696489.711777] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.718264] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.725846] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696489.730915] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.737191] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696489.743819] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.750095] ldlm_cn01_012 S ffff9129ee3cd140 0 19973 2 0x00000080 [Wed Feb 6 09:39:49 2019][1696489.757367] Call Trace: [Wed Feb 6 09:39:49 2019][1696489.759997] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1696489.765184] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.772062] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1696489.777854] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.784339] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.791915] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696489.796983] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.803253] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696489.809875] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.816151] ldlm_cn01_013 S ffff9129ee3ca080 0 19974 2 0x00000080 [Wed Feb 6 09:39:49 2019][1696489.823445] Call Trace: [Wed Feb 6 09:39:49 2019][1696489.826081] [] schedule+0x29/0x70 [Wed Feb 6 09:39:49 2019][1696489.831273] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.838167] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:49 2019][1696489.843965] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.850453] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:49 2019][1696489.858035] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:49 2019][1696489.863098] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:49 2019][1696489.869379] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:49 2019][1696489.875993] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696489.882274] ldlm_cn00_021 S ffff9129ee3c9040 0 19975 2 0x00000080 [Wed Feb 6 09:39:50 2019][1696489.889566] Call Trace: [Wed Feb 6 09:39:50 2019][1696489.892200] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1696489.897400] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696489.904288] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1696489.910075] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696489.916556] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696489.924138] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1696489.929204] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696489.935481] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1696489.942100] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696489.948375] ldlm_cn00_022 S ffff9149ca786180 0 20039 2 0x00000080 [Wed Feb 6 09:39:50 2019][1696489.955661] Call Trace: [Wed Feb 6 09:39:50 2019][1696489.958295] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1696489.963484] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696489.970378] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1696489.976174] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696489.982664] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696489.990247] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1696489.995309] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.001579] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1696490.008205] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.014475] ldlm_cn00_023 S ffff9149ca785140 0 20040 2 0x00000080 [Wed Feb 6 09:39:50 2019][1696490.021769] Call Trace: [Wed Feb 6 09:39:50 2019][1696490.024402] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1696490.029588] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.036474] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1696490.042267] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.048752] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.056334] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1696490.061400] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.067677] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1696490.074296] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.080567] ldlm_cn00_024 S ffff9139f700a080 0 20043 2 0x00000080 [Wed Feb 6 09:39:50 2019][1696490.087845] Call Trace: [Wed Feb 6 09:39:50 2019][1696490.090476] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1696490.095673] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.102567] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1696490.108363] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.114850] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.122429] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1696490.127491] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.133767] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1696490.140388] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.146662] mdt_io02_024 S ffff9143cc486180 0 20150 2 0x00000080 [Wed Feb 6 09:39:50 2019][1696490.153938] Call Trace: [Wed Feb 6 09:39:50 2019][1696490.156571] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1696490.161766] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.168658] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1696490.174453] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.180944] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.188523] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1696490.193585] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.199862] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1696490.206483] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.212759] ldlm_cn00_025 S ffff9159f1f62080 0 20158 2 0x00000080 [Wed Feb 6 09:39:50 2019][1696490.220047] Call Trace: [Wed Feb 6 09:39:50 2019][1696490.222680] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1696490.227879] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.234771] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1696490.240562] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.247041] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.254624] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1696490.259688] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.265967] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1696490.272585] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.278861] mdt_io02_025 S ffff9159ff284100 0 20190 2 0x00000080 [Wed Feb 6 09:39:50 2019][1696490.286133] Call Trace: [Wed Feb 6 09:39:50 2019][1696490.288764] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1696490.293950] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.300835] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1696490.306633] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.313123] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.320700] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1696490.325766] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.332036] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1696490.338659] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.344933] mdt_io00_025 S ffff9159f6649040 0 20197 2 0x00000080 [Wed Feb 6 09:39:50 2019][1696490.352210] Call Trace: [Wed Feb 6 09:39:50 2019][1696490.354843] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1696490.360031] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.366920] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1696490.372716] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.379198] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.386777] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1696490.391840] [] ? finish_task_switch+0x54/0x1c0 [Wed Feb 6 09:39:50 2019][1696490.398117] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.404394] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1696490.411020] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.417298] mdt_io00_026 S ffff9159f664e180 0 20198 2 0x00000080 [Wed Feb 6 09:39:50 2019][1696490.424578] Call Trace: [Wed Feb 6 09:39:50 2019][1696490.427211] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1696490.432401] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.439284] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1696490.445096] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.451601] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.459182] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1696490.464246] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.470521] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1696490.477140] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.483418] mdt_io00_027 S ffff9159a0ead140 0 20199 2 0x00000080 [Wed Feb 6 09:39:50 2019][1696490.490696] Call Trace: [Wed Feb 6 09:39:50 2019][1696490.493333] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1696490.498530] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.505418] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1696490.511216] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.517707] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.525303] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1696490.530423] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.536732] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1696490.543353] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.549628] ldlm_cn00_026 S ffff9146d2b8d140 0 20331 2 0x00000080 [Wed Feb 6 09:39:50 2019][1696490.556923] Call Trace: [Wed Feb 6 09:39:50 2019][1696490.559551] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1696490.564740] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.571630] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1696490.577414] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.583900] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.591486] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1696490.596553] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.602830] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1696490.609459] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.615733] ldlm_cn00_027 S ffff9146d2b8c100 0 20464 2 0x00000080 [Wed Feb 6 09:39:50 2019][1696490.623004] Call Trace: [Wed Feb 6 09:39:50 2019][1696490.625636] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1696490.630829] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.637717] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1696490.643514] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.649997] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.657579] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1696490.662654] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.668963] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1696490.675583] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.681862] ldlm_cn00_028 S ffff9146d2b8a080 0 20558 2 0x00000080 [Wed Feb 6 09:39:50 2019][1696490.689151] Call Trace: [Wed Feb 6 09:39:50 2019][1696490.691781] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1696490.696971] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.703858] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1696490.709647] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.716128] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.723709] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1696490.728766] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.735043] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1696490.741659] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.747934] ldlm_cn03_006 S ffff9146d2b8b0c0 0 20593 2 0x00000080 [Wed Feb 6 09:39:50 2019][1696490.755220] Call Trace: [Wed Feb 6 09:39:50 2019][1696490.757849] [] schedule+0x29/0x70 [Wed Feb 6 09:39:50 2019][1696490.763043] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.769935] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:50 2019][1696490.775728] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.782221] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:50 2019][1696490.789804] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:50 2019][1696490.794870] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:50 2019][1696490.801190] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:50 2019][1696490.807944] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696490.814247] mdt_io02_026 S ffff915946ef1040 0 20859 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696490.821542] Call Trace: [Wed Feb 6 09:39:51 2019][1696490.824180] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696490.829373] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696490.836263] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696490.842061] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696490.848542] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696490.856120] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696490.861180] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696490.867457] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1696490.874071] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696490.880349] mdt_io01_039 S ffff915946ef4100 0 20860 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696490.887624] Call Trace: [Wed Feb 6 09:39:51 2019][1696490.890256] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696490.895443] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696490.902329] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696490.908124] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696490.914615] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696490.922193] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696490.927258] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696490.933528] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1696490.940149] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696490.946451] mdt_io01_040 S ffff9129fb93c4c0 0 20861 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696490.953737] Call Trace: [Wed Feb 6 09:39:51 2019][1696490.956368] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696490.961554] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696490.968431] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696490.974219] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696490.980710] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696490.988297] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696490.993355] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696490.999636] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1696491.006263] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.012537] mdt_io01_041 S ffff915946ef2080 0 20862 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696491.019813] Call Trace: [Wed Feb 6 09:39:51 2019][1696491.022450] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696491.027641] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.034532] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696491.040327] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.046819] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.054402] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696491.059471] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.065743] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1696491.072388] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.078692] mdt_io01_042 S ffff915946ef6180 0 20863 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696491.085963] Call Trace: [Wed Feb 6 09:39:51 2019][1696491.088601] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696491.093797] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.100691] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696491.106489] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.112977] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.120563] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696491.125620] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.131900] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1696491.138521] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.144795] mdt_io01_043 S ffff915946ef0000 0 20864 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696491.152076] Call Trace: [Wed Feb 6 09:39:51 2019][1696491.154707] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696491.159902] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.166784] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696491.172573] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.179059] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.186657] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696491.191713] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.197995] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1696491.204632] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.210910] mdt_io01_044 S ffff9159f6c92080 0 20865 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696491.218199] Call Trace: [Wed Feb 6 09:39:51 2019][1696491.220846] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696491.226039] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.232926] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696491.238726] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.245207] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.252786] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696491.257847] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.264124] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1696491.270751] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.277023] mdt_io01_045 S ffff914aa942a080 0 20866 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696491.284297] Call Trace: [Wed Feb 6 09:39:51 2019][1696491.286929] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696491.292125] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.299006] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696491.304793] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.311282] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.318861] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696491.323921] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.330194] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1696491.336823] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.343103] mdt_io01_046 S ffff9139f700b0c0 0 20867 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696491.350382] Call Trace: [Wed Feb 6 09:39:51 2019][1696491.353013] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696491.358209] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.365093] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696491.370891] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.377373] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.384958] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696491.390021] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.396297] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1696491.402917] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.409197] mdt_io01_047 S ffff9159fab09040 0 20868 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696491.416476] Call Trace: [Wed Feb 6 09:39:51 2019][1696491.419109] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696491.424302] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.431183] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696491.436972] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.443454] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.451035] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696491.456102] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.462378] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1696491.469037] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.475311] mdt_io01_048 S ffff9159fab0b0c0 0 20869 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696491.482588] Call Trace: [Wed Feb 6 09:39:51 2019][1696491.485221] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696491.490410] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.497301] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696491.503097] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.509588] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.515608] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 09:39:51 2019][1696491.528206] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696491.533271] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.539549] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1696491.546166] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.552441] mdt_io01_049 S ffff9159fab08000 0 20870 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696491.559737] Call Trace: [Wed Feb 6 09:39:51 2019][1696491.562376] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696491.567572] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.574461] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696491.580257] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.586746] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.594331] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696491.599406] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.605683] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1696491.612307] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.618587] mdt_io01_050 S ffff9159fab0c100 0 20871 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696491.625862] Call Trace: [Wed Feb 6 09:39:51 2019][1696491.628499] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696491.633696] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.640582] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696491.646380] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.652867] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.660453] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696491.665516] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.671792] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1696491.678419] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.684704] mdt_io01_051 S ffff9159fab0e180 0 20872 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696491.691987] Call Trace: [Wed Feb 6 09:39:51 2019][1696491.694630] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696491.699826] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.706718] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696491.712536] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.719029] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.726610] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696491.731671] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.737950] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1696491.744611] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.750886] mdt_io01_052 S ffff9159fab0a080 0 20873 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696491.758167] Call Trace: [Wed Feb 6 09:39:51 2019][1696491.760800] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696491.765987] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.772876] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696491.778663] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.785150] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.792727] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696491.797791] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.804069] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:51 2019][1696491.810683] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.816955] mdt_io01_053 S ffff915927e31040 0 20874 2 0x00000080 [Wed Feb 6 09:39:51 2019][1696491.824236] Call Trace: [Wed Feb 6 09:39:51 2019][1696491.826867] [] schedule+0x29/0x70 [Wed Feb 6 09:39:51 2019][1696491.832061] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.838950] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:51 2019][1696491.844744] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.851225] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:51 2019][1696491.858802] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:51 2019][1696491.863869] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:51 2019][1696491.870146] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696491.876783] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696491.883069] mdt_io01_054 S ffff915927e30000 0 20875 2 0x00000080 [Wed Feb 6 09:39:52 2019][1696491.890355] Call Trace: [Wed Feb 6 09:39:52 2019][1696491.892985] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1696491.898179] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696491.905068] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1696491.910866] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696491.917351] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696491.924931] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1696491.929988] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696491.936285] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696491.942909] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696491.949185] mdt_io01_055 S ffff915927e32080 0 20876 2 0x00000080 [Wed Feb 6 09:39:52 2019][1696491.956460] Call Trace: [Wed Feb 6 09:39:52 2019][1696491.959092] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1696491.964286] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696491.971174] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1696491.976970] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696491.983458] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696491.991029] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1696491.996087] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.002363] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696492.008988] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.015286] mdt_io01_056 S ffff9159fc535140 0 20877 2 0x00000080 [Wed Feb 6 09:39:52 2019][1696492.022581] Call Trace: [Wed Feb 6 09:39:52 2019][1696492.025223] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1696492.030411] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.037300] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1696492.043095] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.049585] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.057169] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1696492.062228] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.068507] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696492.075134] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.081407] mdt_io01_057 S ffff9159a0eaa080 0 20878 2 0x00000080 [Wed Feb 6 09:39:52 2019][1696492.088690] Call Trace: [Wed Feb 6 09:39:52 2019][1696492.091322] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1696492.096510] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.103401] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1696492.109199] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.115688] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.123262] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1696492.128323] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.134599] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696492.141236] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.147523] ldlm_cn01_014 S ffff914753371040 0 21074 2 0x00000080 [Wed Feb 6 09:39:52 2019][1696492.154873] Call Trace: [Wed Feb 6 09:39:52 2019][1696492.157511] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1696492.162695] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.169623] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1696492.175483] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.182052] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.189633] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1696492.194692] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.200970] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696492.207597] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.213869] ldlm_cn01_015 S ffff9147533730c0 0 21075 2 0x00000080 [Wed Feb 6 09:39:52 2019][1696492.221145] Call Trace: [Wed Feb 6 09:39:52 2019][1696492.223775] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1696492.228970] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.235855] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1696492.241655] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.248145] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.255728] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1696492.260787] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.267064] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696492.273678] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.279956] ldlm_cn01_016 S ffff914753375140 0 21076 2 0x00000080 [Wed Feb 6 09:39:52 2019][1696492.287241] Call Trace: [Wed Feb 6 09:39:52 2019][1696492.289873] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1696492.295059] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.301948] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1696492.307736] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.314350] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.322099] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1696492.327244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.333523] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696492.340148] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.346423] ldlm_cn00_029 S ffff9159f828b0c0 0 21239 2 0x00000080 [Wed Feb 6 09:39:52 2019][1696492.353717] Call Trace: [Wed Feb 6 09:39:52 2019][1696492.356357] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1696492.361548] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.368442] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1696492.374243] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.380729] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.388315] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1696492.393383] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.399659] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696492.406283] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.412561] ldlm_cn00_030 S ffff9159f8288000 0 21242 2 0x00000080 [Wed Feb 6 09:39:52 2019][1696492.419846] Call Trace: [Wed Feb 6 09:39:52 2019][1696492.422482] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1696492.427674] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.434561] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1696492.440362] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.446847] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.454446] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1696492.459607] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.465991] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696492.472703] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.479112] ldlm_cn00_031 S ffff9159aa3d9040 0 21243 2 0x00000080 [Wed Feb 6 09:39:52 2019][1696492.486430] Call Trace: [Wed Feb 6 09:39:52 2019][1696492.489060] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1696492.494251] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.501141] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1696492.506941] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.513429] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.521006] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1696492.526068] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.532348] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696492.538966] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.545240] ldlm_cn03_007 S ffff9149fbc030c0 0 22118 2 0x00000080 [Wed Feb 6 09:39:52 2019][1696492.552532] Call Trace: [Wed Feb 6 09:39:52 2019][1696492.555162] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1696492.560351] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.567233] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1696492.573022] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.579508] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.587089] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1696492.592147] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.598424] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696492.605043] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.611388] mdt_io00_028 S ffff914aa95230c0 0 22161 2 0x00000080 [Wed Feb 6 09:39:52 2019][1696492.618743] Call Trace: [Wed Feb 6 09:39:52 2019][1696492.621437] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1696492.626690] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.633605] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1696492.639403] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.645894] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.653468] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1696492.658537] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.664808] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696492.671429] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.677706] mdt_io00_029 S ffff9129a2605140 0 22644 2 0x00000080 [Wed Feb 6 09:39:52 2019][1696492.684993] Call Trace: [Wed Feb 6 09:39:52 2019][1696492.687632] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1696492.692820] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.699702] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1696492.705494] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.711973] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.719554] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1696492.724620] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.730899] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696492.737526] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.743802] mdt_io03_013 S ffff9146d2b88000 0 22733 2 0x00000080 [Wed Feb 6 09:39:52 2019][1696492.751074] Call Trace: [Wed Feb 6 09:39:52 2019][1696492.753703] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1696492.758893] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.765781] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1696492.771570] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.778054] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.785630] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1696492.790697] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.796970] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696492.803591] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.809869] mdt_rdpg02_006 S ffff9149ca7430c0 0 23025 2 0x00000080 [Wed Feb 6 09:39:52 2019][1696492.817148] Call Trace: [Wed Feb 6 09:39:52 2019][1696492.819784] [] schedule+0x29/0x70 [Wed Feb 6 09:39:52 2019][1696492.824972] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.831859] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:52 2019][1696492.837658] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.844147] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:52 2019][1696492.851730] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:52 2019][1696492.856787] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.863066] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:52 2019][1696492.869689] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:52 2019][1696492.875962] mdt_out02_005 S ffff9149ca744100 0 23072 2 0x00000080 [Wed Feb 6 09:39:53 2019][1696492.883251] Call Trace: [Wed Feb 6 09:39:53 2019][1696492.885882] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1696492.891070] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696492.897949] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1696492.903738] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696492.910225] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696492.917801] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1696492.922867] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696492.929142] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1696492.935762] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696492.942036] mdt_rdpg00_007 S ffff9159aa3d8000 0 23301 2 0x00000080 [Wed Feb 6 09:39:53 2019][1696492.949319] Call Trace: [Wed Feb 6 09:39:53 2019][1696492.951948] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1696492.957138] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696492.964032] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1696492.969826] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696492.976323] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696492.983899] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1696492.988956] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696492.995227] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1696493.001845] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.008124] ldlm_cn03_008 S ffff9159aa3da080 0 23309 2 0x00000080 [Wed Feb 6 09:39:53 2019][1696493.015406] Call Trace: [Wed Feb 6 09:39:53 2019][1696493.018036] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1696493.023222] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.030106] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1696493.035893] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.042377] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.049954] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1696493.055019] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.061302] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1696493.067920] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.074199] ldlm_cn03_009 S ffff9159f8289040 0 23310 2 0x00000080 [Wed Feb 6 09:39:53 2019][1696493.081471] Call Trace: [Wed Feb 6 09:39:53 2019][1696493.084103] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1696493.089291] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.096180] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1696493.101972] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.108465] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.116047] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1696493.121111] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.127391] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1696493.134009] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.140280] ldlm_cn03_010 S ffff9159f828a080 0 23311 2 0x00000080 [Wed Feb 6 09:39:53 2019][1696493.147561] Call Trace: [Wed Feb 6 09:39:53 2019][1696493.150198] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1696493.155386] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.162273] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1696493.168062] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.174543] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.182124] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1696493.187191] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.193466] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1696493.200084] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.206362] ldlm_cn03_011 S ffff9159a4beb0c0 0 23312 2 0x00000080 [Wed Feb 6 09:39:53 2019][1696493.213642] Call Trace: [Wed Feb 6 09:39:53 2019][1696493.216273] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1696493.221461] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.228352] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1696493.234138] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.240624] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.248200] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1696493.253266] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.259544] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1696493.266166] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.272433] ldlm_cn02_012 S ffff9159a4be8000 0 23313 2 0x00000080 [Wed Feb 6 09:39:53 2019][1696493.279700] Call Trace: [Wed Feb 6 09:39:53 2019][1696493.282331] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1696493.287519] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.294398] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1696493.300188] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.306667] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.314251] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1696493.319316] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.325593] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1696493.332214] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.338487] ldlm_cn03_012 S ffff9159f6b91040 0 23314 2 0x00000080 [Wed Feb 6 09:39:53 2019][1696493.345759] Call Trace: [Wed Feb 6 09:39:53 2019][1696493.348389] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1696493.353580] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.360468] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1696493.366255] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.372744] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.380326] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1696493.385393] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.391667] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1696493.398289] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.404568] ldlm_cn01_017 S ffff914aa9f98000 0 23315 2 0x00000080 [Wed Feb 6 09:39:53 2019][1696493.411840] Call Trace: [Wed Feb 6 09:39:53 2019][1696493.414476] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1696493.419665] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.426555] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1696493.432350] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.438840] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.446421] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1696493.451487] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.457763] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1696493.464382] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.470654] ldlm_cn02_013 S ffff914aa9f9b0c0 0 23316 2 0x00000080 [Wed Feb 6 09:39:53 2019][1696493.477942] Call Trace: [Wed Feb 6 09:39:53 2019][1696493.480582] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1696493.485767] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.492657] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:39:53 2019][1696493.499623] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1696493.505420] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.511908] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.519492] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1696493.524557] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.530830] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1696493.537453] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.543727] ldlm_cn02_014 S ffff9159e6618000 0 23317 2 0x00000080 [Wed Feb 6 09:39:53 2019][1696493.551000] Call Trace: [Wed Feb 6 09:39:53 2019][1696493.553633] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1696493.558820] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.565711] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1696493.571500] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.577992] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.585568] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1696493.590632] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.596910] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1696493.603525] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.609802] ldlm_cn02_015 S ffff9159e661b0c0 0 23318 2 0x00000080 [Wed Feb 6 09:39:53 2019][1696493.617091] Call Trace: [Wed Feb 6 09:39:53 2019][1696493.619725] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1696493.624912] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.631793] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1696493.637586] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.644076] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.651663] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1696493.656726] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.663000] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1696493.669620] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.675897] ldlm_cn02_016 S ffff9159e6619040 0 23319 2 0x00000080 [Wed Feb 6 09:39:53 2019][1696493.683192] Call Trace: [Wed Feb 6 09:39:53 2019][1696493.685831] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1696493.691020] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.697907] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1696493.703704] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.710187] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.717768] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1696493.722827] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.729102] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1696493.735723] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.742002] ldlm_cn02_017 S ffff9159e661c100 0 23320 2 0x00000080 [Wed Feb 6 09:39:53 2019][1696493.749294] Call Trace: [Wed Feb 6 09:39:53 2019][1696493.751935] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1696493.757123] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.764036] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1696493.769834] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.776322] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.783895] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1696493.788960] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.795234] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:53 2019][1696493.801855] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.808129] ldlm_cn01_018 S ffff9159e661a080 0 23321 2 0x00000080 [Wed Feb 6 09:39:53 2019][1696493.815405] Call Trace: [Wed Feb 6 09:39:53 2019][1696493.818044] [] schedule+0x29/0x70 [Wed Feb 6 09:39:53 2019][1696493.823234] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.830116] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:53 2019][1696493.835910] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.842394] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:53 2019][1696493.849976] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:53 2019][1696493.855029] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:53 2019][1696493.861298] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696493.867919] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696493.874196] ldlm_cn02_018 S ffff9159e661e180 0 23322 2 0x00000080 [Wed Feb 6 09:39:54 2019][1696493.881489] Call Trace: [Wed Feb 6 09:39:54 2019][1696493.884116] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1696493.889305] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696493.896184] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1696493.901974] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696493.908453] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696493.916036] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1696493.921101] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696493.927380] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696493.933995] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696493.940273] ldlm_cn01_019 S ffff9159e661d140 0 23323 2 0x00000080 [Wed Feb 6 09:39:54 2019][1696493.947565] Call Trace: [Wed Feb 6 09:39:54 2019][1696493.950192] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1696493.955386] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696493.962273] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1696493.968073] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696493.974559] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696493.982134] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1696493.987192] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696493.993467] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696494.000089] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.006368] ldlm_cn02_019 S ffff91599ff9c100 0 23324 2 0x00000080 [Wed Feb 6 09:39:54 2019][1696494.013651] Call Trace: [Wed Feb 6 09:39:54 2019][1696494.016291] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1696494.021475] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.028356] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1696494.034142] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.040630] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.048207] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1696494.053271] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.059550] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696494.066168] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.072439] ldlm_cn02_020 S ffff91599ff99040 0 23325 2 0x00000080 [Wed Feb 6 09:39:54 2019][1696494.079719] Call Trace: [Wed Feb 6 09:39:54 2019][1696494.082355] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1696494.087547] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.094434] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1696494.100234] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.106723] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.114307] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1696494.119362] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.125639] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696494.132257] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.138538] ldlm_cn02_021 S ffff91599ff9d140 0 23326 2 0x00000080 [Wed Feb 6 09:39:54 2019][1696494.145826] Call Trace: [Wed Feb 6 09:39:54 2019][1696494.148459] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1696494.153649] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.160538] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1696494.166339] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.172824] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.180399] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1696494.185456] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.191732] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696494.198350] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.204618] ldlm_cn01_020 S ffff915929b4a080 0 24196 2 0x00000080 [Wed Feb 6 09:39:54 2019][1696494.211893] Call Trace: [Wed Feb 6 09:39:54 2019][1696494.214529] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1696494.219717] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.226600] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1696494.232389] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.238874] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.246452] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1696494.251514] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.257794] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696494.264414] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.270688] mdt_rdpg01_010 S ffff9149fbc00000 0 24682 2 0x00000080 [Wed Feb 6 09:39:54 2019][1696494.277969] Call Trace: [Wed Feb 6 09:39:54 2019][1696494.280600] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1696494.285790] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.292676] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1696494.298471] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.304958] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.312539] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1696494.317601] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.323878] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696494.330495] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.336770] mdt_rdpg01_011 S ffff9149fbc05140 0 24683 2 0x00000080 [Wed Feb 6 09:39:54 2019][1696494.344046] Call Trace: [Wed Feb 6 09:39:54 2019][1696494.346675] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1696494.351870] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.358760] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1696494.364560] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.371049] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.378630] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1696494.383687] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.389960] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696494.396583] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.402857] mdt_rdpg01_012 S ffff9149fbc04100 0 24684 2 0x00000080 [Wed Feb 6 09:39:54 2019][1696494.410150] Call Trace: [Wed Feb 6 09:39:54 2019][1696494.412782] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1696494.417974] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.424860] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1696494.430661] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.437145] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.444721] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1696494.449782] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.456060] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696494.462688] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.468962] mdt_rdpg02_007 S ffff9159a0eab0c0 0 25566 2 0x00000080 [Wed Feb 6 09:39:54 2019][1696494.476242] Call Trace: [Wed Feb 6 09:39:54 2019][1696494.478873] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1696494.484063] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.490953] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1696494.496751] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.503243] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.510827] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1696494.515894] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.522170] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696494.528789] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.535063] ldlm_cn01_021 S ffff9139f6230000 0 27821 2 0x00000080 [Wed Feb 6 09:39:54 2019][1696494.542357] Call Trace: [Wed Feb 6 09:39:54 2019][1696494.544994] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1696494.550181] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.557061] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1696494.562849] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.569331] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.576905] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1696494.581966] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.588242] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696494.594862] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.601137] ldlm_cn01_022 S ffff915929b4d140 0 28087 2 0x00000080 [Wed Feb 6 09:39:54 2019][1696494.608412] Call Trace: [Wed Feb 6 09:39:54 2019][1696494.611044] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1696494.616240] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.623128] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1696494.628919] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.635405] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.642988] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1696494.648052] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.654325] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696494.660947] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.667221] ldlm_cn01_023 S ffff915929b4c100 0 28088 2 0x00000080 [Wed Feb 6 09:39:54 2019][1696494.674503] Call Trace: [Wed Feb 6 09:39:54 2019][1696494.677140] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1696494.682325] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.689205] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1696494.694994] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.701475] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.709052] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1696494.714109] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.720385] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696494.727006] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.733285] ldlm_cn01_024 S ffff9129a5249040 0 28325 2 0x00000080 [Wed Feb 6 09:39:54 2019][1696494.740565] Call Trace: [Wed Feb 6 09:39:54 2019][1696494.743195] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1696494.748387] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.755271] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1696494.761060] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.767542] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.775124] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1696494.780185] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.786452] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696494.793074] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.799349] ldlm_cn01_025 S ffff9129a524b0c0 0 28326 2 0x00000080 [Wed Feb 6 09:39:54 2019][1696494.806625] Call Trace: [Wed Feb 6 09:39:54 2019][1696494.809255] [] schedule+0x29/0x70 [Wed Feb 6 09:39:54 2019][1696494.814446] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.821335] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:54 2019][1696494.827127] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.833610] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:54 2019][1696494.841187] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:54 2019][1696494.846250] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:54 2019][1696494.852520] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:54 2019][1696494.859144] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696494.865416] ldlm_cn01_026 S ffff9129a524d140 0 28327 2 0x00000080 [Wed Feb 6 09:39:55 2019][1696494.872691] Call Trace: [Wed Feb 6 09:39:55 2019][1696494.875323] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1696494.880509] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696494.887388] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1696494.893182] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696494.899667] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696494.907250] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1696494.912308] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696494.918581] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1696494.925199] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696494.931474] ldlm_cn01_027 S ffff9139f6231040 0 28328 2 0x00000080 [Wed Feb 6 09:39:55 2019][1696494.938759] Call Trace: [Wed Feb 6 09:39:55 2019][1696494.941391] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1696494.946578] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696494.953462] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1696494.959259] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696494.965748] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696494.973324] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1696494.978380] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696494.984657] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1696494.991277] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696494.997553] ldlm_cn01_028 S ffff914aa94f30c0 0 28329 2 0x00000080 [Wed Feb 6 09:39:55 2019][1696495.004838] Call Trace: [Wed Feb 6 09:39:55 2019][1696495.007466] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1696495.012654] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.019535] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1696495.025322] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.031808] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.039381] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1696495.044445] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.050719] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1696495.057340] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.063611] ldlm_cn01_029 S ffff9159aa3de180 0 28330 2 0x00000080 [Wed Feb 6 09:39:55 2019][1696495.070886] Call Trace: [Wed Feb 6 09:39:55 2019][1696495.073516] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1696495.078705] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.085585] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1696495.091375] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.097866] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.105446] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1696495.110506] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.116784] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1696495.123403] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.129681] ldlm_cn01_030 S ffff9159aa3dd140 0 28331 2 0x00000080 [Wed Feb 6 09:39:55 2019][1696495.136953] Call Trace: [Wed Feb 6 09:39:55 2019][1696495.139585] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1696495.144774] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.151659] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1696495.157451] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.163940] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.171520] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1696495.176586] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.182862] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1696495.189479] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.195755] ldlm_cn01_031 S ffff9159aa3db0c0 0 28334 2 0x00000080 [Wed Feb 6 09:39:55 2019][1696495.203037] Call Trace: [Wed Feb 6 09:39:55 2019][1696495.205671] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1696495.210858] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.217751] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1696495.223549] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.230041] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.237624] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1696495.242689] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.248961] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1696495.255582] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.261857] ldlm_cn02_022 S ffff91488911c100 0 28933 2 0x00000080 [Wed Feb 6 09:39:55 2019][1696495.269149] Call Trace: [Wed Feb 6 09:39:55 2019][1696495.271781] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1696495.276969] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.283848] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1696495.289642] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.296131] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.303713] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1696495.308771] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.315051] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1696495.321670] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.327940] ldlm_cn02_023 S ffff91488911a080 0 28934 2 0x00000080 [Wed Feb 6 09:39:55 2019][1696495.335207] Call Trace: [Wed Feb 6 09:39:55 2019][1696495.337839] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1696495.343029] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.349914] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1696495.355705] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.362184] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.369768] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1696495.374834] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.381108] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1696495.387724] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.393992] mdt00_026 S ffff9159ac47a080 0 32929 2 0x00000080 [Wed Feb 6 09:39:55 2019][1696495.401268] Call Trace: [Wed Feb 6 09:39:55 2019][1696495.403900] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1696495.409086] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.415966] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:55 2019][1696495.421755] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.428236] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:55 2019][1696495.435818] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1696495.440883] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.447159] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1696495.453777] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.460050] kworker/23:0 S ffff9156a426d140 0 33134 2 0x00000080 [Wed Feb 6 09:39:55 2019][1696495.467333] Call Trace: [Wed Feb 6 09:39:55 2019][1696495.469965] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:55 2019][1696495.476155] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1696495.481302] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:55 2019][1696495.487058] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:55 2019][1696495.493763] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1696495.498821] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.505093] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1696495.511713] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.517994] kworker/20:2 S ffff9129f80a9040 0 33577 2 0x00000080 [Wed Feb 6 09:39:55 2019][1696495.525274] Call Trace: [Wed Feb 6 09:39:55 2019][1696495.527907] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:55 2019][1696495.534091] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1696495.539230] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:55 2019][1696495.544978] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:55 2019][1696495.551690] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1696495.556747] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.563026] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1696495.569647] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.575922] kworker/8:2 S ffff9129c674b0c0 0 34114 2 0x00000080 [Wed Feb 6 09:39:55 2019][1696495.583216] Call Trace: [Wed Feb 6 09:39:55 2019][1696495.585855] [] schedule+0x29/0x70 [Wed Feb 6 09:39:55 2019][1696495.591003] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:55 2019][1696495.596757] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:55 2019][1696495.603466] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:55 2019][1696495.608529] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.614801] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:55 2019][1696495.621421] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:55 2019][1696495.627700] mdt_io03_014 S ffff913f51e10000 0 35138 2 0x00000080 [Wed Feb 6 09:39:55 2019][1696495.632360] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [migration/3:26] [Wed Feb 6 09:39:55 2019][1696495.632405] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 09:39:55 2019][1696495.632424] CPU: 3 PID: 26 Comm: migration/3 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 09:39:55 2019][1696495.632425] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 09:39:55 2019][1696495.632426] task: ffff914aa9f7b0c0 ti: ffff914aa9f90000 task.ti: ffff914aa9f90000 [Wed Feb 6 09:39:55 2019][1696495.632431] RIP: 0010:[] [] multi_cpu_stop+0xb3/0x110 [Wed Feb 6 09:39:55 2019][1696495.632432] RSP: 0000:ffff914aa9f93d98 EFLAGS: 00000246 [Wed Feb 6 09:39:55 2019][1696495.632433] RAX: 0000000000000001 RBX: ffff914aa95e7c00 RCX: dead000000000200 [Wed Feb 6 09:39:55 2019][1696495.632433] RDX: ffff915a3f415ff0 RSI: 0000000000000282 RDI: ffff9139eef23ab0 [Wed Feb 6 09:39:55 2019][1696495.632434] RBP: ffff914aa9f93dc0 R08: ffff9139eef23a50 R09: 0000000000000001 [Wed Feb 6 09:39:55 2019][1696495.632435] R10: 000000000000b449 R11: 0000000000000001 R12: ffff914aa9f93d20 [Wed Feb 6 09:39:55 2019][1696495.632435] R13: ffff9144c9204100 R14: ffffffffadcd2ea0 R15: ffff914aa9f93d00 [Wed Feb 6 09:39:55 2019][1696495.632437] FS: 00007f78f341b880(0000) GS:ffff915a3f400000(0000) knlGS:0000000000000000 [Wed Feb 6 09:39:55 2019][1696495.632438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 09:39:55 2019][1696495.632438] CR2: 00007f78e33efe98 CR3: 0000004033a9a000 CR4: 00000000003407e0 [Wed Feb 6 09:39:55 2019][1696495.632440] Call Trace: [Wed Feb 6 09:39:55 2019][1696495.632443] [] ? cpu_stop_should_run+0x50/0x50 [Wed Feb 6 09:39:55 2019][1696495.632444] [] cpu_stopper_thread+0x99/0x150 [Wed Feb 6 09:39:55 2019][1696495.632448] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:39:56 2019][1696495.632450] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 09:39:56 2019][1696495.632453] [] smpboot_thread_fn+0x144/0x1a0 [Wed Feb 6 09:39:56 2019][1696495.632454] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696495.632456] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1696495.632458] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696495.632460] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1696495.632461] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696495.632479] Code: 20 83 fb 04 74 0a 89 d8 eb b6 66 0f 1f 44 00 00 4c 89 ef 57 9d 66 66 90 66 90 5b 41 5c 41 5d 41 5e 44 89 f8 41 5f 5d c3 83 f8 01 <76> d9 e8 e6 8b 01 00 eb cd 0f 1f 40 00 fa 66 66 90 66 66 90 eb [Wed Feb 6 09:39:56 2019][1696495.928795] Call Trace: [Wed Feb 6 09:39:56 2019][1696495.931426] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1696495.936619] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696495.943509] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:56 2019][1696495.949308] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696495.955790] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696495.963368] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1696495.968436] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696495.974728] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1696495.981347] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696495.987624] mdt02_024 S ffff91370ca3b0c0 0 35530 2 0x00000080 [Wed Feb 6 09:39:56 2019][1696495.994897] Call Trace: [Wed Feb 6 09:39:56 2019][1696495.997535] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:39:56 2019][1696496.004460] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.012219] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1696496.017410] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.024497] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.031585] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:56 2019][1696496.038729] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:56 2019][1696496.044526] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.051858] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:39:56 2019][1696496.059203] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.067225] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:56 2019][1696496.073945] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:39:56 2019][1696496.080923] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:39:56 2019][1696496.087663] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.095702] [] ? htable_lookup+0x8b/0x130 [obdclass] [Wed Feb 6 09:39:56 2019][1696496.102521] [] ? lu_object_find_at+0x1b3/0x280 [obdclass] [Wed Feb 6 09:39:56 2019][1696496.109761] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:56 2019][1696496.116907] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:39:56 2019][1696496.124008] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.130669] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.137761] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.144915] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:39:56 2019][1696496.151637] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:56 2019][1696496.158266] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:39:56 2019][1696496.165096] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.171985] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:56 2019][1696496.179305] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:56 2019][1696496.185879] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.193149] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.200857] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.207172] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.214267] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.222017] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:56 2019][1696496.229286] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.237153] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.244121] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:56 2019][1696496.250572] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:56 2019][1696496.256623] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.263115] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.270696] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1696496.275753] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696496.282030] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1696496.288649] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696496.294923] kworker/12:0 S ffff9129f80ac100 0 38269 2 0x00000080 [Wed Feb 6 09:39:56 2019][1696496.302206] Call Trace: [Wed Feb 6 09:39:56 2019][1696496.304839] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:56 2019][1696496.311028] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1696496.316180] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:56 2019][1696496.321933] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:56 2019][1696496.328641] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1696496.333702] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696496.339978] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1696496.346599] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696496.352869] mdt00_027 S ffff913c8a664100 0 38457 2 0x00000080 [Wed Feb 6 09:39:56 2019][1696496.360146] Call Trace: [Wed Feb 6 09:39:56 2019][1696496.362780] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1696496.367966] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.374854] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:56 2019][1696496.380646] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.387134] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.394710] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1696496.399767] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696496.406048] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1696496.412664] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696496.418945] kworker/47:2 S ffff9156a426e180 0 42267 2 0x00000080 [Wed Feb 6 09:39:56 2019][1696496.426219] Call Trace: [Wed Feb 6 09:39:56 2019][1696496.428852] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:56 2019][1696496.435042] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1696496.440189] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:56 2019][1696496.445945] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:56 2019][1696496.452654] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1696496.457716] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696496.463991] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1696496.470615] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696496.476889] kworker/2:0 S ffff9149fab78000 0 47521 2 0x00000080 [Wed Feb 6 09:39:56 2019][1696496.484186] Call Trace: [Wed Feb 6 09:39:56 2019][1696496.486820] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:56 2019][1696496.493009] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1696496.498157] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:56 2019][1696496.503913] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:56 2019][1696496.510619] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1696496.515682] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696496.521951] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1696496.528564] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696496.534840] mdt03_014 S ffff9159ac47c100 0 47983 2 0x00000080 [Wed Feb 6 09:39:56 2019][1696496.542116] Call Trace: [Wed Feb 6 09:39:56 2019][1696496.544746] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1696496.549938] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.556823] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:56 2019][1696496.562615] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.569104] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.576679] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1696496.581737] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696496.588018] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1696496.594631] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696496.600899] mdt02_025 D ffff9149d56ec100 0 48636 2 0x00000080 [Wed Feb 6 09:39:56 2019][1696496.608174] Call Trace: [Wed Feb 6 09:39:56 2019][1696496.610807] [] schedule+0x29/0x70 [Wed Feb 6 09:39:56 2019][1696496.615959] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 09:39:56 2019][1696496.622583] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 09:39:56 2019][1696496.629465] [] down_write+0x2d/0x3d [Wed Feb 6 09:39:56 2019][1696496.634801] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 09:39:56 2019][1696496.641686] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 09:39:56 2019][1696496.648597] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:56 2019][1696496.655736] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:56 2019][1696496.661528] [] ? ldlm_cli_enqueue_local+0x27d/0x870 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.669024] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:39:56 2019][1696496.677037] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:39:56 2019][1696496.684450] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:39:56 2019][1696496.691684] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:39:56 2019][1696496.698414] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:56 2019][1696496.705736] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:39:56 2019][1696496.712368] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:39:56 2019][1696496.719033] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.725666] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:56 2019][1696496.732299] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:39:56 2019][1696496.738954] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.745847] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:56 2019][1696496.753171] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:56 2019][1696496.759740] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.767006] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.774715] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.781036] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.788134] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.795887] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:56 2019][1696496.803157] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.811029] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.818004] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:56 2019][1696496.824453] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:56 2019][1696496.830511] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.836996] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:56 2019][1696496.844576] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:56 2019][1696496.849633] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:56 2019][1696496.855910] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:56 2019][1696496.862532] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696496.868802] mdt01_037 S ffff9139fbad2080 0 48673 2 0x00000080 [Wed Feb 6 09:39:57 2019][1696496.876078] Call Trace: [Wed Feb 6 09:39:57 2019][1696496.878709] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1696496.883899] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696496.890787] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1696496.896582] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696496.903073] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696496.910652] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1696496.915719] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696496.921992] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1696496.928615] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696496.934890] mdt02_026 S ffff9139fbad1040 0 48676 2 0x00000080 [Wed Feb 6 09:39:57 2019][1696496.942178] Call Trace: [Wed Feb 6 09:39:57 2019][1696496.944812] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1696496.949999] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696496.956876] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1696496.962672] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696496.969161] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696496.976744] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1696496.981806] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696496.988080] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1696496.994694] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696497.000968] mdt01_038 S ffff91264af7e180 0 48703 2 0x00000080 [Wed Feb 6 09:39:57 2019][1696497.008238] Call Trace: [Wed Feb 6 09:39:57 2019][1696497.010869] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1696497.016059] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.022947] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1696497.028733] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.035213] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.042793] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1696497.047855] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696497.054131] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1696497.060751] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696497.067022] mdt01_039 S ffff91264af79040 0 48704 2 0x00000080 [Wed Feb 6 09:39:57 2019][1696497.074299] Call Trace: [Wed Feb 6 09:39:57 2019][1696497.076930] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1696497.082119] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.089009] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1696497.094803] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.101287] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.108864] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1696497.113931] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696497.120207] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1696497.126824] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696497.133099] mdt01_040 S ffff9159f6d1d140 0 48792 2 0x00000080 [Wed Feb 6 09:39:57 2019][1696497.140373] Call Trace: [Wed Feb 6 09:39:57 2019][1696497.143008] [] ? call_usermodehelper_exec+0x92/0x1a0 [Wed Feb 6 09:39:57 2019][1696497.149804] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1696497.154950] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:57 2019][1696497.160978] [] ? mdt_identity_do_upcall+0x134/0x460 [mdt] [Wed Feb 6 09:39:57 2019][1696497.168213] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:57 2019][1696497.174777] [] upcall_cache_get_entry+0x1da/0x8b0 [obdclass] [Wed Feb 6 09:39:57 2019][1696497.182267] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1696497.188035] [] mdt_identity_get+0x17/0x50 [mdt] [Wed Feb 6 09:39:57 2019][1696497.194414] [] old_init_ucred_common+0x11b/0x2e0 [mdt] [Wed Feb 6 09:39:57 2019][1696497.201394] [] mdt_init_ucred_reint+0x142/0x2e0 [mdt] [Wed Feb 6 09:39:57 2019][1696497.208287] [] mdt_reint_internal+0x273/0xaf0 [mdt] [Wed Feb 6 09:39:57 2019][1696497.215007] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [Wed Feb 6 09:39:57 2019][1696497.222249] [] mdt_intent_open+0x82/0x350 [mdt] [Wed Feb 6 09:39:57 2019][1696497.228639] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Wed Feb 6 09:39:57 2019][1696497.235972] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:57 2019][1696497.242597] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [Wed Feb 6 09:39:57 2019][1696497.248930] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Feb 6 09:39:57 2019][1696497.260991] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.267883] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:57 2019][1696497.275207] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:57 2019][1696497.281788] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.289055] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.296757] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.303073] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.310168] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.317920] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:57 2019][1696497.325185] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.333055] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.340029] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:57 2019][1696497.346484] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:57 2019][1696497.352546] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.359033] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.366617] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1696497.371671] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696497.377950] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1696497.384566] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696497.390842] mdt_rdpg02_008 S ffff9159eb752080 0 50796 2 0x00000080 [Wed Feb 6 09:39:57 2019][1696497.398120] Call Trace: [Wed Feb 6 09:39:57 2019][1696497.400750] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1696497.405935] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.412817] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1696497.418606] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.425088] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.432662] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1696497.437722] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696497.443999] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1696497.450624] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696497.456893] mdt_rdpg02_009 S ffff9159eb751040 0 50797 2 0x00000080 [Wed Feb 6 09:39:57 2019][1696497.464168] Call Trace: [Wed Feb 6 09:39:57 2019][1696497.466801] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1696497.471987] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.478868] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1696497.484655] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.491136] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.498713] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1696497.503772] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696497.510054] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1696497.516672] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696497.522947] mdt00_028 S ffff9139f27bd140 0 53421 2 0x00000080 [Wed Feb 6 09:39:57 2019][1696497.530229] Call Trace: [Wed Feb 6 09:39:57 2019][1696497.532857] [] ? call_usermodehelper_exec+0x92/0x1a0 [Wed Feb 6 09:39:57 2019][1696497.539653] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1696497.544801] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:39:57 2019][1696497.550832] [] ? mdt_identity_do_upcall+0x134/0x460 [mdt] [Wed Feb 6 09:39:57 2019][1696497.558063] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:39:57 2019][1696497.564627] [] upcall_cache_get_entry+0x1da/0x8b0 [obdclass] [Wed Feb 6 09:39:57 2019][1696497.572117] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1696497.577885] [] mdt_identity_get+0x17/0x50 [mdt] [Wed Feb 6 09:39:57 2019][1696497.584258] [] old_init_ucred_common+0x11b/0x2e0 [mdt] [Wed Feb 6 09:39:57 2019][1696497.591235] [] mdt_init_ucred_reint+0x142/0x2e0 [mdt] [Wed Feb 6 09:39:57 2019][1696497.598129] [] mdt_reint_internal+0x273/0xaf0 [mdt] [Wed Feb 6 09:39:57 2019][1696497.604846] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [Wed Feb 6 09:39:57 2019][1696497.612087] [] mdt_intent_open+0x82/0x350 [mdt] [Wed Feb 6 09:39:57 2019][1696497.618473] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Wed Feb 6 09:39:57 2019][1696497.625800] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:57 2019][1696497.632426] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [Wed Feb 6 09:39:57 2019][1696497.639778] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.646670] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:57 2019][1696497.653994] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:57 2019][1696497.660565] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.667833] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.675540] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.681858] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.688957] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.696722] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:57 2019][1696497.703992] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.711863] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.718840] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:57 2019][1696497.725285] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:57 2019][1696497.731336] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.737815] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.745393] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1696497.750454] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696497.756725] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1696497.763346] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696497.769621] mdt02_027 S ffff9139f27bc100 0 53422 2 0x00000080 [Wed Feb 6 09:39:57 2019][1696497.776895] Call Trace: [Wed Feb 6 09:39:57 2019][1696497.779528] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1696497.784714] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.791595] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1696497.797384] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.803871] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.811455] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:57 2019][1696497.816522] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696497.822793] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:57 2019][1696497.829419] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:57 2019][1696497.835697] ldlm_cn03_013 S ffff91383a610000 0 59873 2 0x00000080 [Wed Feb 6 09:39:57 2019][1696497.842980] Call Trace: [Wed Feb 6 09:39:57 2019][1696497.845610] [] schedule+0x29/0x70 [Wed Feb 6 09:39:57 2019][1696497.850804] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.857692] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:57 2019][1696497.863490] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:57 2019][1696497.869978] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:58 2019][1696497.877559] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696497.882621] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696497.888896] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696497.895519] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696497.901792] kworker/40:1 S ffff9128f6369040 0 61854 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696497.909083] Call Trace: [Wed Feb 6 09:39:58 2019][1696497.911715] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:58 2019][1696497.917902] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696497.923053] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:58 2019][1696497.928804] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:58 2019][1696497.935512] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696497.940574] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696497.946848] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696497.953471] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696497.959744] ldlm_cn03_014 S ffff9129fc1ab0c0 0 63239 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696497.967018] Call Trace: [Wed Feb 6 09:39:58 2019][1696497.969652] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696497.974843] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:58 2019][1696497.981728] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:58 2019][1696497.987518] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:58 2019][1696497.993998] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:58 2019][1696498.001575] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696498.006633] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.012908] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696498.019528] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.025797] ldlm_cn03_015 S ffff9149d62d2080 0 63246 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696498.033072] Call Trace: [Wed Feb 6 09:39:58 2019][1696498.035703] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696498.040890] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:58 2019][1696498.047770] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:58 2019][1696498.053558] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:58 2019][1696498.060039] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:58 2019][1696498.067616] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696498.072674] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.078950] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696498.085572] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.091850] ldlm_cn03_016 S ffff9149d62d30c0 0 63247 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696498.099130] Call Trace: [Wed Feb 6 09:39:58 2019][1696498.101763] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696498.106950] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:58 2019][1696498.113836] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:58 2019][1696498.119626] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:58 2019][1696498.126107] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:58 2019][1696498.133685] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696498.138745] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.145018] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696498.151639] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.157914] ldlm_cn03_017 S ffff9149d62d6180 0 63248 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696498.165194] Call Trace: [Wed Feb 6 09:39:58 2019][1696498.167831] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696498.173023] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:58 2019][1696498.179906] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:58 2019][1696498.185693] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:58 2019][1696498.192177] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:58 2019][1696498.199753] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696498.204811] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.211086] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696498.217705] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.223973] kworker/41:2 S ffff9139f83ad140 0 65939 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696498.231255] Call Trace: [Wed Feb 6 09:39:58 2019][1696498.233889] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:58 2019][1696498.240084] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696498.245236] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:58 2019][1696498.250987] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:58 2019][1696498.257698] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696498.262757] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.269032] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696498.275656] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.281929] kworker/44:2 S ffff9129fb859040 0 70988 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696498.289208] Call Trace: [Wed Feb 6 09:39:58 2019][1696498.291842] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696498.296991] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:58 2019][1696498.302747] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:58 2019][1696498.309452] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696498.314506] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.320782] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696498.327402] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.333679] kworker/44:0 S ffff9128f63e8000 0 71199 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696498.340957] Call Trace: [Wed Feb 6 09:39:58 2019][1696498.343584] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:58 2019][1696498.349772] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696498.354923] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:58 2019][1696498.360677] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:58 2019][1696498.367382] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696498.372437] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.378713] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696498.385338] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.391607] kworker/0:1 S ffff91293f2f9040 0 77924 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696498.398890] Call Trace: [Wed Feb 6 09:39:58 2019][1696498.401522] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:58 2019][1696498.407711] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696498.412858] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:58 2019][1696498.418614] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:58 2019][1696498.425320] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696498.430376] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.436649] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696498.443271] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.449546] kworker/1:1 S ffff91383a616180 0 81186 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696498.456828] Call Trace: [Wed Feb 6 09:39:58 2019][1696498.459465] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:58 2019][1696498.465648] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696498.470801] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:58 2019][1696498.476555] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:58 2019][1696498.483258] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696498.488317] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.494590] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696498.501208] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.507479] kworker/17:2 S ffff9139f8940000 0 100481 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696498.514863] Call Trace: [Wed Feb 6 09:39:58 2019][1696498.517497] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:58 2019][1696498.523682] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696498.528830] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:58 2019][1696498.534586] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:58 2019][1696498.541293] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696498.546355] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.552630] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696498.559251] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.565525] kworker/14:1 R running task 0 122079 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696498.572883] Call Trace: [Wed Feb 6 09:39:58 2019][1696498.575510] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:58 2019][1696498.581697] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696498.586848] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:58 2019][1696498.592602] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:58 2019][1696498.599313] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696498.604374] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.610649] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696498.617269] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.623545] kworker/36:1 S ffff9128f636b0c0 0 125245 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696498.630929] Call Trace: [Wed Feb 6 09:39:58 2019][1696498.633565] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:58 2019][1696498.639752] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696498.644898] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:58 2019][1696498.650653] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:58 2019][1696498.657361] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696498.662422] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.668697] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696498.675319] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.681594] kworker/45:1 S ffff91383a6130c0 0 131022 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696498.688960] Call Trace: [Wed Feb 6 09:39:58 2019][1696498.691593] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696498.696733] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:58 2019][1696498.702489] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:58 2019][1696498.709200] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696498.714260] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.720534] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696498.727153] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.733424] kworker/10:0 S ffff9149f4062080 0 7757 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696498.740709] Call Trace: [Wed Feb 6 09:39:58 2019][1696498.743334] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:58 2019][1696498.749515] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696498.754664] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:58 2019][1696498.760418] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:58 2019][1696498.767126] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696498.772192] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.778462] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696498.785086] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.791358] kworker/4:2 S ffff911c10074100 0 8526 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696498.798631] Call Trace: [Wed Feb 6 09:39:58 2019][1696498.801264] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:58 2019][1696498.807448] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696498.812593] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:58 2019][1696498.818347] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:58 2019][1696498.825047] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:58 2019][1696498.830111] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.836389] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:58 2019][1696498.843004] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:58 2019][1696498.849273] kworker/22:1 S ffff9149d7abe180 0 11151 2 0x00000080 [Wed Feb 6 09:39:58 2019][1696498.856558] Call Trace: [Wed Feb 6 09:39:58 2019][1696498.859187] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:58 2019][1696498.865373] [] schedule+0x29/0x70 [Wed Feb 6 09:39:58 2019][1696498.870515] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:59 2019][1696498.876268] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:59 2019][1696498.882975] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1696498.888029] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696498.894303] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1696498.900918] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696498.907191] kworker/6:1 S ffff9149d56ee180 0 13218 2 0x00000080 [Wed Feb 6 09:39:59 2019][1696498.914466] Call Trace: [Wed Feb 6 09:39:59 2019][1696498.917097] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:59 2019][1696498.923279] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1696498.928425] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:59 2019][1696498.934182] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:59 2019][1696498.940890] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1696498.945953] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696498.952228] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1696498.958847] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696498.965122] python S ffff9159f6b230c0 0 14075 1 0x00000080 [Wed Feb 6 09:39:59 2019][1696498.972397] Call Trace: [Wed Feb 6 09:39:59 2019][1696498.975033] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:39:59 2019][1696498.981737] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1696498.986887] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:39:59 2019][1696498.994030] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:39:59 2019][1696498.999955] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:39:59 2019][1696499.007267] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:39:59 2019][1696499.013796] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:39:59 2019][1696499.019033] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1696499.024789] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:39:59 2019][1696499.030539] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:39:59 2019][1696499.036727] kworker/u260:0 S ffff915069f30000 0 14084 2 0x00000080 [Wed Feb 6 09:39:59 2019][1696499.044007] Call Trace: [Wed Feb 6 09:39:59 2019][1696499.046642] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:39:59 2019][1696499.052834] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1696499.057977] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:39:59 2019][1696499.063725] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:39:59 2019][1696499.070434] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1696499.075496] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696499.081770] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1696499.088390] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696499.094664] mdt01_041 S ffff91267db49040 0 15163 2 0x00000080 [Wed Feb 6 09:39:59 2019][1696499.101936] Call Trace: [Wed Feb 6 09:39:59 2019][1696499.104608] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.112358] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1696499.117544] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.124624] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.131704] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:59 2019][1696499.138844] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1696499.144631] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.151983] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.160028] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.167806] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Feb 6 09:39:59 2019][1696499.174732] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.182513] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.190536] [] mdt_reint_unlink+0x331/0x14a0 [mdt] [Wed Feb 6 09:39:59 2019][1696499.197166] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 09:39:59 2019][1696499.203366] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 09:39:59 2019][1696499.210085] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Wed Feb 6 09:39:59 2019][1696499.217061] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 09:39:59 2019][1696499.222958] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.230053] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.237814] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:59 2019][1696499.245087] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.252966] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.259941] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:59 2019][1696499.266382] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:59 2019][1696499.272438] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.278916] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.286494] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1696499.291557] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696499.297827] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1696499.304455] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696499.310732] mdt01_042 S ffff91267db4b0c0 0 15164 2 0x00000080 [Wed Feb 6 09:39:59 2019][1696499.318025] Call Trace: [Wed Feb 6 09:39:59 2019][1696499.320654] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1696499.325844] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.332731] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1696499.338519] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.345006] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.352581] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1696499.357644] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696499.363925] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1696499.370543] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696499.376817] mdt00_029 S ffff91267db4d140 0 15165 2 0x00000080 [Wed Feb 6 09:39:59 2019][1696499.384109] Call Trace: [Wed Feb 6 09:39:59 2019][1696499.386739] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1696499.391927] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.398806] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1696499.404598] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.411078] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.418655] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1696499.423714] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696499.429994] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1696499.436612] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696499.442889] mdt00_030 S ffff91267db48000 0 15169 2 0x00000080 [Wed Feb 6 09:39:59 2019][1696499.450168] Call Trace: [Wed Feb 6 09:39:59 2019][1696499.452803] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1696499.457990] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.464876] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1696499.470664] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.477145] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.484722] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1696499.489780] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696499.496057] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1696499.502675] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696499.508943] mdt01_043 S ffff9149f8e94100 0 15170 2 0x00000080 [Wed Feb 6 09:39:59 2019][1696499.516227] Call Trace: [Wed Feb 6 09:39:59 2019][1696499.518861] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1696499.524050] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.530939] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1696499.536739] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.543222] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.550800] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1696499.555857] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696499.562132] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1696499.568760] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696499.575032] mdt02_028 S ffff91383173a080 0 15178 2 0x00000080 [Wed Feb 6 09:39:59 2019][1696499.582316] Call Trace: [Wed Feb 6 09:39:59 2019][1696499.584955] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1696499.590141] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.597028] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1696499.602822] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.609313] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.616898] [] kthread+0xd1/0xe0 [Wed Feb 6 09:39:59 2019][1696499.621958] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696499.628237] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:39:59 2019][1696499.634856] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:39:59 2019][1696499.641131] mdt02_029 D ffff91383173c100 0 15181 2 0x00000080 [Wed Feb 6 09:39:59 2019][1696499.648415] Call Trace: [Wed Feb 6 09:39:59 2019][1696499.651046] [] schedule+0x29/0x70 [Wed Feb 6 09:39:59 2019][1696499.656193] [] rwsem_down_write_failed+0x225/0x3a0 [Wed Feb 6 09:39:59 2019][1696499.662815] [] call_rwsem_down_write_failed+0x17/0x30 [Wed Feb 6 09:39:59 2019][1696499.669695] [] down_write+0x2d/0x3d [Wed Feb 6 09:39:59 2019][1696499.675030] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [Wed Feb 6 09:39:59 2019][1696499.681915] [] lod_qos_prep_create+0x16a/0x1890 [lod] [Wed Feb 6 09:39:59 2019][1696499.688826] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:39:59 2019][1696499.695972] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:39:59 2019][1696499.701760] [] ? ldlm_cli_enqueue_local+0x27d/0x870 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.709254] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:39:59 2019][1696499.717270] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:39:59 2019][1696499.724681] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:39:59 2019][1696499.731914] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:39:59 2019][1696499.738639] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:39:59 2019][1696499.745972] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:39:59 2019][1696499.752600] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:39:59 2019][1696499.759263] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.765894] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:39:59 2019][1696499.772521] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:39:59 2019][1696499.779181] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.786068] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:39:59 2019][1696499.793392] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:39:59 2019][1696499.799967] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.807238] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.814952] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.821271] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.828367] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.836126] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:39:59 2019][1696499.843394] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.851263] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:39:59 2019][1696499.858232] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:39:59 2019][1696499.864677] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:39:59 2019][1696499.870730] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696499.877212] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696499.884786] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1696499.889849] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696499.896123] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1696499.902746] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696499.909022] mdt02_030 S ffff913831739040 0 15182 2 0x00000080 [Wed Feb 6 09:40:00 2019][1696499.916313] Call Trace: [Wed Feb 6 09:40:00 2019][1696499.918957] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1696499.924155] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696499.931046] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1696499.936847] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696499.943337] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696499.950920] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1696499.955985] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696499.962257] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1696499.968879] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696499.975152] mdt02_031 S ffff9149f8e96180 0 15183 2 0x00000080 [Wed Feb 6 09:40:00 2019][1696499.982435] Call Trace: [Wed Feb 6 09:40:00 2019][1696499.985068] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1696499.990255] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696499.997136] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1696500.002928] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.009414] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.016991] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1696500.022052] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.028327] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1696500.034948] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.041223] mdt02_032 S ffff9149f8e91040 0 15184 2 0x00000080 [Wed Feb 6 09:40:00 2019][1696500.048495] Call Trace: [Wed Feb 6 09:40:00 2019][1696500.051161] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.058910] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1696500.064096] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.071181] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.078274] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:00 2019][1696500.085419] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1696500.091212] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.098562] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.106606] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.114370] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Feb 6 09:40:00 2019][1696500.121288] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.129067] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.137090] [] mdt_reint_unlink+0x331/0x14a0 [mdt] [Wed Feb 6 09:40:00 2019][1696500.143725] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 09:40:00 2019][1696500.149923] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 09:40:00 2019][1696500.156642] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Wed Feb 6 09:40:00 2019][1696500.163613] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 09:40:00 2019][1696500.169513] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.176614] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.184371] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:00 2019][1696500.191640] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.199514] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.206486] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:00 2019][1696500.212929] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:00 2019][1696500.218995] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.225485] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.233067] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1696500.238122] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.244399] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1696500.251019] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.257297] mdt02_033 S ffff9149f8e92080 0 15185 2 0x00000080 [Wed Feb 6 09:40:00 2019][1696500.264591] Call Trace: [Wed Feb 6 09:40:00 2019][1696500.267230] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1696500.272427] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.279316] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1696500.285110] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.291595] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.299180] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1696500.304246] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.310522] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1696500.317142] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.323418] mdt02_034 S ffff9149f8e90000 0 15186 2 0x00000080 [Wed Feb 6 09:40:00 2019][1696500.330710] Call Trace: [Wed Feb 6 09:40:00 2019][1696500.333351] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1696500.338539] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.345428] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1696500.351224] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.357709] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.365292] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1696500.370357] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.376635] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1696500.383258] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.389532] mdt01_044 S ffff9149f8e930c0 0 15187 2 0x00000080 [Wed Feb 6 09:40:00 2019][1696500.396805] Call Trace: [Wed Feb 6 09:40:00 2019][1696500.399444] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1696500.404631] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.411520] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1696500.417320] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.423812] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.431395] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1696500.436459] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.442735] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1696500.449356] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.455629] mdt01_045 S ffff9149f72130c0 0 15188 2 0x00000080 [Wed Feb 6 09:40:00 2019][1696500.462923] Call Trace: [Wed Feb 6 09:40:00 2019][1696500.465564] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1696500.470752] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.477640] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1696500.483449] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.489941] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.497522] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1696500.502588] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.508862] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1696500.515486] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.521757] mdt00_031 S ffff9149f7210000 0 15202 2 0x00000080 [Wed Feb 6 09:40:00 2019][1696500.529051] Call Trace: [Wed Feb 6 09:40:00 2019][1696500.531684] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1696500.536871] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.543762] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1696500.549553] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.556038] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.563612] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1696500.568674] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.574946] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1696500.581569] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.587839] mdt00_032 S ffff9149f7216180 0 15218 2 0x00000080 [Wed Feb 6 09:40:00 2019][1696500.595116] Call Trace: [Wed Feb 6 09:40:00 2019][1696500.597746] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1696500.602934] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.609818] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1696500.615616] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.622105] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.629683] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1696500.634746] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.641022] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1696500.647646] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.653919] mdt00_033 S ffff9149f7212080 0 15219 2 0x00000080 [Wed Feb 6 09:40:00 2019][1696500.661193] Call Trace: [Wed Feb 6 09:40:00 2019][1696500.663827] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1696500.669022] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.675908] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1696500.681697] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.688183] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.695759] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1696500.700825] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.707103] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1696500.713731] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.720005] mdt00_034 S ffff9149f3d92080 0 15220 2 0x00000080 [Wed Feb 6 09:40:00 2019][1696500.727287] Call Trace: [Wed Feb 6 09:40:00 2019][1696500.729918] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1696500.735104] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.741988] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1696500.744501] LustreError: 48792:0:(upcall_cache.c:233:upcall_cache_get_entry()) acquire for key 307822: error -110 [Wed Feb 6 09:40:00 2019][1696500.744504] LustreError: 48792:0:(upcall_cache.c:233:upcall_cache_get_entry()) Skipped 3 previous similar messages [Wed Feb 6 09:40:00 2019][1696500.768745] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.775235] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.782820] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:00 2019][1696500.787885] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.794162] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:00 2019][1696500.800787] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:00 2019][1696500.807064] mdt00_035 S ffff9149f3d91040 0 15221 2 0x00000080 [Wed Feb 6 09:40:00 2019][1696500.814337] Call Trace: [Wed Feb 6 09:40:00 2019][1696500.816978] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:40:00 2019][1696500.823898] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.831657] [] schedule+0x29/0x70 [Wed Feb 6 09:40:00 2019][1696500.836844] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.843930] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.851018] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:00 2019][1696500.858164] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:00 2019][1696500.863953] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:00 2019][1696500.871279] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:40:01 2019][1696500.878630] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696500.886659] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:01 2019][1696500.893381] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:40:01 2019][1696500.900365] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:01 2019][1696500.907108] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696500.915136] [] ? lod_attr_get+0xab/0x130 [lod] [Wed Feb 6 09:40:01 2019][1696500.921425] [] ? mdd_la_get+0x84/0x1a0 [mdd] [Wed Feb 6 09:40:01 2019][1696500.927535] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:01 2019][1696500.934685] [] ? mdt_attr_get_complex+0x4ec/0x850 [mdt] [Wed Feb 6 09:40:01 2019][1696500.941749] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:40:01 2019][1696500.948855] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696500.955522] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696500.962616] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696500.969770] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:40:01 2019][1696500.976493] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:01 2019][1696500.983154] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696500.990047] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:40:01 2019][1696500.996877] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.003765] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:01 2019][1696501.011084] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:01 2019][1696501.017656] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.024927] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.032639] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.038966] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.046067] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.053823] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:01 2019][1696501.061092] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.068959] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.075929] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:01 2019][1696501.082378] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:01 2019][1696501.088427] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.094907] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.102484] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:01 2019][1696501.107542] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1696501.113819] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1696501.120436] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1696501.126716] mdt01_046 S ffff9149f3d95140 0 15222 2 0x00000080 [Wed Feb 6 09:40:01 2019][1696501.134000] Call Trace: [Wed Feb 6 09:40:01 2019][1696501.136645] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:40:01 2019][1696501.143569] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.151327] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1696501.156515] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.163602] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.170686] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:01 2019][1696501.177840] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1696501.183635] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.190965] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:40:01 2019][1696501.198312] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.206334] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:01 2019][1696501.213061] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:40:01 2019][1696501.220041] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:01 2019][1696501.226784] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.234798] [] ? lod_attr_get+0xab/0x130 [lod] [Wed Feb 6 09:40:01 2019][1696501.241083] [] ? mdd_la_get+0x84/0x1a0 [mdd] [Wed Feb 6 09:40:01 2019][1696501.247199] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:01 2019][1696501.254351] [] ? mdt_attr_get_complex+0x4ec/0x850 [mdt] [Wed Feb 6 09:40:01 2019][1696501.261417] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:40:01 2019][1696501.268518] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.275187] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.282280] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.289437] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:40:01 2019][1696501.296156] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:01 2019][1696501.302811] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.309706] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:40:01 2019][1696501.316547] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.323434] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:01 2019][1696501.330754] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:01 2019][1696501.337334] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.344603] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.352309] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.358623] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.365722] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.373477] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:01 2019][1696501.380743] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.388614] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.395583] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:01 2019][1696501.402025] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:01 2019][1696501.408083] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.414572] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.422154] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:01 2019][1696501.427216] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1696501.433492] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1696501.440110] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1696501.446382] mdt01_047 S ffff9149f6956180 0 15223 2 0x00000080 [Wed Feb 6 09:40:01 2019][1696501.453662] Call Trace: [Wed Feb 6 09:40:01 2019][1696501.456304] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:40:01 2019][1696501.463226] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.470977] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1696501.476161] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.483251] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.490338] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:01 2019][1696501.497486] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1696501.503278] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.510602] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:40:01 2019][1696501.517956] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.525974] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:01 2019][1696501.532695] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:40:01 2019][1696501.539677] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:01 2019][1696501.546422] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.554432] [] ? lod_attr_get+0xab/0x130 [lod] [Wed Feb 6 09:40:01 2019][1696501.560716] [] ? mdd_la_get+0x84/0x1a0 [mdd] [Wed Feb 6 09:40:01 2019][1696501.566831] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:01 2019][1696501.573982] [] ? mdt_attr_get_complex+0x4ec/0x850 [mdt] [Wed Feb 6 09:40:01 2019][1696501.581041] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:40:01 2019][1696501.588138] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.594806] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.601892] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.609047] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:40:01 2019][1696501.615760] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:01 2019][1696501.622395] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:40:01 2019][1696501.629217] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.636109] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:01 2019][1696501.643429] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:01 2019][1696501.649994] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.657260] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.664961] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.671278] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.678381] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.686135] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:01 2019][1696501.693400] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.701274] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.708253] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:01 2019][1696501.714703] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:01 2019][1696501.720754] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.727239] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.734811] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:01 2019][1696501.739873] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1696501.746146] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1696501.752765] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1696501.759045] mdt01_048 S ffff9149f6952080 0 15224 2 0x00000080 [Wed Feb 6 09:40:01 2019][1696501.766338] Call Trace: [Wed Feb 6 09:40:01 2019][1696501.768972] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1696501.774159] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.781044] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1696501.786832] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.793319] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.800896] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:01 2019][1696501.805959] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1696501.812236] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:01 2019][1696501.818862] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:01 2019][1696501.825136] mdt00_036 S ffff913aa99cb0c0 0 15228 2 0x00000080 [Wed Feb 6 09:40:01 2019][1696501.832412] Call Trace: [Wed Feb 6 09:40:01 2019][1696501.835079] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.842831] [] schedule+0x29/0x70 [Wed Feb 6 09:40:01 2019][1696501.848019] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.855111] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:01 2019][1696501.862202] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:01 2019][1696501.869346] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:01 2019][1696501.875134] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696501.882487] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696501.890534] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696501.898302] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Feb 6 09:40:02 2019][1696501.905224] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696501.913012] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696501.921035] [] mdt_reint_unlink+0x331/0x14a0 [mdt] [Wed Feb 6 09:40:02 2019][1696501.927672] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 09:40:02 2019][1696501.933874] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 09:40:02 2019][1696501.940599] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Wed Feb 6 09:40:02 2019][1696501.947576] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 09:40:02 2019][1696501.953475] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696501.960577] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696501.968333] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:02 2019][1696501.975600] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696501.983471] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696501.990440] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:02 2019][1696501.996886] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:02 2019][1696502.002946] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.009435] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.017016] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1696502.022082] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1696502.028358] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1696502.034979] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1696502.041251] mdt01_049 S ffff913aa9991040 0 15229 2 0x00000080 [Wed Feb 6 09:40:02 2019][1696502.048531] Call Trace: [Wed Feb 6 09:40:02 2019][1696502.051170] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1696502.056356] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.063248] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1696502.069044] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.075532] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.083103] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1696502.088163] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1696502.094437] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1696502.101052] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1696502.107330] mdt01_050 S ffff913aa99930c0 0 15230 2 0x00000080 [Wed Feb 6 09:40:02 2019][1696502.114617] Call Trace: [Wed Feb 6 09:40:02 2019][1696502.117255] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1696502.122438] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.129318] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1696502.135107] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.141589] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.149171] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1696502.154233] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1696502.160513] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1696502.167127] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1696502.173395] mdt01_051 S ffff9149d2e96180 0 15231 2 0x00000080 [Wed Feb 6 09:40:02 2019][1696502.180670] Call Trace: [Wed Feb 6 09:40:02 2019][1696502.183301] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1696502.188489] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.195371] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1696502.201160] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.207638] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.215219] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1696502.220274] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1696502.226555] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1696502.233169] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1696502.239437] mdt00_037 S ffff9149d2e94100 0 15232 2 0x00000080 [Wed Feb 6 09:40:02 2019][1696502.246713] Call Trace: [Wed Feb 6 09:40:02 2019][1696502.249344] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1696502.254535] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.261422] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1696502.267212] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.273705] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.281283] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1696502.286340] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1696502.292622] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1696502.299242] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1696502.305517] mdt01_052 S ffff9149d2e95140 0 15239 2 0x00000080 [Wed Feb 6 09:40:02 2019][1696502.312800] Call Trace: [Wed Feb 6 09:40:02 2019][1696502.315437] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:40:02 2019][1696502.322359] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.330118] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1696502.335307] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.342401] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.349491] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:02 2019][1696502.356639] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1696502.362437] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.369766] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:40:02 2019][1696502.377116] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.385136] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:02 2019][1696502.391854] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:40:02 2019][1696502.398834] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:02 2019][1696502.405580] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.413605] [] ? lod_attr_get+0xab/0x130 [lod] [Wed Feb 6 09:40:02 2019][1696502.419891] [] ? mdd_la_get+0x84/0x1a0 [mdd] [Wed Feb 6 09:40:02 2019][1696502.426005] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:02 2019][1696502.433152] [] ? mdt_attr_get_complex+0x4ec/0x850 [mdt] [Wed Feb 6 09:40:02 2019][1696502.440214] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:40:02 2019][1696502.447311] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.453976] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.461069] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.468222] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:40:02 2019][1696502.474936] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:02 2019][1696502.481565] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:40:02 2019][1696502.488400] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.495287] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:02 2019][1696502.502603] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:02 2019][1696502.509173] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.516438] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.524151] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.530470] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.537571] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.545326] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:02 2019][1696502.552597] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.560476] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.567451] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:02 2019][1696502.573902] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:02 2019][1696502.579954] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.586439] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.594014] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:02 2019][1696502.599077] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1696502.605358] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:02 2019][1696502.611972] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:02 2019][1696502.618240] mdt00_038 S ffff911c10075140 0 15240 2 0x00000080 [Wed Feb 6 09:40:02 2019][1696502.625519] Call Trace: [Wed Feb 6 09:40:02 2019][1696502.628154] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:40:02 2019][1696502.635080] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.642837] [] schedule+0x29/0x70 [Wed Feb 6 09:40:02 2019][1696502.648027] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.655119] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.662206] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:02 2019][1696502.669357] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:02 2019][1696502.675154] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.682478] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:40:02 2019][1696502.689826] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.697845] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:02 2019][1696502.704567] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:40:02 2019][1696502.711547] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:02 2019][1696502.718295] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.726313] [] ? lod_attr_get+0xab/0x130 [lod] [Wed Feb 6 09:40:02 2019][1696502.732592] [] ? mdd_la_get+0x84/0x1a0 [mdd] [Wed Feb 6 09:40:02 2019][1696502.738705] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:02 2019][1696502.745855] [] ? mdt_attr_get_complex+0x4ec/0x850 [mdt] [Wed Feb 6 09:40:02 2019][1696502.752917] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:40:02 2019][1696502.760012] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.766676] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.773770] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.780928] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:40:02 2019][1696502.787648] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:02 2019][1696502.794288] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:40:02 2019][1696502.801117] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.808018] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:02 2019][1696502.815334] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:02 2019][1696502.821907] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.829170] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.836871] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.843190] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.850285] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.858049] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:02 2019][1696502.865317] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.873192] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:02 2019][1696502.880158] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:03 2019][1696502.886607] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:03 2019][1696502.892656] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696502.899134] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696502.899555] LustreError: 53421:0:(upcall_cache.c:233:upcall_cache_get_entry()) acquire for key 298495: error -110 [Wed Feb 6 09:40:03 2019][1696502.917150] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1696502.922210] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696502.928492] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1696502.935110] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696502.941390] mdt01_053 S ffff912918968000 0 15241 2 0x00000080 [Wed Feb 6 09:40:03 2019][1696502.948665] Call Trace: [Wed Feb 6 09:40:03 2019][1696502.951302] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1696502.956490] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696502.963381] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1696502.969178] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696502.975670] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696502.983241] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1696502.988302] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696502.994575] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1696503.001198] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.007475] mdt01_054 S ffff91291896d140 0 15242 2 0x00000080 [Wed Feb 6 09:40:03 2019][1696503.014758] Call Trace: [Wed Feb 6 09:40:03 2019][1696503.017398] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1696503.022595] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.029480] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:40:03 2019][1696503.036445] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1696503.042237] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.048718] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.056293] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1696503.061353] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.067629] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1696503.074249] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.080530] mdt01_055 S ffff91291896a080 0 15243 2 0x00000080 [Wed Feb 6 09:40:03 2019][1696503.087799] Call Trace: [Wed Feb 6 09:40:03 2019][1696503.090429] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1696503.095620] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.102505] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1696503.108301] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.114784] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.122361] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1696503.127421] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.133697] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1696503.140317] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.146595] mdt01_056 S ffff912918969040 0 15244 2 0x00000080 [Wed Feb 6 09:40:03 2019][1696503.153870] Call Trace: [Wed Feb 6 09:40:03 2019][1696503.156497] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1696503.161689] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.168578] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1696503.174373] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.180859] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.188443] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1696503.193508] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.199780] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1696503.206402] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.212677] mdt01_057 S ffff91291896e180 0 15248 2 0x00000080 [Wed Feb 6 09:40:03 2019][1696503.219953] Call Trace: [Wed Feb 6 09:40:03 2019][1696503.222583] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1696503.227771] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.234661] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1696503.240456] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.246953] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.254538] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1696503.259599] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.265878] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1696503.272492] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.278761] mdt02_035 S ffff91291896c100 0 15249 2 0x00000080 [Wed Feb 6 09:40:03 2019][1696503.286045] Call Trace: [Wed Feb 6 09:40:03 2019][1696503.288679] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1696503.293868] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.300753] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1696503.306548] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.313043] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.320622] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1696503.325688] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.331964] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1696503.338590] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.344866] mdt01_058 S ffff912da2b730c0 0 15256 2 0x00000080 [Wed Feb 6 09:40:03 2019][1696503.352139] Call Trace: [Wed Feb 6 09:40:03 2019][1696503.354774] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1696503.359959] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.366845] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1696503.372634] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.379118] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.386698] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1696503.391760] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.398036] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1696503.404656] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.410931] mdt02_036 S ffff912da2b75140 0 15257 2 0x00000080 [Wed Feb 6 09:40:03 2019][1696503.418224] Call Trace: [Wed Feb 6 09:40:03 2019][1696503.420853] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1696503.426043] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.432928] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1696503.438728] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.445218] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.452797] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1696503.457854] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.464131] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1696503.470750] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.477025] mdt01_059 S ffff912da2b76180 0 15258 2 0x00000080 [Wed Feb 6 09:40:03 2019][1696503.484300] Call Trace: [Wed Feb 6 09:40:03 2019][1696503.486931] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1696503.492123] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.499012] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1696503.504807] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.511293] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.518879] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1696503.523942] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.530219] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1696503.536846] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.543121] mdt01_060 S ffff912da2b72080 0 15259 2 0x00000080 [Wed Feb 6 09:40:03 2019][1696503.550407] Call Trace: [Wed Feb 6 09:40:03 2019][1696503.553045] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1696503.558233] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.565125] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1696503.570920] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.577405] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.584982] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1696503.590045] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.596320] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1696503.602940] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.609211] mdt01_061 S ffff912da2b74100 0 15260 2 0x00000080 [Wed Feb 6 09:40:03 2019][1696503.616501] Call Trace: [Wed Feb 6 09:40:03 2019][1696503.619139] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1696503.624330] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.631216] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1696503.637015] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.643511] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.651091] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1696503.656155] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.662431] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1696503.669051] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.675325] mdt00_039 S ffff914321219040 0 15261 2 0x00000080 [Wed Feb 6 09:40:03 2019][1696503.682597] Call Trace: [Wed Feb 6 09:40:03 2019][1696503.685231] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1696503.690425] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.697311] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1696503.703111] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.709602] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.717183] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1696503.722249] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.728528] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1696503.735155] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.741428] mdt02_037 S ffff9149f4060000 0 15265 2 0x00000080 [Wed Feb 6 09:40:03 2019][1696503.748709] Call Trace: [Wed Feb 6 09:40:03 2019][1696503.751343] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1696503.756533] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.763422] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1696503.769217] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.775702] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.783287] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1696503.788347] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.794627] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:03 2019][1696503.801244] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.807521] mdt02_038 S ffff9149f4066180 0 15266 2 0x00000080 [Wed Feb 6 09:40:03 2019][1696503.814795] Call Trace: [Wed Feb 6 09:40:03 2019][1696503.817425] [] schedule+0x29/0x70 [Wed Feb 6 09:40:03 2019][1696503.822614] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.829506] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:03 2019][1696503.835306] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.841795] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:03 2019][1696503.849370] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:03 2019][1696503.854437] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:03 2019][1696503.860713] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1696503.867341] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696503.873617] mdt02_039 S ffff9149f4064100 0 15267 2 0x00000080 [Wed Feb 6 09:40:04 2019][1696503.880897] Call Trace: [Wed Feb 6 09:40:04 2019][1696503.883529] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1696503.888719] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696503.895605] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1696503.901397] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696503.907882] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696503.915464] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1696503.920531] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696503.926806] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1696503.933429] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696503.939702] mdt02_040 S ffff9149d7abb0c0 0 15268 2 0x00000080 [Wed Feb 6 09:40:04 2019][1696503.946994] Call Trace: [Wed Feb 6 09:40:04 2019][1696503.949626] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1696503.954823] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696503.961713] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1696503.967512] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696503.974000] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696503.981592] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1696503.986658] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696503.992936] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1696503.999554] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696504.005828] mdt01_062 S ffff9149d7ab8000 0 15275 2 0x00000080 [Wed Feb 6 09:40:04 2019][1696504.013122] Call Trace: [Wed Feb 6 09:40:04 2019][1696504.015750] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1696504.020939] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.027826] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1696504.033622] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.040108] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.047686] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1696504.052753] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696504.059027] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1696504.065647] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696504.071919] mdt01_063 S ffff9149d7aba080 0 15276 2 0x00000080 [Wed Feb 6 09:40:04 2019][1696504.079195] Call Trace: [Wed Feb 6 09:40:04 2019][1696504.081825] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1696504.087014] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.093897] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1696504.099687] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.106180] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.113763] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1696504.118827] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696504.125105] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1696504.131732] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696504.138005] mdt00_040 S ffff9149d7abc100 0 15277 2 0x00000080 [Wed Feb 6 09:40:04 2019][1696504.145280] Call Trace: [Wed Feb 6 09:40:04 2019][1696504.147911] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1696504.153099] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.159982] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1696504.165770] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.172258] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.179834] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1696504.184897] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696504.191172] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1696504.197797] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696504.204072] mdt00_041 S ffff914997ef9040 0 15278 2 0x00000080 [Wed Feb 6 09:40:04 2019][1696504.211348] Call Trace: [Wed Feb 6 09:40:04 2019][1696504.213982] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1696504.219171] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.226057] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1696504.231849] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.238342] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.245921] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1696504.250978] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696504.257257] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1696504.263876] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696504.270150] mdt00_042 S ffff914997efd140 0 15279 2 0x00000080 [Wed Feb 6 09:40:04 2019][1696504.277425] Call Trace: [Wed Feb 6 09:40:04 2019][1696504.280062] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:40:04 2019][1696504.286986] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.294743] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1696504.299922] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.307008] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.314099] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:04 2019][1696504.321247] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1696504.327036] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.334367] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:40:04 2019][1696504.341718] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.349745] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:04 2019][1696504.356464] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:40:04 2019][1696504.363442] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:04 2019][1696504.370188] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.378209] [] ? lod_attr_get+0xab/0x130 [lod] [Wed Feb 6 09:40:04 2019][1696504.384491] [] ? mdd_la_get+0x84/0x1a0 [mdd] [Wed Feb 6 09:40:04 2019][1696504.390608] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:04 2019][1696504.397759] [] ? mdt_attr_get_complex+0x4ec/0x850 [mdt] [Wed Feb 6 09:40:04 2019][1696504.404824] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:40:04 2019][1696504.411928] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.418594] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.425689] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.432846] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:40:04 2019][1696504.439566] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:04 2019][1696504.446196] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:40:04 2019][1696504.453032] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.459927] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:04 2019][1696504.467254] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:04 2019][1696504.473824] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.481090] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.488804] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.495128] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.502230] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.509992] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:04 2019][1696504.517270] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.525159] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.532134] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:04 2019][1696504.538589] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:04 2019][1696504.544647] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.551137] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.558719] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1696504.563787] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696504.570059] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1696504.576684] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696504.582957] mdt00_043 S ffff914997efa080 0 15280 2 0x00000080 [Wed Feb 6 09:40:04 2019][1696504.590234] Call Trace: [Wed Feb 6 09:40:04 2019][1696504.592870] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1696504.598057] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.604951] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1696504.610752] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.617234] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.624829] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:04 2019][1696504.629886] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696504.636167] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:04 2019][1696504.642786] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:04 2019][1696504.649066] mdt00_044 S ffff914997efc100 0 15281 2 0x00000080 [Wed Feb 6 09:40:04 2019][1696504.656364] Call Trace: [Wed Feb 6 09:40:04 2019][1696504.659004] [] schedule+0x29/0x70 [Wed Feb 6 09:40:04 2019][1696504.664160] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:40:04 2019][1696504.670175] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:40:04 2019][1696504.676732] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [Wed Feb 6 09:40:04 2019][1696504.683622] [] osp_precreate_reserve+0x2e8/0x800 [osp] [Wed Feb 6 09:40:04 2019][1696504.690595] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:04 2019][1696504.696356] [] osp_declare_create+0x199/0x5b0 [osp] [Wed Feb 6 09:40:04 2019][1696504.703087] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Wed Feb 6 09:40:04 2019][1696504.710422] [] lod_sub_declare_create+0xdf/0x210 [lod] [Wed Feb 6 09:40:04 2019][1696504.717395] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [Wed Feb 6 09:40:04 2019][1696504.724629] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [Wed Feb 6 09:40:04 2019][1696504.732081] [] ? ptlrpc_request_cache_free+0x90/0x1e0 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.739754] [] ? lod_qos_calc_weight+0x80/0x80 [lod] [Wed Feb 6 09:40:04 2019][1696504.746560] [] ? lod_obj_is_ost_use_skip_cb+0x30/0x30 [lod] [Wed Feb 6 09:40:04 2019][1696504.753971] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [Wed Feb 6 09:40:04 2019][1696504.760977] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.768054] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:04 2019][1696504.775235] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.783165] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:40:04 2019][1696504.791178] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:40:04 2019][1696504.798592] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:40:04 2019][1696504.805827] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:40:04 2019][1696504.812549] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:04 2019][1696504.819876] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:40:04 2019][1696504.826518] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:40:04 2019][1696504.833190] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.839820] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:04 2019][1696504.846477] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.853368] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:40:04 2019][1696504.860040] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:04 2019][1696504.866931] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:05 2019][1696504.874259] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:05 2019][1696504.880833] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696504.888104] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696504.895815] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696504.902129] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696504.909229] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696504.916989] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:05 2019][1696504.924268] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696504.932145] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696504.939119] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:05 2019][1696504.945571] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:05 2019][1696504.951630] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696504.958123] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696504.965701] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1696504.970768] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1696504.977042] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1696504.983661] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1696504.989937] mdt01_064 S ffff913aa9804100 0 15291 2 0x00000080 [Wed Feb 6 09:40:05 2019][1696504.997210] Call Trace: [Wed Feb 6 09:40:05 2019][1696504.999841] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1696505.005028] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.011909] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1696505.017696] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.024178] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.031763] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1696505.036820] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1696505.043090] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1696505.049709] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1696505.055989] mdt01_065 S ffff91432121a080 0 15292 2 0x00000080 [Wed Feb 6 09:40:05 2019][1696505.063282] Call Trace: [Wed Feb 6 09:40:05 2019][1696505.065924] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1696505.071118] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.078002] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1696505.083793] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.090274] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.097850] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1696505.102912] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1696505.109185] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1696505.115806] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1696505.122077] mdt01_066 S ffff9149fec29040 0 15293 2 0x00000080 [Wed Feb 6 09:40:05 2019][1696505.129354] Call Trace: [Wed Feb 6 09:40:05 2019][1696505.131984] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1696505.137174] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.144061] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1696505.149856] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.156339] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.163917] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1696505.168972] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1696505.175242] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1696505.181860] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1696505.188131] mdt01_067 S ffff9149fec28000 0 15294 2 0x00000080 [Wed Feb 6 09:40:05 2019][1696505.195405] Call Trace: [Wed Feb 6 09:40:05 2019][1696505.198040] [] ? ktime_[-- root@localhost attached -- Wed Feb 6 09:40:05 2019] get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:40:05 2019][1696505.204967] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.212723] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1696505.217911] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.225010] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.232098] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:05 2019][1696505.239250] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1696505.245041] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.252366] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:40:05 2019][1696505.259722] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.267741] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:05 2019][1696505.274465] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:40:05 2019][1696505.281437] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:05 2019][1696505.288179] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.296203] [] ? lod_attr_get+0xab/0x130 [lod] [Wed Feb 6 09:40:05 2019][1696505.302492] [] ? mdd_la_get+0x84/0x1a0 [mdd] [Wed Feb 6 09:40:05 2019][1696505.308607] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:05 2019][1696505.315755] [] ? mdt_attr_get_complex+0x4ec/0x850 [mdt] [Wed Feb 6 09:40:05 2019][1696505.322811] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:40:05 2019][1696505.329909] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.336576] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.343665] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.350820] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:40:05 2019][1696505.357535] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:05 2019][1696505.364169] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:40:05 2019][1696505.370999] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.377895] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:05 2019][1696505.385212] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:05 2019][1696505.391788] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.399048] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.406764] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.413077] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.420173] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.427939] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:05 2019][1696505.435200] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.443068] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.450042] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:05 2019][1696505.456490] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:05 2019][1696505.462549] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.469038] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.476623] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1696505.481685] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1696505.487964] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1696505.494585] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1696505.500855] mdt02_041 S ffff9149d56ed140 0 15329 2 0x00000080 [Wed Feb 6 09:40:05 2019][1696505.508132] Call Trace: [Wed Feb 6 09:40:05 2019][1696505.510767] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1696505.515912] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:40:05 2019][1696505.521932] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:40:05 2019][1696505.528476] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [Wed Feb 6 09:40:05 2019][1696505.535365] [] osp_precreate_reserve+0x2e8/0x800 [osp] [Wed Feb 6 09:40:05 2019][1696505.542342] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1696505.548093] [] osp_declare_create+0x199/0x5b0 [osp] [Wed Feb 6 09:40:05 2019][1696505.554829] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Wed Feb 6 09:40:05 2019][1696505.562157] [] lod_sub_declare_create+0xdf/0x210 [lod] [Wed Feb 6 09:40:05 2019][1696505.569133] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [Wed Feb 6 09:40:05 2019][1696505.576372] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [Wed Feb 6 09:40:05 2019][1696505.583782] [] ? lod_qos_calc_weight+0x80/0x80 [lod] [Wed Feb 6 09:40:05 2019][1696505.590580] [] ? lod_obj_is_ost_use_skip_cb+0x30/0x30 [lod] [Wed Feb 6 09:40:05 2019][1696505.597983] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [Wed Feb 6 09:40:05 2019][1696505.604982] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:05 2019][1696505.612123] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1696505.617883] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:40:05 2019][1696505.625902] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:40:05 2019][1696505.633315] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:40:05 2019][1696505.640551] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:40:05 2019][1696505.647274] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:05 2019][1696505.654601] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:40:05 2019][1696505.661231] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:40:05 2019][1696505.667895] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.674528] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:05 2019][1696505.681184] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.688073] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:40:05 2019][1696505.694738] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.701632] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:05 2019][1696505.708953] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:05 2019][1696505.715530] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.722791] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.730496] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.736817] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.743910] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.751666] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:05 2019][1696505.758937] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.766811] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.773778] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:05 2019][1696505.780224] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:05 2019][1696505.786276] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.792756] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.800332] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:05 2019][1696505.805391] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1696505.811670] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:05 2019][1696505.818287] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:05 2019][1696505.824569] mdt02_042 S ffff911b02b6a080 0 15330 2 0x00000080 [Wed Feb 6 09:40:05 2019][1696505.831849] Call Trace: [Wed Feb 6 09:40:05 2019][1696505.834480] [] schedule+0x29/0x70 [Wed Feb 6 09:40:05 2019][1696505.839670] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.846553] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:05 2019][1696505.852344] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.858827] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:05 2019][1696505.866401] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1696505.871467] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696505.877738] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1696505.884355] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696505.890633] mdt02_043 S ffff911b02b6b0c0 0 15331 2 0x00000080 [Wed Feb 6 09:40:06 2019][1696505.897917] Call Trace: [Wed Feb 6 09:40:06 2019][1696505.900548] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1696505.905738] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696505.912630] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1696505.918423] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696505.924915] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696505.932496] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1696505.937556] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696505.943832] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1696505.950450] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696505.956727] mdt02_044 S ffff911b02b69040 0 15370 2 0x00000080 [Wed Feb 6 09:40:06 2019][1696505.964000] Call Trace: [Wed Feb 6 09:40:06 2019][1696505.966632] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1696505.971777] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:40:06 2019][1696505.977787] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:40:06 2019][1696505.984330] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [Wed Feb 6 09:40:06 2019][1696505.991218] [] osp_precreate_reserve+0x2e8/0x800 [osp] [Wed Feb 6 09:40:06 2019][1696505.998188] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1696506.003946] [] osp_declare_create+0x199/0x5b0 [osp] [Wed Feb 6 09:40:06 2019][1696506.010676] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Wed Feb 6 09:40:06 2019][1696506.018005] [] lod_sub_declare_create+0xdf/0x210 [lod] [Wed Feb 6 09:40:06 2019][1696506.024981] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [Wed Feb 6 09:40:06 2019][1696506.032210] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [Wed Feb 6 09:40:06 2019][1696506.039613] [] ? lod_qos_calc_weight+0x80/0x80 [lod] [Wed Feb 6 09:40:06 2019][1696506.046417] [] ? lod_obj_is_ost_use_skip_cb+0x30/0x30 [lod] [Wed Feb 6 09:40:06 2019][1696506.053826] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [Wed Feb 6 09:40:06 2019][1696506.060814] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:06 2019][1696506.067958] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1696506.073718] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:40:06 2019][1696506.081733] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:40:06 2019][1696506.089145] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:40:06 2019][1696506.096377] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:40:06 2019][1696506.103106] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:06 2019][1696506.110428] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:40:06 2019][1696506.117066] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:40:06 2019][1696506.123728] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.130362] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:06 2019][1696506.136987] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:40:06 2019][1696506.143648] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.150544] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:06 2019][1696506.157864] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:06 2019][1696506.164441] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.171738] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.179444] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.185764] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.192865] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.200627] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:06 2019][1696506.207898] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.215786] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.222759] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:06 2019][1696506.229208] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:06 2019][1696506.235263] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.241749] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.249327] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1696506.254394] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696506.260677] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1696506.267300] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696506.273570] mdt02_045 S ffff911b02b6c100 0 15371 2 0x00000080 [Wed Feb 6 09:40:06 2019][1696506.280848] Call Trace: [Wed Feb 6 09:40:06 2019][1696506.283476] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1696506.288680] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.295569] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1696506.301362] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.307849] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.315423] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1696506.320486] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696506.326762] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1696506.333382] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696506.339654] mdt02_046 S ffff911b02b6d140 0 15372 2 0x00000080 [Wed Feb 6 09:40:06 2019][1696506.346938] Call Trace: [Wed Feb 6 09:40:06 2019][1696506.349568] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1696506.354756] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.361640] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1696506.367434] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.373921] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.381505] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1696506.386567] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696506.392848] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1696506.399466] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696506.405739] mdt01_068 S ffff9149fec2a080 0 15382 2 0x00000080 [Wed Feb 6 09:40:06 2019][1696506.413013] Call Trace: [Wed Feb 6 09:40:06 2019][1696506.415646] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1696506.420844] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.427735] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1696506.433540] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.440037] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.447618] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1696506.452678] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696506.458960] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1696506.465575] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696506.471854] mdt01_069 S ffff913f51e12080 0 15383 2 0x00000080 [Wed Feb 6 09:40:06 2019][1696506.479145] Call Trace: [Wed Feb 6 09:40:06 2019][1696506.481778] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1696506.486968] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.493854] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1696506.499661] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.506160] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.513752] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1696506.518811] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696506.525083] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1696506.531706] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696506.537981] mdt01_070 S ffff913f51e15140 0 15384 2 0x00000080 [Wed Feb 6 09:40:06 2019][1696506.545262] Call Trace: [Wed Feb 6 09:40:06 2019][1696506.547894] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1696506.553091] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.559982] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1696506.565780] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.572267] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.579845] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:06 2019][1696506.584902] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696506.591177] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:06 2019][1696506.597796] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:06 2019][1696506.604070] mdt01_071 S ffff913f51e14100 0 15385 2 0x00000080 [Wed Feb 6 09:40:06 2019][1696506.611338] Call Trace: [Wed Feb 6 09:40:06 2019][1696506.613978] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:40:06 2019][1696506.620895] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.628648] [] schedule+0x29/0x70 [Wed Feb 6 09:40:06 2019][1696506.633831] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.640911] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.647991] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:06 2019][1696506.655137] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:06 2019][1696506.660934] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.668260] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:40:06 2019][1696506.675609] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.683638] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:06 2019][1696506.690359] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:40:06 2019][1696506.697338] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:06 2019][1696506.704081] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.712100] [] ? lod_attr_get+0xab/0x130 [lod] [Wed Feb 6 09:40:06 2019][1696506.718380] [] ? mdd_la_get+0x84/0x1a0 [mdd] [Wed Feb 6 09:40:06 2019][1696506.724493] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:06 2019][1696506.731644] [] ? mdt_attr_get_complex+0x4ec/0x850 [mdt] [Wed Feb 6 09:40:06 2019][1696506.738719] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:40:06 2019][1696506.745842] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.752511] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.759603] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.766754] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:40:06 2019][1696506.773473] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:06 2019][1696506.780106] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:40:06 2019][1696506.786938] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.793831] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:06 2019][1696506.801159] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:06 2019][1696506.807732] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.814998] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.822707] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.829026] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.836126] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.843883] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:06 2019][1696506.851153] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.859020] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:06 2019][1696506.865988] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:06 2019][1696506.872433] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:07 2019][1696506.878486] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696506.884967] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696506.892548] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1696506.897601] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1696506.903883] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1696506.910500] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1696506.916773] mdt00_045 S ffff911c100730c0 0 15386 2 0x00000080 [Wed Feb 6 09:40:07 2019][1696506.924058] Call Trace: [Wed Feb 6 09:40:07 2019][1696506.926690] [] ? ttwu_do_activate+0x6f/0x80 [Wed Feb 6 09:40:07 2019][1696506.932704] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1696506.937856] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:40:07 2019][1696506.943867] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:40:07 2019][1696506.950408] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [Wed Feb 6 09:40:07 2019][1696506.957297] [] osp_precreate_reserve+0x2e8/0x800 [osp] [Wed Feb 6 09:40:07 2019][1696506.964267] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1696506.970027] [] osp_declare_create+0x199/0x5b0 [osp] [Wed Feb 6 09:40:07 2019][1696506.976759] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Wed Feb 6 09:40:07 2019][1696506.984090] [] lod_sub_declare_create+0xdf/0x210 [lod] [Wed Feb 6 09:40:07 2019][1696506.991069] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [Wed Feb 6 09:40:07 2019][1696506.998302] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [Wed Feb 6 09:40:07 2019][1696507.005712] [] ? lod_qos_calc_weight+0x80/0x80 [lod] [Wed Feb 6 09:40:07 2019][1696507.012513] [] ? lod_obj_is_ost_use_skip_cb+0x30/0x30 [lod] [Wed Feb 6 09:40:07 2019][1696507.019923] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [Wed Feb 6 09:40:07 2019][1696507.026913] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:07 2019][1696507.034054] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1696507.039816] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:40:07 2019][1696507.047847] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:40:07 2019][1696507.055257] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:40:07 2019][1696507.062501] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:40:07 2019][1696507.069221] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:07 2019][1696507.076542] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:40:07 2019][1696507.083173] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:40:07 2019][1696507.089838] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.096474] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:07 2019][1696507.103110] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:40:07 2019][1696507.109781] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.116678] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:07 2019][1696507.123994] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:07 2019][1696507.130566] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.137831] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.145541] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.151868] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.158964] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.166725] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:07 2019][1696507.173998] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.181875] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.188849] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:07 2019][1696507.195297] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:07 2019][1696507.201349] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.207842] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.215432] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1696507.220490] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1696507.226767] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1696507.233399] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1696507.239667] mdt01_072 S ffff911c10076180 0 15388 2 0x00000080 [Wed Feb 6 09:40:07 2019][1696507.246942] Call Trace: [Wed Feb 6 09:40:07 2019][1696507.249580] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1696507.254776] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.261659] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1696507.267454] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.273942] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.281524] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1696507.286585] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1696507.292871] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1696507.299492] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1696507.305759] mdt01_073 S ffff911c10072080 0 15389 2 0x00000080 [Wed Feb 6 09:40:07 2019][1696507.313028] Call Trace: [Wed Feb 6 09:40:07 2019][1696507.315662] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1696507.320849] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.327733] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1696507.333525] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.340008] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.347585] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1696507.352658] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1696507.358936] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1696507.365555] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1696507.371827] mdt01_074 S ffff9149d56e8000 0 15390 2 0x00000080 [Wed Feb 6 09:40:07 2019][1696507.379104] Call Trace: [Wed Feb 6 09:40:07 2019][1696507.381734] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1696507.386926] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.393812] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1696507.399600] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.406083] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.413662] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1696507.418726] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1696507.425006] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1696507.431620] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1696507.437899] mdt00_046 S ffff9149d56e9040 0 15391 2 0x00000080 [Wed Feb 6 09:40:07 2019][1696507.445181] Call Trace: [Wed Feb 6 09:40:07 2019][1696507.447814] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1696507.453004] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.459887] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1696507.465681] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.472167] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.479742] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1696507.484806] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1696507.491089] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1696507.497712] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1696507.503987] mdt01_075 S ffff9149d56eb0c0 0 15392 2 0x00000080 [Wed Feb 6 09:40:07 2019][1696507.511266] Call Trace: [Wed Feb 6 09:40:07 2019][1696507.513899] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1696507.519090] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.525978] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1696507.531777] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.538261] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.545841] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:07 2019][1696507.550907] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1696507.557179] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:07 2019][1696507.563806] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:07 2019][1696507.570088] mdt01_076 S ffff9149d56ea080 0 15393 2 0x00000080 [Wed Feb 6 09:40:07 2019][1696507.577357] Call Trace: [Wed Feb 6 09:40:07 2019][1696507.579991] [] schedule+0x29/0x70 [Wed Feb 6 09:40:07 2019][1696507.585137] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:40:07 2019][1696507.591154] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:40:07 2019][1696507.597699] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [Wed Feb 6 09:40:07 2019][1696507.604594] [] osp_precreate_reserve+0x2e8/0x800 [osp] [Wed Feb 6 09:40:07 2019][1696507.611577] [] ? osd_trunc_lock+0xe0/0x210 [osd_ldiskfs] [Wed Feb 6 09:40:07 2019][1696507.618720] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1696507.624487] [] osp_declare_create+0x199/0x5b0 [osp] [Wed Feb 6 09:40:07 2019][1696507.631224] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Wed Feb 6 09:40:07 2019][1696507.638557] [] lod_sub_declare_create+0xdf/0x210 [lod] [Wed Feb 6 09:40:07 2019][1696507.645533] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [Wed Feb 6 09:40:07 2019][1696507.652766] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [Wed Feb 6 09:40:07 2019][1696507.660173] [] ? del_timer_sync+0x52/0x60 [Wed Feb 6 09:40:07 2019][1696507.666018] [] ? lod_qos_calc_weight+0x80/0x80 [lod] [Wed Feb 6 09:40:07 2019][1696507.672822] [] ? lod_obj_is_ost_use_skip_cb+0x30/0x30 [lod] [Wed Feb 6 09:40:07 2019][1696507.680229] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [Wed Feb 6 09:40:07 2019][1696507.687225] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:07 2019][1696507.694372] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:07 2019][1696507.700131] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:40:07 2019][1696507.708145] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:40:07 2019][1696507.715556] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:40:07 2019][1696507.722790] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:40:07 2019][1696507.729508] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:07 2019][1696507.736837] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:40:07 2019][1696507.743471] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:40:07 2019][1696507.750141] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.756781] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:07 2019][1696507.763408] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:40:07 2019][1696507.770068] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.776961] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:07 2019][1696507.784281] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:07 2019][1696507.790847] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.798113] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.805823] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.812145] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.819239] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.827000] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:07 2019][1696507.834269] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.842140] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.849108] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:07 2019][1696507.855555] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:07 2019][1696507.861606] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.868093] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:07 2019][1696507.875680] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696507.880743] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696507.887015] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1696507.893638] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696507.899910] mdt00_047 S ffff915069f330c0 0 15406 2 0x00000080 [Wed Feb 6 09:40:08 2019][1696507.907186] Call Trace: [Wed Feb 6 09:40:08 2019][1696507.909817] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1696507.915006] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696507.921893] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1696507.927687] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696507.934187] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696507.941773] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696507.946836] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696507.953113] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1696507.959738] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696507.966012] mdt00_048 S ffff915069f36180 0 15407 2 0x00000080 [Wed Feb 6 09:40:08 2019][1696507.973292] Call Trace: [Wed Feb 6 09:40:08 2019][1696507.975930] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1696507.981118] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696507.988007] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1696507.993803] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.000285] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.007863] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696508.012931] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.019206] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1696508.025821] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.032089] mdt00_049 S ffff915069f35140 0 15408 2 0x00000080 [Wed Feb 6 09:40:08 2019][1696508.039364] Call Trace: [Wed Feb 6 09:40:08 2019][1696508.041996] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1696508.047192] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.054072] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1696508.059867] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.066354] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.073935] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696508.078995] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.085275] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1696508.091893] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.098168] mdt00_050 S ffff915069f32080 0 15409 2 0x00000080 [Wed Feb 6 09:40:08 2019][1696508.105442] Call Trace: [Wed Feb 6 09:40:08 2019][1696508.108072] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1696508.113263] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.120154] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1696508.125947] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.132448] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.140030] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696508.145088] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.151368] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1696508.157985] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.164257] mdt_io01_058 S ffff915069f31040 0 15425 2 0x00000080 [Wed Feb 6 09:40:08 2019][1696508.171535] Call Trace: [Wed Feb 6 09:40:08 2019][1696508.174167] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1696508.179354] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.186240] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1696508.192033] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.198516] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.206102] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696508.211171] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.217453] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1696508.224068] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.230344] mdt_io01_059 S ffff915929b49040 0 15426 2 0x00000080 [Wed Feb 6 09:40:08 2019][1696508.237625] Call Trace: [Wed Feb 6 09:40:08 2019][1696508.240262] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1696508.245457] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.252340] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1696508.258138] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.264630] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.272209] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696508.277266] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.283548] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1696508.290180] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.296459] mdt_io01_060 S ffff915929b48000 0 15427 2 0x00000080 [Wed Feb 6 09:40:08 2019][1696508.303731] Call Trace: [Wed Feb 6 09:40:08 2019][1696508.306362] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1696508.311553] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.318437] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1696508.324225] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.330711] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.338291] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696508.343350] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.349622] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1696508.356242] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.362517] mdt_io01_061 S ffff915929b4e180 0 15428 2 0x00000080 [Wed Feb 6 09:40:08 2019][1696508.369790] Call Trace: [Wed Feb 6 09:40:08 2019][1696508.372421] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1696508.377610] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.384494] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1696508.390285] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.396767] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.404343] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696508.409403] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.415678] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1696508.422297] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.428571] mdt_io02_027 S ffff913e10668000 0 15429 2 0x00000080 [Wed Feb 6 09:40:08 2019][1696508.435840] Call Trace: [Wed Feb 6 09:40:08 2019][1696508.438469] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1696508.443658] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.450537] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1696508.456328] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.462808] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.470384] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696508.475442] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.481725] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1696508.488344] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.494618] mdt_io01_062 S ffff913e10669040 0 15430 2 0x00000080 [Wed Feb 6 09:40:08 2019][1696508.501894] Call Trace: [Wed Feb 6 09:40:08 2019][1696508.504534] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1696508.509718] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.516607] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1696508.522395] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.528885] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.536467] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696508.541530] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.547808] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1696508.554429] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.560705] mdt_io01_063 S ffff913e1066a080 0 15431 2 0x00000080 [Wed Feb 6 09:40:08 2019][1696508.567996] Call Trace: [Wed Feb 6 09:40:08 2019][1696508.570633] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1696508.575835] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.582726] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1696508.588524] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.595014] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.602591] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696508.607649] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.613925] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1696508.620550] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.626821] mdt_io01_064 S ffff913e1066b0c0 0 15432 2 0x00000080 [Wed Feb 6 09:40:08 2019][1696508.634102] Call Trace: [Wed Feb 6 09:40:08 2019][1696508.636734] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1696508.641922] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.648803] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1696508.654591] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.661078] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.668659] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696508.673720] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.679994] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1696508.686612] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.692888] mdt_io01_065 S ffff913e1066c100 0 15433 2 0x00000080 [Wed Feb 6 09:40:08 2019][1696508.700171] Call Trace: [Wed Feb 6 09:40:08 2019][1696508.702804] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1696508.707990] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.714869] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1696508.720658] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.727141] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.734719] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696508.739778] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.746053] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1696508.752670] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.758942] mdt_io01_066 S ffff913e1066d140 0 15434 2 0x00000080 [Wed Feb 6 09:40:08 2019][1696508.766212] Call Trace: [Wed Feb 6 09:40:08 2019][1696508.768846] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1696508.774037] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.780921] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1696508.786709] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.793193] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.800772] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696508.805837] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.812110] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:08 2019][1696508.818730] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.825009] mdt_io01_067 S ffff913e1066e180 0 15435 2 0x00000080 [Wed Feb 6 09:40:08 2019][1696508.832282] Call Trace: [Wed Feb 6 09:40:08 2019][1696508.834911] [] schedule+0x29/0x70 [Wed Feb 6 09:40:08 2019][1696508.840102] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.846990] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:08 2019][1696508.852792] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.859279] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:08 2019][1696508.866855] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:08 2019][1696508.871914] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:08 2019][1696508.878187] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696508.884809] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696508.891079] mdt_io01_068 S ffff9159ecb2b200 0 15436 2 0x00000080 [Wed Feb 6 09:40:09 2019][1696508.898352] Call Trace: [Wed Feb 6 09:40:09 2019][1696508.900989] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1696508.906188] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696508.913079] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1696508.918876] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696508.925369] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696508.932951] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1696508.938014] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696508.944291] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696508.950918] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696508.957200] mdt_io01_069 S ffff913e0e789040 0 15437 2 0x00000080 [Wed Feb 6 09:40:09 2019][1696508.964494] Call Trace: [Wed Feb 6 09:40:09 2019][1696508.967128] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1696508.972319] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696508.979203] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1696508.984992] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696508.991471] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696508.999048] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1696509.004105] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.010382] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696509.017003] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.023278] mdt_io01_070 S ffff913f51e130c0 0 15438 2 0x00000080 [Wed Feb 6 09:40:09 2019][1696509.030566] Call Trace: [Wed Feb 6 09:40:09 2019][1696509.033202] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1696509.038387] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.045267] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1696509.051057] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.057545] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.065129] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1696509.070197] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.076469] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696509.083090] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.089365] mdt_io01_071 S ffff913e0e7a0000 0 15439 2 0x00000080 [Wed Feb 6 09:40:09 2019][1696509.096651] Call Trace: [Wed Feb 6 09:40:09 2019][1696509.099287] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1696509.104472] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.111359] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1696509.117152] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.123633] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.131208] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1696509.136267] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.142548] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696509.149167] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.155448] mdt_io01_072 S ffff913e0e7a1040 0 15440 2 0x00000080 [Wed Feb 6 09:40:09 2019][1696509.162740] Call Trace: [Wed Feb 6 09:40:09 2019][1696509.165373] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1696509.170564] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.177448] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1696509.183238] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.189721] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.197305] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1696509.202360] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.208636] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696509.215255] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.221524] mdt_io01_073 S ffff911c10071040 0 15441 2 0x00000080 [Wed Feb 6 09:40:09 2019][1696509.228800] Call Trace: [Wed Feb 6 09:40:09 2019][1696509.231433] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1696509.236618] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.243497] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1696509.249287] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.255765] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.263350] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1696509.268410] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.274678] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696509.281297] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.287565] mdt_io02_028 S ffff912427a4b0c0 0 15442 2 0x00000080 [Wed Feb 6 09:40:09 2019][1696509.294850] Call Trace: [Wed Feb 6 09:40:09 2019][1696509.297480] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1696509.302668] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.309551] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1696509.315340] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.321834] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.329416] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1696509.334479] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.340753] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696509.347374] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.353648] mdt_io01_074 S ffff912427a4d140 0 15443 2 0x00000080 [Wed Feb 6 09:40:09 2019][1696509.360920] Call Trace: [Wed Feb 6 09:40:09 2019][1696509.363560] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1696509.368768] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.375657] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1696509.381452] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.387945] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.395528] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1696509.400594] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.406870] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696509.413488] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.419760] mdt_io01_075 S ffff912427a48000 0 15444 2 0x00000080 [Wed Feb 6 09:40:09 2019][1696509.427037] Call Trace: [Wed Feb 6 09:40:09 2019][1696509.429668] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1696509.434857] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.441745] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1696509.447535] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.454016] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.461591] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1696509.466656] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.472939] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696509.479556] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.485833] mdt_io02_029 S ffff912427a4e180 0 15445 2 0x00000080 [Wed Feb 6 09:40:09 2019][1696509.493106] Call Trace: [Wed Feb 6 09:40:09 2019][1696509.495735] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1696509.500929] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.507815] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1696509.513627] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.520137] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.527733] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1696509.532799] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.539076] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696509.545689] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.551960] mdt_io01_076 S ffff912427a4c100 0 15446 2 0x00000080 [Wed Feb 6 09:40:09 2019][1696509.559246] Call Trace: [Wed Feb 6 09:40:09 2019][1696509.561889] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1696509.567088] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.573979] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1696509.579777] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.586269] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.593854] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1696509.598913] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.605187] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696509.611802] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.618077] mdt_io01_077 S ffff912427a4a080 0 15447 2 0x00000080 [Wed Feb 6 09:40:09 2019][1696509.625361] Call Trace: [Wed Feb 6 09:40:09 2019][1696509.627992] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1696509.633198] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.640090] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1696509.645887] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.652386] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.659959] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1696509.665016] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.671299] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696509.677924] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.684201] mdt_io02_030 S ffff912427a49040 0 15448 2 0x00000080 [Wed Feb 6 09:40:09 2019][1696509.691492] Call Trace: [Wed Feb 6 09:40:09 2019][1696509.694120] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1696509.699310] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.706196] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1696509.711984] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.718467] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.726042] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1696509.731101] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.737377] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696509.743998] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.750274] mdt_io02_031 S ffff913e0ccd8000 0 15449 2 0x00000080 [Wed Feb 6 09:40:09 2019][1696509.757550] Call Trace: [Wed Feb 6 09:40:09 2019][1696509.760179] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1696509.765364] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.772250] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1696509.778039] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.784526] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.792104] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1696509.797161] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.803435] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696509.810061] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.816335] mdt_io02_032 S ffff913e0ccd9040 0 15450 2 0x00000080 [Wed Feb 6 09:40:09 2019][1696509.823607] Call Trace: [Wed Feb 6 09:40:09 2019][1696509.826240] [] schedule+0x29/0x70 [Wed Feb 6 09:40:09 2019][1696509.831431] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.838313] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:09 2019][1696509.844105] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.850585] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:09 2019][1696509.858165] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:09 2019][1696509.863219] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:09 2019][1696509.869494] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:09 2019][1696509.876115] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696509.882390] mdt_io02_033 S ffff913e0ccda080 0 15451 2 0x00000080 [Wed Feb 6 09:40:10 2019][1696509.889665] Call Trace: [Wed Feb 6 09:40:10 2019][1696509.892297] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1696509.897487] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696509.904374] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1696509.910165] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696509.916648] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696509.924225] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1696509.929291] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696509.935563] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1696509.942184] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696509.948458] mdt_io02_034 S ffff913e0ccdb0c0 0 15452 2 0x00000080 [Wed Feb 6 09:40:10 2019][1696509.955733] Call Trace: [Wed Feb 6 09:40:10 2019][1696509.958370] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1696509.963564] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696509.970450] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1696509.976243] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696509.982735] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696509.990310] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1696509.995371] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.001648] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1696510.008267] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.014533] mdt_io02_035 S ffff913e0ccdc100 0 15453 2 0x00000080 [Wed Feb 6 09:40:10 2019][1696510.021801] Call Trace: [Wed Feb 6 09:40:10 2019][1696510.024431] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1696510.029620] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.036512] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1696510.042318] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.048804] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.056386] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1696510.061448] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.067732] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1696510.074353] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.080629] mdt_io02_036 S ffff913e0ccdd140 0 15454 2 0x00000080 [Wed Feb 6 09:40:10 2019][1696510.087904] Call Trace: [Wed Feb 6 09:40:10 2019][1696510.090540] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1696510.095732] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.102622] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1696510.108412] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.114902] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.122476] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1696510.127533] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.133810] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1696510.140430] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.146704] mdt_io02_037 S ffff913e0ccde180 0 15455 2 0x00000080 [Wed Feb 6 09:40:10 2019][1696510.153990] Call Trace: [Wed Feb 6 09:40:10 2019][1696510.156620] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1696510.161810] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.168696] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1696510.174485] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.180971] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.188544] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1696510.193604] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.199880] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1696510.206502] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.212777] mdt_io02_038 S ffff9149fedc0000 0 15456 2 0x00000080 [Wed Feb 6 09:40:10 2019][1696510.220049] Call Trace: [Wed Feb 6 09:40:10 2019][1696510.222681] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1696510.227880] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.234772] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1696510.240563] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.247048] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.254623] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1696510.259686] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.265967] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1696510.272582] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.278860] mdt_io02_039 S ffff9149fedc1040 0 15457 2 0x00000080 [Wed Feb 6 09:40:10 2019][1696510.286134] Call Trace: [Wed Feb 6 09:40:10 2019][1696510.288768] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1696510.293955] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.300839] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1696510.306630] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.313111] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.320687] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1696510.325749] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.332026] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1696510.338655] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.344930] mdt_io00_030 S ffff9149fedc2080 0 15458 2 0x00000080 [Wed Feb 6 09:40:10 2019][1696510.352205] Call Trace: [Wed Feb 6 09:40:10 2019][1696510.354846] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1696510.360040] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.366930] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1696510.372739] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.379248] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.386830] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1696510.391896] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.398172] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1696510.404803] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.411080] mdt_io01_078 S ffff9149fedc30c0 0 15459 2 0x00000080 [Wed Feb 6 09:40:10 2019][1696510.418368] Call Trace: [Wed Feb 6 09:40:10 2019][1696510.421005] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1696510.426195] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.433079] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1696510.438875] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.445359] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.452935] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1696510.457994] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.464268] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1696510.470888] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.477165] mdt_io01_079 S ffff9149fedc4100 0 15460 2 0x00000080 [Wed Feb 6 09:40:10 2019][1696510.484444] Call Trace: [Wed Feb 6 09:40:10 2019][1696510.487081] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1696510.492269] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.499155] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1696510.504943] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.511429] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.519007] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1696510.524069] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.530347] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1696510.536965] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.543243] mdt_io01_080 S ffff9149fedc5140 0 15461 2 0x00000080 [Wed Feb 6 09:40:10 2019][1696510.550521] Call Trace: [Wed Feb 6 09:40:10 2019][1696510.553163] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1696510.558358] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.565242] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1696510.571042] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.577525] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.585102] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1696510.590167] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.596452] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1696510.603070] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.609343] mdt_io00_031 S ffff9149fedc6180 0 15462 2 0x00000080 [Wed Feb 6 09:40:10 2019][1696510.616617] Call Trace: [Wed Feb 6 09:40:10 2019][1696510.619250] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1696510.624442] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.631325] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1696510.637121] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.643608] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.651180] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1696510.656241] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.662520] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1696510.669134] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.675407] mdt_io00_032 S ffff913e0fef0000 0 15463 2 0x00000080 [Wed Feb 6 09:40:10 2019][1696510.682695] Call Trace: [Wed Feb 6 09:40:10 2019][1696510.685326] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1696510.690519] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.697412] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1696510.703207] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.709697] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.717279] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1696510.722334] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.728611] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1696510.735231] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.741506] mdt_io00_033 S ffff913e0fef1040 0 15464 2 0x00000080 [Wed Feb 6 09:40:10 2019][1696510.748781] Call Trace: [Wed Feb 6 09:40:10 2019][1696510.751413] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1696510.756601] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.763486] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1696510.769275] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.775761] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.783343] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1696510.788401] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.794679] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:10 2019][1696510.801301] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.807575] mdt_io00_034 S ffff913e0fef2080 0 15465 2 0x00000080 [Wed Feb 6 09:40:10 2019][1696510.814849] Call Trace: [Wed Feb 6 09:40:10 2019][1696510.817479] [] schedule+0x29/0x70 [Wed Feb 6 09:40:10 2019][1696510.822666] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.829545] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:10 2019][1696510.835335] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.841816] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:10 2019][1696510.849396] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:10 2019][1696510.854453] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:10 2019][1696510.860731] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1696510.867352] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696510.873620] mdt_io00_035 S ffff913e0fef30c0 0 15466 2 0x00000080 [Wed Feb 6 09:40:11 2019][1696510.880891] Call Trace: [Wed Feb 6 09:40:11 2019][1696510.883520] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1696510.888709] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696510.895597] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1696510.901383] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696510.907868] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696510.915443] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1696510.920501] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696510.926782] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1696510.933397] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696510.939672] mdt_io00_036 S ffff913e0fef4100 0 15467 2 0x00000080 [Wed Feb 6 09:40:11 2019][1696510.946949] Call Trace: [Wed Feb 6 09:40:11 2019][1696510.949579] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1696510.954769] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696510.961653] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1696510.967451] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696510.973959] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696510.981549] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1696510.986610] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696510.992879] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1696510.999503] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.005780] mdt_io00_037 S ffff913e0fef5140 0 15468 2 0x00000080 [Wed Feb 6 09:40:11 2019][1696511.013052] Call Trace: [Wed Feb 6 09:40:11 2019][1696511.015682] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1696511.020880] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.027760] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1696511.033550] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.040040] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.047613] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1696511.052672] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.058953] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1696511.065570] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.071846] mdt02_047 S ffff913ccad98000 0 15509 2 0x00000080 [Wed Feb 6 09:40:11 2019][1696511.079123] Call Trace: [Wed Feb 6 09:40:11 2019][1696511.081763] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1696511.086953] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.093842] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1696511.099640] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.106126] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.113703] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1696511.118769] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.125046] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1696511.131674] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.137948] mdt02_048 S ffff913ccad99040 0 15510 2 0x00000080 [Wed Feb 6 09:40:11 2019][1696511.145234] Call Trace: [Wed Feb 6 09:40:11 2019][1696511.147870] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1696511.153028] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:40:11 2019][1696511.159043] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:40:11 2019][1696511.165589] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [Wed Feb 6 09:40:11 2019][1696511.172478] [] osp_precreate_reserve+0x2e8/0x800 [osp] [Wed Feb 6 09:40:11 2019][1696511.179443] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1696511.185201] [] osp_declare_create+0x199/0x5b0 [osp] [Wed Feb 6 09:40:11 2019][1696511.191933] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Wed Feb 6 09:40:11 2019][1696511.199264] [] lod_sub_declare_create+0xdf/0x210 [lod] [Wed Feb 6 09:40:11 2019][1696511.206242] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [Wed Feb 6 09:40:11 2019][1696511.213478] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [Wed Feb 6 09:40:11 2019][1696511.220895] [] ? lod_qos_calc_weight+0x80/0x80 [lod] [Wed Feb 6 09:40:11 2019][1696511.227696] [] ? lod_obj_is_ost_use_skip_cb+0x30/0x30 [lod] [Wed Feb 6 09:40:11 2019][1696511.235099] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [Wed Feb 6 09:40:11 2019][1696511.242104] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:11 2019][1696511.249248] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1696511.255008] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [Wed Feb 6 09:40:11 2019][1696511.263030] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [Wed Feb 6 09:40:11 2019][1696511.270440] [] mdd_declare_layout_change+0x62/0x120 [mdd] [Wed Feb 6 09:40:11 2019][1696511.277677] [] mdd_layout_change+0x882/0x1000 [mdd] [Wed Feb 6 09:40:11 2019][1696511.284394] [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:11 2019][1696511.291727] [] mdt_layout_change+0x337/0x430 [mdt] [Wed Feb 6 09:40:11 2019][1696511.298358] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Feb 6 09:40:11 2019][1696511.305032] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.311662] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:11 2019][1696511.318301] [] ? mdt_intent_open+0x350/0x350 [mdt] [Wed Feb 6 09:40:11 2019][1696511.324956] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.331848] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:11 2019][1696511.339164] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:11 2019][1696511.345738] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.353007] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.360718] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.367056] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.374156] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.381914] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:11 2019][1696511.389190] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.397062] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.404036] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:11 2019][1696511.410488] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:11 2019][1696511.416536] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.423020] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.430597] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1696511.435655] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.441929] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1696511.448550] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.454823] mdt02_049 S ffff913ccad9a080 0 15511 2 0x00000080 [Wed Feb 6 09:40:11 2019][1696511.462100] Call Trace: [Wed Feb 6 09:40:11 2019][1696511.464736] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1696511.469931] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.476827] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1696511.482619] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.489109] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.496694] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1696511.501755] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.508030] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1696511.514655] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.520930] ldlm_bl_04 S ffff913ccad9e180 0 15961 2 0x00000080 [Wed Feb 6 09:40:11 2019][1696511.528210] Call Trace: [Wed Feb 6 09:40:11 2019][1696511.530842] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:11 2019][1696511.536710] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:11 2019][1696511.544118] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1696511.549308] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.556369] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1696511.562160] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.569731] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1696511.574792] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.581071] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1696511.587691] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.593963] ldlm_bl_05 S ffff9149ca745140 0 15962 2 0x00000080 [Wed Feb 6 09:40:11 2019][1696511.601259] Call Trace: [Wed Feb 6 09:40:11 2019][1696511.603899] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:11 2019][1696511.609766] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:11 2019][1696511.617170] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1696511.622362] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.629427] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1696511.635222] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.642808] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1696511.647872] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.654152] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1696511.660774] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.667053] ldlm_bl_06 S ffff9159f6bac100 0 15963 2 0x00000080 [Wed Feb 6 09:40:11 2019][1696511.674336] Call Trace: [Wed Feb 6 09:40:11 2019][1696511.676965] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:11 2019][1696511.682838] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:11 2019][1696511.690242] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1696511.695433] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.702494] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1696511.708278] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.715856] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1696511.720919] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.727193] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1696511.733811] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.740085] ldlm_bl_07 S ffff9159ed230000 0 15964 2 0x00000080 [Wed Feb 6 09:40:11 2019][1696511.747361] Call Trace: [Wed Feb 6 09:40:11 2019][1696511.749995] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:11 2019][1696511.755857] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:11 2019][1696511.763264] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1696511.768447] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.775504] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1696511.781287] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.788870] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:11 2019][1696511.793931] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.800204] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:11 2019][1696511.806821] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:11 2019][1696511.813095] ldlm_bl_08 S ffff9129f3048000 0 15965 2 0x00000080 [Wed Feb 6 09:40:11 2019][1696511.820370] Call Trace: [Wed Feb 6 09:40:11 2019][1696511.823010] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:11 2019][1696511.828884] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:11 2019][1696511.836283] [] schedule+0x29/0x70 [Wed Feb 6 09:40:11 2019][1696511.841469] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.848530] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:11 2019][1696511.854317] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:11 2019][1696511.861894] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1696511.866951] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696511.873228] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1696511.879847] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696511.886123] ldlm_bl_09 S ffff911b29b930c0 0 15966 2 0x00000080 [Wed Feb 6 09:40:12 2019][1696511.893399] Call Trace: [Wed Feb 6 09:40:12 2019][1696511.896030] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:12 2019][1696511.901893] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:12 2019][1696511.909297] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1696511.914483] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696511.921539] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1696511.927332] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696511.934910] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1696511.939969] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696511.946249] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1696511.952866] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696511.959146] ldlm_bl_10 S ffff911b29b91040 0 15967 2 0x00000080 [Wed Feb 6 09:40:12 2019][1696511.966426] Call Trace: [Wed Feb 6 09:40:12 2019][1696511.969058] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:12 2019][1696511.974927] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:12 2019][1696511.982347] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1696511.987531] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696511.994588] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1696512.000379] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.007955] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1696512.013015] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.019296] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1696512.025911] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.032188] ldlm_bl_11 R running task 0 15968 2 0x00000080 [Wed Feb 6 09:40:12 2019][1696512.039462] Call Trace: [Wed Feb 6 09:40:12 2019][1696512.042133] [] ? ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.050794] [] ? ldlm_cancel_locks_for_export_cb+0x3f/0x50 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.058899] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [Wed Feb 6 09:40:12 2019][1696512.066505] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.075155] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.083786] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [Wed Feb 6 09:40:12 2019][1696512.091314] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.098921] [] ? ldlm_bl_thread_main+0x4c8/0x700 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.106157] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1696512.111948] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.119525] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1696512.124756] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.131038] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1696512.137834] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.144109] ldlm_bl_12 S ffff9144c9200000 0 15969 2 0x00000080 [Wed Feb 6 09:40:12 2019][1696512.151404] Call Trace: [Wed Feb 6 09:40:12 2019][1696512.154035] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:12 2019][1696512.159905] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:12 2019][1696512.167305] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1696512.172491] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.179544] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1696512.185331] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.192907] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1696512.197966] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.204246] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1696512.210865] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.217144] ldlm_bl_13 S ffff9144c9201040 0 15970 2 0x00000080 [Wed Feb 6 09:40:12 2019][1696512.224412] Call Trace: [Wed Feb 6 09:40:12 2019][1696512.227047] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:12 2019][1696512.232911] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:12 2019][1696512.240319] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1696512.245503] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.252564] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1696512.258360] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.265940] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1696512.271005] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.277281] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1696512.283901] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.290176] ldlm_bl_14 S ffff9144c9202080 0 15971 2 0x00000080 [Wed Feb 6 09:40:12 2019][1696512.297456] Call Trace: [Wed Feb 6 09:40:12 2019][1696512.300090] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:12 2019][1696512.305953] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:12 2019][1696512.313365] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1696512.318552] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.325608] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1696512.331399] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.338969] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1696512.344030] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.350310] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1696512.356928] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.363201] ldlm_bl_15 S ffff9144c92030c0 0 15972 2 0x00000080 [Wed Feb 6 09:40:12 2019][1696512.370493] Call Trace: [Wed Feb 6 09:40:12 2019][1696512.373133] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:12 2019][1696512.379002] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:12 2019][1696512.386403] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1696512.391594] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.398650] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1696512.404438] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.412016] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1696512.417074] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.423349] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1696512.429968] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.436245] ldlm_bl_16 R running task 0 15973 2 0x00000080 [Wed Feb 6 09:40:12 2019][1696512.443521] Call Trace: [Wed Feb 6 09:40:12 2019][1696512.446185] [] ? ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.454836] [] ? ldlm_cancel_locks_for_export_cb+0x3f/0x50 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.462945] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [Wed Feb 6 09:40:12 2019][1696512.470551] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.479198] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.487827] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [Wed Feb 6 09:40:12 2019][1696512.495347] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.502951] [] ? ldlm_bl_thread_main+0x4c8/0x700 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.510186] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1696512.515976] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.523559] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1696512.528792] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.535070] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1696512.541862] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.548139] ldlm_bl_17 S ffff9144c9205140 0 15974 2 0x00000080 [Wed Feb 6 09:40:12 2019][1696512.555419] Call Trace: [Wed Feb 6 09:40:12 2019][1696512.558050] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:12 2019][1696512.563920] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:12 2019][1696512.571324] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1696512.576518] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.583586] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1696512.589378] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.596956] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1696512.602018] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.608298] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1696512.614923] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.621201] ldlm_bl_18 S ffff9144c9206180 0 15975 2 0x00000080 [Wed Feb 6 09:40:12 2019][1696512.628480] Call Trace: [Wed Feb 6 09:40:12 2019][1696512.631116] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:12 2019][1696512.636978] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:12 2019][1696512.644384] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1696512.649567] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.656626] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1696512.662420] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.669999] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1696512.675063] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.681340] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1696512.687957] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.694233] ldlm_bl_19 S ffff9149a0fc9040 0 15982 2 0x00000080 [Wed Feb 6 09:40:12 2019][1696512.701526] Call Trace: [Wed Feb 6 09:40:12 2019][1696512.704157] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:12 2019][1696512.710020] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:12 2019][1696512.717426] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1696512.722610] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.729666] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1696512.735452] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.743029] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1696512.748087] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.754366] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1696512.760985] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.767259] ldlm_bl_20 S ffff9149a0fca080 0 15983 2 0x00000080 [Wed Feb 6 09:40:12 2019][1696512.774543] Call Trace: [Wed Feb 6 09:40:12 2019][1696512.777174] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:12 2019][1696512.783038] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:12 2019][1696512.790437] [] schedule+0x29/0x70 [Wed Feb 6 09:40:12 2019][1696512.795622] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.802675] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:12 2019][1696512.808461] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:12 2019][1696512.816038] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:12 2019][1696512.821097] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.827375] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:12 2019][1696512.834001] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:12 2019][1696512.840277] ldlm_bl_21 S ffff9149a0fcb0c0 0 15984 2 0x00000080 [Wed Feb 6 09:40:12 2019][1696512.847553] Call Trace: [Wed Feb 6 09:40:12 2019][1696512.850184] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:12 2019][1696512.856052] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:12 2019][1696512.863461] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1696512.868653] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696512.875716] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1696512.881510] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696512.889088] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1696512.894147] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696512.900419] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1696512.907037] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696512.913313] ldlm_bl_22 S ffff9149a0fcc100 0 15985 2 0x00000080 [Wed Feb 6 09:40:13 2019][1696512.920598] Call Trace: [Wed Feb 6 09:40:13 2019][1696512.923229] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:13 2019][1696512.929094] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:13 2019][1696512.936500] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1696512.941689] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696512.948753] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1696512.954540] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696512.962120] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1696512.967183] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696512.973460] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1696512.980085] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696512.986361] ldlm_bl_23 S ffff9149a0fcd140 0 15986 2 0x00000080 [Wed Feb 6 09:40:13 2019][1696512.993641] Call Trace: [Wed Feb 6 09:40:13 2019][1696512.996281] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:13 2019][1696513.002157] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:13 2019][1696513.009576] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1696513.014776] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.021849] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1696513.027645] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.035225] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1696513.040285] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.046567] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1696513.053187] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.059464] ldlm_bl_24 S ffff9149a0fce180 0 15987 2 0x00000080 [Wed Feb 6 09:40:13 2019][1696513.066739] Call Trace: [Wed Feb 6 09:40:13 2019][1696513.069371] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:13 2019][1696513.075239] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:13 2019][1696513.082640] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1696513.087826] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.094886] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1696513.100675] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.108252] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1696513.113309] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.119586] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1696513.126208] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.132482] ldlm_bl_25 R running task 0 15988 2 0x00000080 [Wed Feb 6 09:40:13 2019][1696513.139770] Call Trace: [Wed Feb 6 09:40:13 2019][1696513.142440] [] ? ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.151093] [] ? ldlm_cancel_locks_for_export_cb+0x3f/0x50 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.159200] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [Wed Feb 6 09:40:13 2019][1696513.166802] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.175447] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.184068] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [Wed Feb 6 09:40:13 2019][1696513.191589] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.199197] [] ? ldlm_bl_thread_main+0x4c8/0x700 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.206428] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1696513.212223] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.219809] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1696513.225043] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.231325] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1696513.238116] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.244392] ldlm_bl_26 S ffff9141bfb81040 0 15989 2 0x00000080 [Wed Feb 6 09:40:13 2019][1696513.251683] Call Trace: [Wed Feb 6 09:40:13 2019][1696513.254320] [] ? call_rcu_sched+0x1d/0x20 [Wed Feb 6 09:40:13 2019][1696513.260194] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Wed Feb 6 09:40:13 2019][1696513.267592] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1696513.272781] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.279861] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1696513.285664] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.293249] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1696513.298304] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.304583] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1696513.311200] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.317476] kworker/u257:2 S ffff9129fb99d140 0 16011 2 0x00000080 [Wed Feb 6 09:40:13 2019][1696513.324770] Call Trace: [Wed Feb 6 09:40:13 2019][1696513.327404] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:13 2019][1696513.333591] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1696513.338737] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:13 2019][1696513.344492] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:13 2019][1696513.351199] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1696513.356260] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.362537] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1696513.369156] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.375432] kworker/28:2 S ffff9129fb99e180 0 16073 2 0x00000080 [Wed Feb 6 09:40:13 2019][1696513.382725] Call Trace: [Wed Feb 6 09:40:13 2019][1696513.385358] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:13 2019][1696513.391550] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1696513.396694] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:13 2019][1696513.402447] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:13 2019][1696513.409161] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1696513.414219] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.420497] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1696513.427111] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.433380] kworker/u264:1 S ffff91462c092080 0 16127 2 0x00000080 [Wed Feb 6 09:40:13 2019][1696513.440660] Call Trace: [Wed Feb 6 09:40:13 2019][1696513.443318] [] ? ib_cq_poll_work+0x48/0x70 [ib_core] [Wed Feb 6 09:40:13 2019][1696513.450105] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:13 2019][1696513.456297] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1696513.461445] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:13 2019][1696513.467202] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:13 2019][1696513.473906] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1696513.478966] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.485246] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1696513.491867] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.498141] kworker/38:1 S ffff9149f3e99040 0 16288 2 0x00000080 [Wed Feb 6 09:40:13 2019][1696513.505442] Call Trace: [Wed Feb 6 09:40:13 2019][1696513.508073] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:13 2019][1696513.514259] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1696513.519409] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:13 2019][1696513.525164] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:13 2019][1696513.531871] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1696513.536936] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.543209] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1696513.549830] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.556106] mdt_fld_0003 S ffff911b297c0000 0 16814 2 0x00000080 [Wed Feb 6 09:40:13 2019][1696513.563399] Call Trace: [Wed Feb 6 09:40:13 2019][1696513.566040] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1696513.571234] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.578123] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1696513.583923] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.590416] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.597997] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1696513.603068] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.609354] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1696513.615978] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.622252] mdt_rdpg00_008 S ffff9159ac47d140 0 17028 2 0x00000080 [Wed Feb 6 09:40:13 2019][1696513.629545] Call Trace: [Wed Feb 6 09:40:13 2019][1696513.632179] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1696513.637372] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.644263] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1696513.650070] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.656561] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.664143] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1696513.669210] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.675481] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1696513.682100] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.688378] mdt_rdpg00_009 S ffff914aa9f9a080 0 17029 2 0x00000080 [Wed Feb 6 09:40:13 2019][1696513.695661] Call Trace: [Wed Feb 6 09:40:13 2019][1696513.698296] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1696513.703481] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.710367] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1696513.716161] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.722646] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.730225] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1696513.735283] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.741561] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1696513.748179] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.754457] mdt_rdpg00_010 S ffff9159ff3a6180 0 17030 2 0x00000080 [Wed Feb 6 09:40:13 2019][1696513.761735] Call Trace: [Wed Feb 6 09:40:13 2019][1696513.764370] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1696513.769554] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.776436] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:13 2019][1696513.782224] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.788703] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.796286] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:13 2019][1696513.801346] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.807616] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:13 2019][1696513.814236] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:13 2019][1696513.820512] mdt03_015 S ffff9159a7202080 0 17050 2 0x00000080 [Wed Feb 6 09:40:13 2019][1696513.827787] Call Trace: [Wed Feb 6 09:40:13 2019][1696513.830456] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.838202] [] schedule+0x29/0x70 [Wed Feb 6 09:40:13 2019][1696513.843381] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.850475] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:13 2019][1696513.857557] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:13 2019][1696513.864696] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1696513.870486] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696513.877842] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696513.885886] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696513.893652] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Feb 6 09:40:14 2019][1696513.900575] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696513.908358] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696513.916384] [] mdt_reint_unlink+0x331/0x14a0 [mdt] [Wed Feb 6 09:40:14 2019][1696513.923024] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 09:40:14 2019][1696513.929222] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 09:40:14 2019][1696513.935943] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Wed Feb 6 09:40:14 2019][1696513.942923] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 09:40:14 2019][1696513.948830] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696513.955933] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696513.963694] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:14 2019][1696513.970959] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696513.978839] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696513.985808] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:14 2019][1696513.992257] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:14 2019][1696513.998312] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.004802] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.012375] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1696514.017433] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.023708] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1696514.030327] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.036609] mdt03_016 S ffff91599ff9e180 0 17052 2 0x00000080 [Wed Feb 6 09:40:14 2019][1696514.043893] Call Trace: [Wed Feb 6 09:40:14 2019][1696514.046531] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1696514.051721] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.058604] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1696514.064396] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.070883] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.078459] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1696514.083516] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.089799] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1696514.096421] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.102702] mdt03_017 S ffff91599ff9a080 0 17053 2 0x00000080 [Wed Feb 6 09:40:14 2019][1696514.109993] Call Trace: [Wed Feb 6 09:40:14 2019][1696514.112636] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1696514.117836] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.124733] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1696514.130519] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.137002] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.144581] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1696514.149635] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.155918] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1696514.162545] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.168816] mdt_io03_015 S ffff912da3d26180 0 17097 2 0x00000080 [Wed Feb 6 09:40:14 2019][1696514.176101] Call Trace: [Wed Feb 6 09:40:14 2019][1696514.178733] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1696514.183924] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.190810] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1696514.196603] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.203092] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.210677] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1696514.215739] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.222021] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1696514.228642] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.234909] mdt_io03_016 S ffff9139f6b92080 0 17102 2 0x00000080 [Wed Feb 6 09:40:14 2019][1696514.242193] Call Trace: [Wed Feb 6 09:40:14 2019][1696514.244825] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1696514.250015] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.256906] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1696514.262705] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.269194] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.276779] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1696514.281844] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.288116] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1696514.294740] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.301016] mdt_io03_017 S ffff9139f6b930c0 0 17103 2 0x00000080 [Wed Feb 6 09:40:14 2019][1696514.308297] Call Trace: [Wed Feb 6 09:40:14 2019][1696514.310926] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1696514.316113] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.323000] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1696514.328797] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.335288] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.342863] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1696514.347931] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.354202] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1696514.360844] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.367117] mdt_io03_018 S ffff9139f6b95140 0 17104 2 0x00000080 [Wed Feb 6 09:40:14 2019][1696514.374391] Call Trace: [Wed Feb 6 09:40:14 2019][1696514.377023] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1696514.382213] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.389100] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1696514.394898] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.401379] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.408967] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1696514.414032] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.420307] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1696514.426926] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.433199] mdt_io03_019 S ffff9159fa255140 0 17105 2 0x00000080 [Wed Feb 6 09:40:14 2019][1696514.440475] Call Trace: [Wed Feb 6 09:40:14 2019][1696514.443109] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1696514.448297] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.455188] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1696514.460986] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.467475] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.475064] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1696514.480127] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.486402] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1696514.493016] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.499289] mdt03_018 S ffff9139f6b94100 0 17203 2 0x00000080 [Wed Feb 6 09:40:14 2019][1696514.506570] Call Trace: [Wed Feb 6 09:40:14 2019][1696514.509204] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1696514.514392] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.521276] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1696514.527066] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.533550] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.541127] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1696514.546191] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.552469] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1696514.559085] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.565363] mdt03_019 S ffff9139f6b96180 0 17204 2 0x00000080 [Wed Feb 6 09:40:14 2019][1696514.572645] Call Trace: [Wed Feb 6 09:40:14 2019][1696514.575284] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:40:14 2019][1696514.582203] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.589958] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1696514.595140] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.602231] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.609318] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:14 2019][1696514.616472] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1696514.622276] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.629643] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.637674] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:14 2019][1696514.644390] [] mdt_object_local_lock+0x438/0xb20 [mdt] [Wed Feb 6 09:40:14 2019][1696514.651365] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:14 2019][1696514.658106] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.666129] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:14 2019][1696514.673286] [] mdt_object_lock+0x20/0x30 [mdt] [Wed Feb 6 09:40:14 2019][1696514.679570] [] mdt_create+0x569/0x1090 [mdt] [Wed Feb 6 09:40:14 2019][1696514.685710] [] ? lprocfs_job_stats_log+0xd1/0x640 [obdclass] [Wed Feb 6 09:40:14 2019][1696514.693215] [] mdt_reint_create+0x16b/0x360 [mdt] [Wed Feb 6 09:40:14 2019][1696514.699765] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Feb 6 09:40:14 2019][1696514.705970] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Feb 6 09:40:14 2019][1696514.712686] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Wed Feb 6 09:40:14 2019][1696514.719658] [] mdt_reint+0x67/0x140 [mdt] [Wed Feb 6 09:40:14 2019][1696514.725558] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.732663] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.740424] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:14 2019][1696514.747695] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.755580] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.762554] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:14 2019][1696514.768998] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:14 2019][1696514.775064] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.781558] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.789139] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1696514.794212] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.800489] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1696514.807119] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.813395] mdt03_020 S ffff9159fa2530c0 0 17205 2 0x00000080 [Wed Feb 6 09:40:14 2019][1696514.820679] Call Trace: [Wed Feb 6 09:40:14 2019][1696514.823322] [] schedule+0x29/0x70 [Wed Feb 6 09:40:14 2019][1696514.828522] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.835413] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:14 2019][1696514.841213] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.847714] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:14 2019][1696514.855301] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:14 2019][1696514.860360] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:14 2019][1696514.866636] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:14 2019][1696514.873256] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696514.879530] mdt03_021 S ffff9159fa252080 0 17206 2 0x00000080 [Wed Feb 6 09:40:15 2019][1696514.886821] Call Trace: [Wed Feb 6 09:40:15 2019][1696514.889453] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1696514.894645] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696514.901533] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1696514.907331] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696514.913816] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696514.921394] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1696514.926452] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696514.932729] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1696514.939345] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696514.945623] mdt03_022 S ffff9159fa254100 0 17207 2 0x00000080 [Wed Feb 6 09:40:15 2019][1696514.952897] Call Trace: [Wed Feb 6 09:40:15 2019][1696514.955532] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1696514.960719] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696514.967608] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1696514.973407] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696514.979896] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696514.987473] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1696514.992538] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696514.998811] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1696515.005433] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.011711] mdt_io03_020 S ffff9159fa251040 0 17208 2 0x00000080 [Wed Feb 6 09:40:15 2019][1696515.018993] Call Trace: [Wed Feb 6 09:40:15 2019][1696515.021626] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1696515.026816] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.033703] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1696515.039501] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.046009] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.053592] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1696515.058657] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.064938] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1696515.071565] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.077850] mdt_io03_021 S ffff9159fa250000 0 17209 2 0x00000080 [Wed Feb 6 09:40:15 2019][1696515.085145] Call Trace: [Wed Feb 6 09:40:15 2019][1696515.087781] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1696515.092981] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.099865] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1696515.105661] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.112145] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.119729] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1696515.124796] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.131076] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1696515.137696] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.143971] mdt03_023 S ffff9159fa256180 0 17218 2 0x00000080 [Wed Feb 6 09:40:15 2019][1696515.151255] Call Trace: [Wed Feb 6 09:40:15 2019][1696515.153888] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1696515.159076] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.165964] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1696515.171752] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.178232] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.185808] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1696515.190871] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.197143] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1696515.203764] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.210039] mdt03_024 S ffff914aa95b8000 0 17219 2 0x00000080 [Wed Feb 6 09:40:15 2019][1696515.217322] Call Trace: [Wed Feb 6 09:40:15 2019][1696515.219955] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1696515.225150] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.232030] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1696515.237823] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.244314] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.251896] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1696515.256953] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.263229] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1696515.269847] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.276122] mdt03_025 S ffff9159f6cf2080 0 17220 2 0x00000080 [Wed Feb 6 09:40:15 2019][1696515.283390] Call Trace: [Wed Feb 6 09:40:15 2019][1696515.286023] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1696515.291208] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.298089] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1696515.303883] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.310372] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.317948] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1696515.323012] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.329289] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1696515.335906] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.342182] mdt03_026 S ffff9159f7d12080 0 17221 2 0x00000080 [Wed Feb 6 09:40:15 2019][1696515.349462] Call Trace: [Wed Feb 6 09:40:15 2019][1696515.352103] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1696515.357289] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.364175] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1696515.369964] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.376446] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.384021] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1696515.389079] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.395356] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1696515.401974] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.408266] mdt03_027 S ffff9129fb85d140 0 17222 2 0x00000080 [Wed Feb 6 09:40:15 2019][1696515.415564] Call Trace: [Wed Feb 6 09:40:15 2019][1696515.418206] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1696515.423411] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.430300] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1696515.436097] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.442585] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.450168] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1696515.455223] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.461492] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1696515.468113] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.474388] mdt03_028 S ffff9159f7d10000 0 17223 2 0x00000080 [Wed Feb 6 09:40:15 2019][1696515.481663] Call Trace: [Wed Feb 6 09:40:15 2019][1696515.484293] [] ? call_usermodehelper_exec+0x92/0x1a0 [Wed Feb 6 09:40:15 2019][1696515.491088] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1696515.496240] [] schedule_timeout+0x168/0x2d0 [Wed Feb 6 09:40:15 2019][1696515.502266] [] ? mdt_identity_do_upcall+0x134/0x460 [mdt] [Wed Feb 6 09:40:15 2019][1696515.509500] [] ? __internal_add_timer+0x130/0x130 [Wed Feb 6 09:40:15 2019][1696515.516064] [] upcall_cache_get_entry+0x1da/0x8b0 [obdclass] [Wed Feb 6 09:40:15 2019][1696515.523555] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1696515.529325] [] mdt_identity_get+0x17/0x50 [mdt] [Wed Feb 6 09:40:15 2019][1696515.535696] [] old_init_ucred_common+0x11b/0x2e0 [mdt] [Wed Feb 6 09:40:15 2019][1696515.542675] [] mdt_init_ucred_reint+0x142/0x2e0 [mdt] [Wed Feb 6 09:40:15 2019][1696515.549564] [] mdt_reint_internal+0x273/0xaf0 [mdt] [Wed Feb 6 09:40:15 2019][1696515.556286] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [Wed Feb 6 09:40:15 2019][1696515.563519] [] mdt_intent_open+0x82/0x350 [mdt] [Wed Feb 6 09:40:15 2019][1696515.569906] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Wed Feb 6 09:40:15 2019][1696515.577231] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:15 2019][1696515.583859] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [Wed Feb 6 09:40:15 2019][1696515.591205] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.598093] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:15 2019][1696515.605409] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:15 2019][1696515.611976] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.619246] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.626953] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.633272] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.640395] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.648289] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:15 2019][1696515.655623] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.663494] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.670463] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:15 2019][1696515.676906] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:15 2019][1696515.682967] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.689453] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.697038] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1696515.702106] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.708378] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1696515.714999] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.721277] mdt03_029 S ffff9159f7d130c0 0 17224 2 0x00000080 [Wed Feb 6 09:40:15 2019][1696515.728548] Call Trace: [Wed Feb 6 09:40:15 2019][1696515.731180] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1696515.736374] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.743256] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1696515.749045] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.755525] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.763101] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1696515.768161] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.774441] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1696515.781059] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.787386] mdt_rdpg03_005 S ffff9159fa65b0c0 0 17240 2 0x00000080 [Wed Feb 6 09:40:15 2019][1696515.794660] Call Trace: [Wed Feb 6 09:40:15 2019][1696515.797291] [] schedule+0x29/0x70 [Wed Feb 6 09:40:15 2019][1696515.802475] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.809357] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:15 2019][1696515.815147] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.821648] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:15 2019][1696515.829226] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:15 2019][1696515.834294] [] ? finish_task_switch+0x54/0x1c0 [Wed Feb 6 09:40:15 2019][1696515.840561] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.846833] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:15 2019][1696515.853453] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:15 2019][1696515.859729] mdt_io03_022 S ffff9129c185b0c0 0 17257 2 0x00000080 [Wed Feb 6 09:40:15 2019][1696515.867000] Call Trace: [Wed Feb 6 09:40:15 2019][1696515.869636] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1696515.874821] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696515.881707] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1696515.887497] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696515.893986] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696515.901571] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1696515.906634] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696515.912917] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1696515.919588] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696515.925861] mdt_rdpg03_006 S ffff9159fc2cd140 0 17292 2 0x00000080 [Wed Feb 6 09:40:16 2019][1696515.933139] Call Trace: [Wed Feb 6 09:40:16 2019][1696515.935769] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1696515.940959] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696515.947845] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1696515.953635] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696515.960115] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696515.967693] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1696515.972751] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696515.979032] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1696515.985649] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696515.991924] mdt_out03_003 S ffff9129a524e180 0 17417 2 0x00000080 [Wed Feb 6 09:40:16 2019][1696515.999198] Call Trace: [Wed Feb 6 09:40:16 2019][1696516.001830] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1696516.007014] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.013902] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1696516.019692] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.026174] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.033751] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1696516.038807] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.045095] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1696516.051714] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.057989] mdt_rdpg03_007 S ffff9129a524c100 0 17450 2 0x00000080 [Wed Feb 6 09:40:16 2019][1696516.065266] Call Trace: [Wed Feb 6 09:40:16 2019][1696516.067896] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1696516.073088] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.079976] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1696516.085769] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.092251] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.099827] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1696516.104885] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.111162] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1696516.117787] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.124061] mdt_rdpg00_011 S ffff9129a524a080 0 17451 2 0x00000080 [Wed Feb 6 09:40:16 2019][1696516.131342] Call Trace: [Wed Feb 6 09:40:16 2019][1696516.133974] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1696516.139165] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.146048] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1696516.151838] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.158322] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.165899] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1696516.170962] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.177264] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1696516.183883] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.190160] mdt_rdpg03_008 S ffff9159f662a080 0 17452 2 0x00000080 [Wed Feb 6 09:40:16 2019][1696516.197435] Call Trace: [Wed Feb 6 09:40:16 2019][1696516.200066] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1696516.205253] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.212136] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1696516.217934] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.224420] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.231998] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1696516.237056] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.243331] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1696516.249951] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.256226] mdt_rdpg00_012 S ffff913c8a665140 0 17453 2 0x00000080 [Wed Feb 6 09:40:16 2019][1696516.263501] Call Trace: [Wed Feb 6 09:40:16 2019][1696516.266133] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1696516.271321] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.278199] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1696516.283987] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.290470] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.298050] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1696516.303114] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.309473] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1696516.316100] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.322372] kworker/11:2 S ffff914aa94f1040 0 17937 2 0x00000080 [Wed Feb 6 09:40:16 2019][1696516.329669] Call Trace: [Wed Feb 6 09:40:16 2019][1696516.332309] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:16 2019][1696516.338493] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1696516.343646] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:16 2019][1696516.349404] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:16 2019][1696516.356114] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1696516.361175] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.367453] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1696516.374078] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.380358] kworker/31:0 S ffff9159fbfa5140 0 18035 2 0x00000080 [Wed Feb 6 09:40:16 2019][1696516.387652] Call Trace: [Wed Feb 6 09:40:16 2019][1696516.390296] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1696516.395450] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:16 2019][1696516.401209] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:16 2019][1696516.407919] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1696516.412978] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.419253] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1696516.425873] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.432152] ldlm_cn03_018 S ffff912fe0b62080 0 18063 2 0x00000080 [Wed Feb 6 09:40:16 2019][1696516.439458] Call Trace: [Wed Feb 6 09:40:16 2019][1696516.442086] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1696516.447275] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.454158] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1696516.459963] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.466453] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.474036] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1696516.479095] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.485373] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1696516.491990] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.498257] kworker/12:1 S ffff9128f6368000 0 18456 2 0x00000080 [Wed Feb 6 09:40:16 2019][1696516.505533] Call Trace: [Wed Feb 6 09:40:16 2019][1696516.508164] [] ? process_one_work+0x336/0x440 [Wed Feb 6 09:40:16 2019][1696516.514354] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1696516.519500] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:16 2019][1696516.525255] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:16 2019][1696516.531963] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1696516.537029] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.543299] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1696516.549925] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.556198] kworker/u257:0 S ffff9129c674c100 0 22744 2 0x00000080 [Wed Feb 6 09:40:16 2019][1696516.563485] Call Trace: [Wed Feb 6 09:40:16 2019][1696516.566120] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:16 2019][1696516.572325] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1696516.577476] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:16 2019][1696516.583229] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:16 2019][1696516.589936] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:16 2019][1696516.594997] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.601277] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:16 2019][1696516.607908] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:16 2019][1696516.614182] mdt03_030 S ffff911b2955a080 0 23853 2 0x00000080 [Wed Feb 6 09:40:16 2019][1696516.621454] Call Trace: [Wed Feb 6 09:40:16 2019][1696516.624095] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Wed Feb 6 09:40:16 2019][1696516.631013] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.638767] [] schedule+0x29/0x70 [Wed Feb 6 09:40:16 2019][1696516.643955] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.651051] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.658130] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Wed Feb 6 09:40:16 2019][1696516.665274] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:16 2019][1696516.671062] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.678383] [] ? __osd_oi_lookup+0x16c/0x3c0 [osd_ldiskfs] [Wed Feb 6 09:40:16 2019][1696516.685733] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.693753] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:16 2019][1696516.700482] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:40:16 2019][1696516.707474] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Wed Feb 6 09:40:16 2019][1696516.714216] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.722253] [] ? htable_lookup+0x8b/0x130 [obdclass] [Wed Feb 6 09:40:16 2019][1696516.729070] [] ? lu_object_find_at+0x1b3/0x280 [obdclass] [Wed Feb 6 09:40:16 2019][1696516.736307] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:40:16 2019][1696516.743459] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:40:16 2019][1696516.750553] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.757232] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.764343] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.771504] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:40:16 2019][1696516.778217] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:40:16 2019][1696516.784847] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Wed Feb 6 09:40:16 2019][1696516.791676] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.798568] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:40:16 2019][1696516.805893] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:40:16 2019][1696516.812473] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.819739] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:40:16 2019][1696516.827454] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696516.833841] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696516.840943] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696516.848694] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:40:17 2019][1696516.855960] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696516.863829] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696516.870803] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:17 2019][1696516.877248] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:17 2019][1696516.883298] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696516.889785] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696516.897368] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696516.902444] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696516.908721] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696516.915348] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696516.921621] mdt03_031 S ffff913f71bae180 0 24224 2 0x00000080 [Wed Feb 6 09:40:17 2019][1696516.928915] Call Trace: [Wed Feb 6 09:40:17 2019][1696516.931559] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1696516.936752] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696516.943643] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1696516.949440] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696516.955944] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696516.963532] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696516.968622] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696516.974908] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696516.981538] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696516.987820] mdt03_032 S ffff912c13625140 0 24899 2 0x00000080 [Wed Feb 6 09:40:17 2019][1696516.995101] Call Trace: [Wed Feb 6 09:40:17 2019][1696516.997737] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1696517.002934] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.009827] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1696517.015620] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.022120] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.029709] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696517.034765] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.041041] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696517.047665] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.053936] mdt03_033 S ffff9149d2e90000 0 24900 2 0x00000080 [Wed Feb 6 09:40:17 2019][1696517.061213] Call Trace: [Wed Feb 6 09:40:17 2019][1696517.063844] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1696517.069036] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.075924] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1696517.081719] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.088201] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.095808] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696517.100892] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.107169] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696517.113794] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.120068] mdt03_034 S ffff912be6b9d140 0 24923 2 0x00000080 [Wed Feb 6 09:40:17 2019][1696517.127348] Call Trace: [Wed Feb 6 09:40:17 2019][1696517.129979] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1696517.135173] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.142059] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1696517.147851] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.154337] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.161916] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696517.166979] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.173263] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696517.179884] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.186157] kworker/20:0 S ffff91264af7a080 0 25001 2 0x00000080 [Wed Feb 6 09:40:17 2019][1696517.193440] Call Trace: [Wed Feb 6 09:40:17 2019][1696517.196074] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:17 2019][1696517.202264] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1696517.207419] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:17 2019][1696517.213177] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:17 2019][1696517.219940] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696517.225036] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.231304] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696517.237924] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.244200] kworker/21:1 S ffff9139f27ba080 0 25297 2 0x00000080 [Wed Feb 6 09:40:17 2019][1696517.251482] Call Trace: [Wed Feb 6 09:40:17 2019][1696517.254113] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:17 2019][1696517.260293] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1696517.265432] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:17 2019][1696517.271187] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:17 2019][1696517.277896] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696517.282959] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.289235] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696517.295854] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.302129] ldlm_cn03_019 S ffff9139f3766180 0 25318 2 0x00000080 [Wed Feb 6 09:40:17 2019][1696517.309411] Call Trace: [Wed Feb 6 09:40:17 2019][1696517.312047] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1696517.317239] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.324121] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1696517.329914] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.336397] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.343980] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696517.349046] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.355337] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696517.362021] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.368292] ldlm_cn03_020 S ffff9139f37630c0 0 25319 2 0x00000080 [Wed Feb 6 09:40:17 2019][1696517.375597] Call Trace: [Wed Feb 6 09:40:17 2019][1696517.378239] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1696517.383512] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.390536] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1696517.396337] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.402840] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.410417] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696517.415480] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.421755] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696517.428374] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.434645] ldlm_cn03_021 S ffff9141bfb82080 0 26053 2 0x00000080 [Wed Feb 6 09:40:17 2019][1696517.441923] Call Trace: [Wed Feb 6 09:40:17 2019][1696517.444554] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1696517.449745] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.456634] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1696517.462430] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.468922] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.476503] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696517.481563] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.487841] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696517.494460] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.500731] kworker/25:1 S ffff9139f65e4100 0 26159 2 0x00000080 [Wed Feb 6 09:40:17 2019][1696517.508026] Call Trace: [Wed Feb 6 09:40:17 2019][1696517.510656] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:17 2019][1696517.516846] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1696517.522042] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:17 2019][1696517.527883] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:17 2019][1696517.534653] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696517.539740] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.546098] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696517.552783] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.559132] ldlm_cn03_022 S ffff9129f375d140 0 26176 2 0x00000080 [Wed Feb 6 09:40:17 2019][1696517.566405] Call Trace: [Wed Feb 6 09:40:17 2019][1696517.569043] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1696517.574242] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.581124] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1696517.586936] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.593431] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.601068] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696517.606132] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.612408] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696517.619025] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.625302] kworker/32:2 S ffff9129fb852080 0 26624 2 0x00000080 [Wed Feb 6 09:40:17 2019][1696517.632591] Call Trace: [Wed Feb 6 09:40:17 2019][1696517.635223] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:17 2019][1696517.641415] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1696517.646560] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:17 2019][1696517.652320] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:17 2019][1696517.659023] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696517.664082] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.670353] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696517.676985] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.683260] kworker/24:0 S ffff91267db4e180 0 27009 2 0x00000080 [Wed Feb 6 09:40:17 2019][1696517.690539] Call Trace: [Wed Feb 6 09:40:17 2019][1696517.693175] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:17 2019][1696517.699359] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1696517.704506] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:17 2019][1696517.710259] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:17 2019][1696517.716960] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696517.722025] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.728300] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696517.734925] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.741194] mdt_io03_023 S ffff912da3d21040 0 27457 2 0x00000080 [Wed Feb 6 09:40:17 2019][1696517.748546] Call Trace: [Wed Feb 6 09:40:17 2019][1696517.751233] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1696517.756509] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.763444] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1696517.769237] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.775742] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.783422] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696517.788582] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.795008] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696517.801637] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.807922] mdt_io03_024 S ffff912da3d20000 0 27458 2 0x00000080 [Wed Feb 6 09:40:17 2019][1696517.815197] Call Trace: [Wed Feb 6 09:40:17 2019][1696517.817852] [] schedule+0x29/0x70 [Wed Feb 6 09:40:17 2019][1696517.823043] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.829934] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:17 2019][1696517.835757] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.842284] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:17 2019][1696517.849918] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:17 2019][1696517.854978] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:17 2019][1696517.861248] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:17 2019][1696517.867869] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696517.874193] mdt_io03_025 S ffff915964bd30c0 0 27459 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696517.881473] Call Trace: [Wed Feb 6 09:40:18 2019][1696517.884102] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1696517.889293] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696517.896179] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1696517.902011] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696517.908527] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696517.916109] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1696517.921173] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696517.927464] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1696517.934118] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696517.940420] mdt_io03_026 S ffff91218e9d6180 0 27460 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696517.947698] Call Trace: [Wed Feb 6 09:40:18 2019][1696517.950344] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1696517.955531] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696517.962453] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1696517.968261] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696517.974785] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696517.982395] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1696517.987454] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696517.993731] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1696518.000354] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.006627] mdt_io03_027 S ffff91218e9d2080 0 27461 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696518.013902] Call Trace: [Wed Feb 6 09:40:18 2019][1696518.016532] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1696518.021721] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.028608] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1696518.034402] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.040891] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.048477] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1696518.053539] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.059818] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1696518.066438] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.072711] mdt_io03_028 S ffff912da3d22080 0 27462 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696518.079988] Call Trace: [Wed Feb 6 09:40:18 2019][1696518.082618] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1696518.087805] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.094685] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1696518.100476] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.106964] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.114545] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1696518.119611] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.125883] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1696518.132507] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.138784] kworker/0:2 S ffff9128f636a080 0 27671 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696518.146070] Call Trace: [Wed Feb 6 09:40:18 2019][1696518.148704] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:18 2019][1696518.154897] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1696518.160052] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:18 2019][1696518.165807] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:18 2019][1696518.172515] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1696518.177570] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.183852] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1696518.190470] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.196743] mdt_io00_038 S ffff9128f636c100 0 27672 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696518.204019] Call Trace: [Wed Feb 6 09:40:18 2019][1696518.206648] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1696518.211841] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.218726] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1696518.224517] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.231012] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.238594] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1696518.243660] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.249937] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1696518.256559] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.262827] mdt_io00_039 S ffff9128f636e180 0 27673 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696518.270103] Call Trace: [Wed Feb 6 09:40:18 2019][1696518.272733] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1696518.277926] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.284806] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1696518.290600] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.297085] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.304688] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1696518.309753] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.316027] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1696518.322647] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.328933] mdt_io00_040 S ffff9128f636d140 0 27674 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696518.336209] Call Trace: [Wed Feb 6 09:40:18 2019][1696518.338836] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1696518.344027] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.350914] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1696518.356703] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.363181] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.370759] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1696518.375819] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.382095] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1696518.388713] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.394990] mdt_io00_041 S ffff9159eb7530c0 0 27675 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696518.402265] Call Trace: [Wed Feb 6 09:40:18 2019][1696518.404894] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1696518.410084] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.416970] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1696518.422761] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.429242] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.436824] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1696518.441886] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.448161] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1696518.454780] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.461054] mdt_io00_042 S ffff9155c6a69040 0 27676 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696518.468335] Call Trace: [Wed Feb 6 09:40:18 2019][1696518.470979] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1696518.476172] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.483057] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1696518.488849] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.495342] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.502925] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1696518.508000] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.514278] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1696518.520905] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.527179] mdt_io01_081 S ffff9155c6a6a080 0 27677 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696518.534466] Call Trace: [Wed Feb 6 09:40:18 2019][1696518.537107] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1696518.542305] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.549223] [] ? ptlrpc_server_post_idle_rqbds+0x8b/0xf0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.557149] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1696518.562941] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.569429] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.577006] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1696518.582068] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.588343] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1696518.594964] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.601238] mdt_io01_082 S ffff9155c6a6b0c0 0 27678 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696518.608518] Call Trace: [Wed Feb 6 09:40:18 2019][1696518.611149] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1696518.616344] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.623227] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1696518.629029] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.635522] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.643101] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1696518.648157] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.654438] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1696518.661067] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.667343] mdt_io00_043 S ffff9155c6a6c100 0 27679 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696518.674622] Call Trace: [Wed Feb 6 09:40:18 2019][1696518.677253] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1696518.682446] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.689333] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1696518.695132] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.701634] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.709219] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1696518.714276] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.720558] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1696518.727177] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.733449] mdt_io02_040 S ffff9155c6a6d140 0 27680 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696518.740731] Call Trace: [Wed Feb 6 09:40:18 2019][1696518.743369] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1696518.748565] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.755453] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1696518.761251] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.767740] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.775313] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1696518.780370] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.786646] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1696518.793270] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.799543] mdt_io00_044 S ffff9155c6a6e180 0 27681 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696518.806818] Call Trace: [Wed Feb 6 09:40:18 2019][1696518.809449] [] schedule+0x29/0x70 [Wed Feb 6 09:40:18 2019][1696518.814637] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.821525] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:18 2019][1696518.827320] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.833801] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:18 2019][1696518.841380] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:18 2019][1696518.846438] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.852713] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:18 2019][1696518.859335] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:18 2019][1696518.865610] mdt_io01_083 S ffff9159f6628000 0 27682 2 0x00000080 [Wed Feb 6 09:40:18 2019][1696518.872886] Call Trace: [Wed Feb 6 09:40:19 2019][1696518.875517] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696518.880703] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696518.887581] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1696518.893370] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696518.899860] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696518.907444] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1696518.912508] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696518.918781] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1696518.925403] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696518.931678] mdt_io02_041 S ffff9159f0a1c100 0 27683 2 0x00000080 [Wed Feb 6 09:40:19 2019][1696518.938952] Call Trace: [Wed Feb 6 09:40:19 2019][1696518.941588] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696518.946777] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696518.953658] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1696518.959449] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696518.965930] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696518.973507] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1696518.978566] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696518.984845] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1696518.991461] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696518.997746] mdt_io01_084 S ffff9159ac046180 0 27684 2 0x00000080 [Wed Feb 6 09:40:19 2019][1696519.005022] Call Trace: [Wed Feb 6 09:40:19 2019][1696519.007657] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696519.012853] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.019737] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1696519.025526] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.032012] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.039596] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1696519.044668] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.050954] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1696519.057575] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.063851] mdt_io01_085 S ffff914aa958a080 0 27685 2 0x00000080 [Wed Feb 6 09:40:19 2019][1696519.071123] Call Trace: [Wed Feb 6 09:40:19 2019][1696519.073756] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696519.078945] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.085827] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1696519.091623] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.098114] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.105686] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1696519.110744] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.117019] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1696519.123639] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.129920] mdt_io02_042 S ffff9130aa786180 0 27686 2 0x00000080 [Wed Feb 6 09:40:19 2019][1696519.137204] Call Trace: [Wed Feb 6 09:40:19 2019][1696519.139844] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696519.145031] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.151923] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1696519.157715] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.164194] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.171771] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1696519.176829] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.183108] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1696519.189727] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.195999] mdt_io00_045 S ffff9130aa784100 0 27687 2 0x00000080 [Wed Feb 6 09:40:19 2019][1696519.203277] Call Trace: [Wed Feb 6 09:40:19 2019][1696519.205907] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696519.211099] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.217985] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1696519.223779] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.230267] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.237858] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1696519.242918] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.249195] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1696519.255813] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.262090] mdt_io00_046 S ffff9130aa782080 0 27688 2 0x00000080 [Wed Feb 6 09:40:19 2019][1696519.269362] Call Trace: [Wed Feb 6 09:40:19 2019][1696519.271996] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696519.277189] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.284077] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1696519.289866] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.296348] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.303925] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1696519.308985] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.315263] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1696519.321893] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.328164] mdt_io01_086 S ffff9149bbf6c100 0 27689 2 0x00000080 [Wed Feb 6 09:40:19 2019][1696519.335437] Call Trace: [Wed Feb 6 09:40:19 2019][1696519.338069] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696519.343257] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.350153] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1696519.355957] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.362440] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.370022] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1696519.375087] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.381366] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1696519.387991] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.394267] mdt_io02_043 S ffff9139f27be180 0 27690 2 0x00000080 [Wed Feb 6 09:40:19 2019][1696519.401550] Call Trace: [Wed Feb 6 09:40:19 2019][1696519.404181] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696519.409372] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.416257] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1696519.422045] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.428529] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.436104] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1696519.441159] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.447441] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1696519.454062] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.460340] mdt_io01_087 S ffff9139f27b8000 0 27691 2 0x00000080 [Wed Feb 6 09:40:19 2019][1696519.467618] Call Trace: [Wed Feb 6 09:40:19 2019][1696519.470251] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696519.475434] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.482314] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1696519.488109] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.494602] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.502178] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1696519.507236] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.513512] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1696519.520134] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.526407] mdt_io00_047 S ffff913e0e7a2080 0 27692 2 0x00000080 [Wed Feb 6 09:40:19 2019][1696519.533683] Call Trace: [Wed Feb 6 09:40:19 2019][1696519.536315] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696519.541502] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.548381] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1696519.554170] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.560651] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.568228] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1696519.573288] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.579564] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1696519.586184] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.592457] mdt_io00_048 S ffff913e0e7a5140 0 27693 2 0x00000080 [Wed Feb 6 09:40:19 2019][1696519.599732] Call Trace: [Wed Feb 6 09:40:19 2019][1696519.602366] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696519.607553] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.614441] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1696519.620227] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.626717] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.634299] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1696519.639366] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.645640] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1696519.652259] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.658541] mdt_io01_088 S ffff91462c090000 0 27694 2 0x00000080 [Wed Feb 6 09:40:19 2019][1696519.665819] Call Trace: [Wed Feb 6 09:40:19 2019][1696519.668450] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696519.673643] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.680530] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1696519.686335] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.692827] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.700402] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1696519.705465] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.711741] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1696519.718368] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.724636] mdt_io01_089 S ffff91462c091040 0 27695 2 0x00000080 [Wed Feb 6 09:40:19 2019][1696519.731925] Call Trace: [Wed Feb 6 09:40:19 2019][1696519.734561] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696519.739752] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.746640] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1696519.752443] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.758932] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.766511] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1696519.771569] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.777849] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1696519.784468] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.790742] mdt_io01_090 S ffff91462c0930c0 0 27696 2 0x00000080 [Wed Feb 6 09:40:19 2019][1696519.798015] Call Trace: [Wed Feb 6 09:40:19 2019][1696519.800650] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696519.805847] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.812736] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:19 2019][1696519.818538] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.825022] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:19 2019][1696519.832607] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:19 2019][1696519.837662] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.843937] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:19 2019][1696519.850559] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:19 2019][1696519.856836] mdt_io01_091 S ffff91462c094100 0 27697 2 0x00000080 [Wed Feb 6 09:40:19 2019][1696519.864127] Call Trace: [Wed Feb 6 09:40:19 2019][1696519.866761] [] schedule+0x29/0x70 [Wed Feb 6 09:40:19 2019][1696519.871956] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696519.878842] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696519.884631] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696519.891112] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696519.898687] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1696519.903751] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696519.910027] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1696519.916647] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696519.922920] mdt_io01_092 S ffff91462c096180 0 27698 2 0x00000080 [Wed Feb 6 09:40:20 2019][1696519.930202] Call Trace: [Wed Feb 6 09:40:20 2019][1696519.932841] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1696519.938036] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696519.944923] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696519.950723] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696519.957214] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696519.964796] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1696519.969862] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696519.976137] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1696519.982756] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696519.989028] mdt_io02_044 S ffff9150817b0000 0 27699 2 0x00000080 [Wed Feb 6 09:40:20 2019][1696519.996305] Call Trace: [Wed Feb 6 09:40:20 2019][1696519.998935] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1696520.004125] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.011012] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696520.016801] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.023284] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.030858] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1696520.035915] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.042192] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1696520.048811] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.055078] mdt_io02_045 S ffff9150817b1040 0 27700 2 0x00000080 [Wed Feb 6 09:40:20 2019][1696520.062354] Call Trace: [Wed Feb 6 09:40:20 2019][1696520.064987] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1696520.070174] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.077061] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696520.082852] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.089333] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.096909] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1696520.101967] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.108242] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1696520.114864] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.121138] mdt_io02_046 S ffff9150817b2080 0 27701 2 0x00000080 [Wed Feb 6 09:40:20 2019][1696520.128413] Call Trace: [Wed Feb 6 09:40:20 2019][1696520.131043] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1696520.136233] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.143120] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696520.148908] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.155392] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.162971] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1696520.168031] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.174302] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1696520.180922] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.187198] mdt_io02_047 S ffff915964bd5140 0 27702 2 0x00000080 [Wed Feb 6 09:40:20 2019][1696520.194473] Call Trace: [Wed Feb 6 09:40:20 2019][1696520.197105] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1696520.202293] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.209187] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696520.214985] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.221470] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.229044] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1696520.234102] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.240382] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1696520.247000] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.253277] mdt_io00_049 S ffff915964bd6180 0 27703 2 0x00000080 [Wed Feb 6 09:40:20 2019][1696520.260548] Call Trace: [Wed Feb 6 09:40:20 2019][1696520.263183] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1696520.268367] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.275251] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696520.281034] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.287518] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.295094] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1696520.300157] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.306433] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1696520.313069] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.319345] mdt_io01_093 S ffff915964bd4100 0 27704 2 0x00000080 [Wed Feb 6 09:40:20 2019][1696520.326627] Call Trace: [Wed Feb 6 09:40:20 2019][1696520.329262] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1696520.334461] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.341347] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696520.347141] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.353624] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.361201] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1696520.366269] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.372557] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1696520.379182] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.385453] mdt_io00_050 S ffff915964bd2080 0 27705 2 0x00000080 [Wed Feb 6 09:40:20 2019][1696520.392727] Call Trace: [Wed Feb 6 09:40:20 2019][1696520.395361] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1696520.400551] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.407436] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696520.413226] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.419703] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.427280] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1696520.432344] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.438620] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1696520.445235] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.451514] mdt_io00_051 S ffff914fc4b88000 0 27706 2 0x00000080 [Wed Feb 6 09:40:20 2019][1696520.458796] Call Trace: [Wed Feb 6 09:40:20 2019][1696520.461429] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1696520.466619] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.473503] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696520.479294] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.485772] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.493347] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1696520.498408] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.504683] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1696520.511312] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.517591] mdt_io01_094 S ffff914fc4b89040 0 27707 2 0x00000080 [Wed Feb 6 09:40:20 2019][1696520.524880] Call Trace: [Wed Feb 6 09:40:20 2019][1696520.527513] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1696520.532700] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.539587] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696520.545377] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.551856] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.559434] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1696520.564492] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.570769] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1696520.577389] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.583665] mdt_io00_052 S ffff914fc4b8a080 0 27708 2 0x00000080 [Wed Feb 6 09:40:20 2019][1696520.590948] Call Trace: [Wed Feb 6 09:40:20 2019][1696520.593578] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1696520.598767] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.605645] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696520.611434] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.617920] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.625497] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1696520.630559] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.636838] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1696520.643471] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.649741] mdt_io00_053 S ffff914fc4b8b0c0 0 27709 2 0x00000080 [Wed Feb 6 09:40:20 2019][1696520.657019] Call Trace: [Wed Feb 6 09:40:20 2019][1696520.659648] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1696520.664840] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.671723] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696520.677517] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.684006] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.691587] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1696520.696644] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.702922] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1696520.709541] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.715815] mdt_io02_048 S ffff914fc4b8c100 0 27710 2 0x00000080 [Wed Feb 6 09:40:20 2019][1696520.723083] Call Trace: [Wed Feb 6 09:40:20 2019][1696520.725719] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1696520.730916] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.737798] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696520.743589] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.750069] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.757650] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1696520.762720] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.769006] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1696520.775627] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.781901] mdt_io02_049 S ffff914fc4b8d140 0 27711 2 0x00000080 [Wed Feb 6 09:40:20 2019][1696520.789186] Call Trace: [Wed Feb 6 09:40:20 2019][1696520.791818] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1696520.797011] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.803889] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696520.809682] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.816164] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.823738] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:20 2019][1696520.828797] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.835072] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:20 2019][1696520.841694] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:20 2019][1696520.847968] mdt_io00_054 S ffff914fc4b8e180 0 27712 2 0x00000080 [Wed Feb 6 09:40:20 2019][1696520.855246] Call Trace: [Wed Feb 6 09:40:20 2019][1696520.857877] [] schedule+0x29/0x70 [Wed Feb 6 09:40:20 2019][1696520.863062] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:20 2019][1696520.869943] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:20 2019][1696520.875737] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696520.882222] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696520.889801] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1696520.894857] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696520.901133] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1696520.907757] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696520.914032] mdt_io00_055 S ffff914da0f90000 0 27713 2 0x00000080 [Wed Feb 6 09:40:21 2019][1696520.921311] Call Trace: [Wed Feb 6 09:40:21 2019][1696520.923947] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1696520.929153] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696520.936039] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1696520.941841] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696520.948330] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696520.955908] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1696520.960971] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696520.967251] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1696520.973881] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696520.980157] mdt_io00_056 S ffff914da0f91040 0 27714 2 0x00000080 [Wed Feb 6 09:40:21 2019][1696520.987432] Call Trace: [Wed Feb 6 09:40:21 2019][1696520.990064] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1696520.995254] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.002139] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1696521.007928] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.014409] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.021986] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1696521.027045] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.033319] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1696521.039942] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.046217] mdt_io02_050 S ffff9149bbf6a080 0 27715 2 0x00000080 [Wed Feb 6 09:40:21 2019][1696521.053490] Call Trace: [Wed Feb 6 09:40:21 2019][1696521.056123] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1696521.061309] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.068189] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1696521.073975] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.080461] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.088044] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1696521.093106] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.099385] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1696521.106008] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.112285] mdt_io02_051 S ffff9149bbf6d140 0 27716 2 0x00000080 [Wed Feb 6 09:40:21 2019][1696521.119570] Call Trace: [Wed Feb 6 09:40:21 2019][1696521.122202] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1696521.127393] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.134278] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1696521.140075] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.146562] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.154145] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1696521.159204] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.165474] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1696521.172096] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.178368] mdt_io01_095 S ffff9149bbf69040 0 27718 2 0x00000080 [Wed Feb 6 09:40:21 2019][1696521.185647] Call Trace: [Wed Feb 6 09:40:21 2019][1696521.188281] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1696521.193467] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.200356] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1696521.206148] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.212629] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.220223] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1696521.225283] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.231558] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1696521.238187] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.244469] mdt_io02_052 S ffff9149bbf6e180 0 27719 2 0x00000080 [Wed Feb 6 09:40:21 2019][1696521.251758] Call Trace: [Wed Feb 6 09:40:21 2019][1696521.254395] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1696521.259590] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.266471] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1696521.272266] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.278754] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.286336] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1696521.291397] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.297671] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1696521.304291] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.310564] mdt_io01_096 S ffff9149bbf68000 0 27720 2 0x00000080 [Wed Feb 6 09:40:21 2019][1696521.317840] Call Trace: [Wed Feb 6 09:40:21 2019][1696521.320476] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1696521.325680] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.332572] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1696521.338368] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.344855] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.352429] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1696521.357486] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.363762] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1696521.370382] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.376661] mdt_io01_097 S ffff9155a3be0000 0 27721 2 0x00000080 [Wed Feb 6 09:40:21 2019][1696521.383954] Call Trace: [Wed Feb 6 09:40:21 2019][1696521.386591] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1696521.391780] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.398670] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1696521.404461] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.410949] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.418529] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1696521.423590] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.429866] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1696521.436486] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.442760] mdt_io02_053 S ffff9155a3be1040 0 27722 2 0x00000080 [Wed Feb 6 09:40:21 2019][1696521.450047] Call Trace: [Wed Feb 6 09:40:21 2019][1696521.452679] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1696521.457871] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.464753] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1696521.470546] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.477032] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.484613] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1696521.489674] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.495958] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1696521.502584] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.508867] mdt_io01_098 S ffff9155a3be2080 0 27723 2 0x00000080 [Wed Feb 6 09:40:21 2019][1696521.516150] Call Trace: [Wed Feb 6 09:40:21 2019][1696521.518787] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1696521.523980] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.530863] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1696521.536652] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.543132] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.550710] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1696521.555768] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.562049] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1696521.568667] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.574943] mdt_io02_054 S ffff9155a3be30c0 0 27724 2 0x00000080 [Wed Feb 6 09:40:21 2019][1696521.582223] Call Trace: [Wed Feb 6 09:40:21 2019][1696521.584854] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1696521.590043] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.596928] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1696521.602720] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.609200] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.616777] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1696521.621840] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.628115] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1696521.634731] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.641019] mdt_io01_099 S ffff9150817b30c0 0 27725 2 0x00000080 [Wed Feb 6 09:40:21 2019][1696521.648291] Call Trace: [Wed Feb 6 09:40:21 2019][1696521.650926] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1696521.656128] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.663012] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1696521.668808] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.675295] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.682874] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1696521.687949] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.694231] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1696521.700860] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.707135] mdt_io02_055 S ffff914da0f92080 0 27726 2 0x00000080 [Wed Feb 6 09:40:21 2019][1696521.714411] Call Trace: [Wed Feb 6 09:40:21 2019][1696521.717045] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1696521.722235] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.729118] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1696521.734907] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.741387] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.748965] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1696521.754023] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.760300] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1696521.766917] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.773189] mdt_io01_100 S ffff914da0f930c0 0 27727 2 0x00000080 [Wed Feb 6 09:40:21 2019][1696521.780466] Call Trace: [Wed Feb 6 09:40:21 2019][1696521.783101] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1696521.788293] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.795176] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1696521.800966] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.807448] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.815028] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:21 2019][1696521.820093] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.826369] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:21 2019][1696521.832987] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:21 2019][1696521.839263] mdt_io00_057 S ffff914da0f94100 0 27728 2 0x00000080 [Wed Feb 6 09:40:21 2019][1696521.846547] Call Trace: [Wed Feb 6 09:40:21 2019][1696521.849176] [] schedule+0x29/0x70 [Wed Feb 6 09:40:21 2019][1696521.854365] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.861247] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:21 2019][1696521.867052] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.873539] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:21 2019][1696521.881121] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1696521.886175] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696521.892457] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1696521.899082] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696521.905357] mdt_io00_058 S ffff914da0f95140 0 27729 2 0x00000080 [Wed Feb 6 09:40:22 2019][1696521.912631] Call Trace: [Wed Feb 6 09:40:22 2019][1696521.915271] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1696521.920472] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696521.927362] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1696521.933159] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696521.939643] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696521.947218] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1696521.952276] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696521.958547] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1696521.965170] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696521.971443] mdt_io00_059 S ffff914da0f96180 0 27730 2 0x00000080 [Wed Feb 6 09:40:22 2019][1696521.978715] Call Trace: [Wed Feb 6 09:40:22 2019][1696521.981352] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1696521.986548] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696521.993449] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1696521.999238] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.005722] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.013297] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1696522.018359] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.024630] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1696522.031250] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.037529] mdt_io00_060 S ffff9155a0e00000 0 27731 2 0x00000080 [Wed Feb 6 09:40:22 2019][1696522.044810] Call Trace: [Wed Feb 6 09:40:22 2019][1696522.047444] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1696522.052638] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.059524] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1696522.065320] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.071808] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.079397] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1696522.084459] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.090732] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1696522.097353] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.103630] mdt_io02_056 S ffff9155a3be4100 0 27732 2 0x00000080 [Wed Feb 6 09:40:22 2019][1696522.110903] Call Trace: [Wed Feb 6 09:40:22 2019][1696522.113538] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1696522.118727] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.125613] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1696522.131411] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.137896] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.145474] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1696522.150533] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.156816] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1696522.163441] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.169717] mdt_io00_061 S ffff9128f63ee180 0 27733 2 0x00000080 [Wed Feb 6 09:40:22 2019][1696522.177006] Call Trace: [Wed Feb 6 09:40:22 2019][1696522.179641] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1696522.184826] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.191718] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1696522.197509] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.204004] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.211577] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1696522.216641] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.222916] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1696522.229544] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.235820] mdt_io00_062 S ffff9128f63ea080 0 27734 2 0x00000080 [Wed Feb 6 09:40:22 2019][1696522.243099] Call Trace: [Wed Feb 6 09:40:22 2019][1696522.245735] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1696522.250922] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.257808] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1696522.263596] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.270082] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.277672] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1696522.282735] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.289007] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1696522.295624] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.301899] mdt_io02_057 S ffff9128f63eb0c0 0 27735 2 0x00000080 [Wed Feb 6 09:40:22 2019][1696522.309192] Call Trace: [Wed Feb 6 09:40:22 2019][1696522.311828] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1696522.317019] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.323904] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1696522.329699] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.336179] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.343758] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1696522.348816] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.355094] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1696522.361725] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.367996] mdt_io00_063 S ffff9149fbd530c0 0 27736 2 0x00000080 [Wed Feb 6 09:40:22 2019][1696522.375278] Call Trace: [Wed Feb 6 09:40:22 2019][1696522.377909] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1696522.383100] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.389985] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1696522.395780] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.402270] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.409848] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1696522.414907] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.421185] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1696522.427803] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.434077] mdt_io02_058 S ffff9149fbd50000 0 27737 2 0x00000080 [Wed Feb 6 09:40:22 2019][1696522.441347] Call Trace: [Wed Feb 6 09:40:22 2019][1696522.443977] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1696522.449163] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.456044] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1696522.461831] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.468317] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.475894] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1696522.480950] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.487224] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1696522.493846] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.500118] mdt_io02_059 S ffff9149fbd54100 0 27738 2 0x00000080 [Wed Feb 6 09:40:22 2019][1696522.507388] Call Trace: [Wed Feb 6 09:40:22 2019][1696522.510018] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1696522.515209] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.522093] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1696522.527886] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.534382] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.541959] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1696522.547016] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.553292] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1696522.559913] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.566191] mdt_io02_060 S ffff9149fbd52080 0 27739 2 0x00000080 [Wed Feb 6 09:40:22 2019][1696522.573464] Call Trace: [Wed Feb 6 09:40:22 2019][1696522.576097] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1696522.581286] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.588177] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1696522.593983] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.600470] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.608054] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1696522.613111] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.619393] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1696522.626019] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.632294] mdt_io02_061 S ffff9149fbd56180 0 27740 2 0x00000080 [Wed Feb 6 09:40:22 2019][1696522.639574] Call Trace: [Wed Feb 6 09:40:22 2019][1696522.642211] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1696522.647398] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.654282] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1696522.660088] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.666573] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.674160] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1696522.679220] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.685497] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1696522.692117] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.698396] mdt_io02_062 S ffff9149fbd55140 0 27741 2 0x00000080 [Wed Feb 6 09:40:22 2019][1696522.705669] Call Trace: [Wed Feb 6 09:40:22 2019][1696522.708303] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1696522.713486] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.720367] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1696522.726159] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.732635] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.740216] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1696522.745272] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.751548] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1696522.758173] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.764441] mdt_io01_101 S ffff9149fbd51040 0 27742 2 0x00000080 [Wed Feb 6 09:40:22 2019][1696522.771719] Call Trace: [Wed Feb 6 09:40:22 2019][1696522.774356] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1696522.779554] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.786438] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1696522.792226] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.798704] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.806282] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:22 2019][1696522.811342] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.817621] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:22 2019][1696522.824238] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:22 2019][1696522.830513] mdt_io02_063 S ffff9129f304a080 0 27743 2 0x00000080 [Wed Feb 6 09:40:22 2019][1696522.837802] Call Trace: [Wed Feb 6 09:40:22 2019][1696522.840442] [] schedule+0x29/0x70 [Wed Feb 6 09:40:22 2019][1696522.845639] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:22 2019][1696522.852525] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:22 2019][1696522.858322] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696522.864811] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696522.872397] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1696522.877457] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696522.883747] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1696522.890367] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696522.896640] mdt_io01_102 S ffff913aa991b0c0 0 27744 2 0x00000080 [Wed Feb 6 09:40:23 2019][1696522.903923] Call Trace: [Wed Feb 6 09:40:23 2019][1696522.906554] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1696522.911745] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696522.918630] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1696522.924421] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696522.930898] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696522.938480] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1696522.943539] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696522.949815] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1696522.956434] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696522.962707] mdt_io01_103 S ffff913aa9918000 0 27745 2 0x00000080 [Wed Feb 6 09:40:23 2019][1696522.969983] Call Trace: [Wed Feb 6 09:40:23 2019][1696522.972613] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1696522.977800] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696522.984678] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1696522.990471] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696522.996959] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.004543] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1696523.009606] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.015880] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1696523.022501] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.028775] mdt_io02_064 S ffff91267db4a080 0 27746 2 0x00000080 [Wed Feb 6 09:40:23 2019][1696523.036052] Call Trace: [Wed Feb 6 09:40:23 2019][1696523.038681] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1696523.043870] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.050759] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1696523.056544] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.063026] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.070604] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1696523.075663] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.081945] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1696523.088581] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.094855] mdt_io02_065 S ffff91267db4c100 0 27747 2 0x00000080 [Wed Feb 6 09:40:23 2019][1696523.102126] Call Trace: [Wed Feb 6 09:40:23 2019][1696523.104759] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1696523.109949] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.116835] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1696523.122624] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.129109] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.136684] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1696523.141750] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.148024] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1696523.154645] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.160922] mdt_io02_066 S ffff9129fb856180 0 27749 2 0x00000080 [Wed Feb 6 09:40:23 2019][1696523.168197] Call Trace: [Wed Feb 6 09:40:23 2019][1696523.170834] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1696523.176024] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.182912] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1696523.188702] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.195188] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.202769] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1696523.207832] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.214115] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1696523.220743] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.227029] mdt_io01_104 S ffff9129fb855140 0 27750 2 0x00000080 [Wed Feb 6 09:40:23 2019][1696523.234317] Call Trace: [Wed Feb 6 09:40:23 2019][1696523.236956] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1696523.242147] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.249035] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1696523.254833] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.261318] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.268896] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1696523.273953] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.280235] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1696523.286858] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.293133] mdt_io01_105 S ffff912a3f25e180 0 27751 2 0x00000080 [Wed Feb 6 09:40:23 2019][1696523.300414] Call Trace: [Wed Feb 6 09:40:23 2019][1696523.303047] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1696523.308240] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.315124] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1696523.320914] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.327397] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.334975] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1696523.340036] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.346316] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1696523.352938] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.359221] mdt_io02_067 S ffff91293f2fc100 0 27752 2 0x00000080 [Wed Feb 6 09:40:23 2019][1696523.366510] Call Trace: [Wed Feb 6 09:40:23 2019][1696523.369142] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1696523.374337] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.381223] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:40:23 2019][1696523.388204] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1696523.394003] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.400494] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.408080] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1696523.413146] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.419428] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1696523.426048] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.432322] mdt_io02_068 S ffff91293f2fd140 0 27753 2 0x00000080 [Wed Feb 6 09:40:23 2019][1696523.439600] Call Trace: [Wed Feb 6 09:40:23 2019][1696523.442236] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1696523.447435] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.454316] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1696523.460113] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.466594] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.474173] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1696523.479235] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.485511] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1696523.492133] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.498407] mdt_io01_106 S ffff9139f66d9040 0 27754 2 0x00000080 [Wed Feb 6 09:40:23 2019][1696523.505682] Call Trace: [Wed Feb 6 09:40:23 2019][1696523.508314] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1696523.513502] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.520389] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1696523.526183] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.532673] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.540253] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1696523.545313] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.551589] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1696523.558208] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.564483] mdt_io01_107 S ffff9139f66de180 0 27755 2 0x00000080 [Wed Feb 6 09:40:23 2019][1696523.571759] Call Trace: [Wed Feb 6 09:40:23 2019][1696523.574390] [] schedule+0x29/0x70 [Wed Feb 6 09:40:23 2019][1696523.579576] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.586456] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:23 2019][1696523.592246] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.598725] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:23 2019][1696523.606304] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:23 2019][1696523.611362] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.617638] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:23 2019][1696523.624259] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:23 2019][1696523.630534] mdt_io02_069 S[1696523.633067] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [migration/3:26] [Wed Feb 6 09:40:23 2019][1696523.633068] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 09:40:23 2019][1696523.633141] CPU: 3 PID: 26 Comm: migration/3 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 09:40:23 2019][1696523.633142] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 09:40:23 2019][1696523.633143] task: ffff914aa9f7b0c0 ti: ffff914aa9f90000 task.ti: ffff914aa9f90000 [Wed Feb 6 09:40:23 2019][1696523.633144] RIP: 0010:[] [] multi_cpu_stop+0x4a/0x110 [Wed Feb 6 09:40:23 2019][1696523.633149] RSP: 0000:ffff914aa9f93d98 EFLAGS: 00000246 [Wed Feb 6 09:40:23 2019][1696523.633150] RAX: 0000000000000001 RBX: ffff914aa95e7c00 RCX: dead000000000200 [Wed Feb 6 09:40:23 2019][1696523.633151] RDX: ffff915a3f415ff0 RSI: 0000000000000282 RDI: ffff9139eef23ab0 [Wed Feb 6 09:40:23 2019][1696523.633152] RBP: ffff914aa9f93dc0 R08: ffff9139eef23a50 R09: 0000000000000001 [Wed Feb 6 09:40:23 2019][1696523.633152] R10: 000000000000b449 R11: 0000000000000001 R12: ffff914aa9f93d20 [Wed Feb 6 09:40:23 2019][1696523.633153] R13: ffff9144c9204100 R14: ffffffffadcd2ea0 R15: ffff914aa9f93d00 [Wed Feb 6 09:40:23 2019][1696523.633154] FS: 00007f78f341b880(0000) GS:ffff915a3f400000(0000) knlGS:0000000000000000 [Wed Feb 6 09:40:23 2019][1696523.633155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 09:40:23 2019][1696523.633156] CR2: 00007f78e33efe98 CR3: 0000004033a9a000 CR4: 00000000003407e0 [Wed Feb 6 09:40:23 2019][1696523.633157] Call Trace: [Wed Feb 6 09:40:23 2019][1696523.633160] [] ? cpu_stop_should_run+0x50/0x50 [Wed Feb 6 09:40:23 2019][1696523.633161] [] cpu_stopper_thread+0x99/0x150 [Wed Feb 6 09:40:23 2019][1696523.633165] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:40:24 2019][1696523.633167] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 09:40:24 2019][1696523.633170] [] smpboot_thread_fn+0x144/0x1a0 [Wed Feb 6 09:40:24 2019][1696523.633171] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696523.633173] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1696523.633175] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696523.633177] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1696523.633179] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696523.633179] Code: 66 90 66 90 49 89 c5 48 8b 47 18 48 85 c0 0f 84 b3 00 00 00 0f a3 18 19 db 85 db 41 0f 95 c6 45 31 ff 31 c0 0f 1f 44 00 00 f3 90 <41> 8b 5c 24 20 39 c3 74 5d 83 fb 02 74 68 83 fb 03 75 05 45 84 [Wed Feb 6 09:40:24 2019] [Wed Feb 6 09:40:24 2019][1696523.931963] ffff9139f66dd140 0 27756 2 0x00000080 [Wed Feb 6 09:40:24 2019][1696523.937951] Call Trace: [Wed Feb 6 09:40:24 2019][1696523.940589] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1696523.945789] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696523.952669] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1696523.958458] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696523.964938] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696523.972514] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1696523.977572] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696523.983851] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1696523.990469] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696523.996744] mdt_io01_108 S ffff9139f1f3b0c0 0 27757 2 0x00000080 [Wed Feb 6 09:40:24 2019][1696524.004026] Call Trace: [Wed Feb 6 09:40:24 2019][1696524.006667] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1696524.011860] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.018746] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1696524.024535] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.031013] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.038595] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1696524.043662] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.049938] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1696524.056572] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.062846] mdt_io02_070 S ffff9139f1f38000 0 27758 2 0x00000080 [Wed Feb 6 09:40:24 2019][1696524.070127] Call Trace: [Wed Feb 6 09:40:24 2019][1696524.072762] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1696524.077955] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.084839] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1696524.090630] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.097115] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.104697] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1696524.109764] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.116039] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1696524.122661] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.128930] mdt_io02_071 S ffff9149f62b0000 0 27759 2 0x00000080 [Wed Feb 6 09:40:24 2019][1696524.136208] Call Trace: [Wed Feb 6 09:40:24 2019][1696524.138842] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1696524.144027] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.150910] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1696524.156704] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.163188] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.170772] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1696524.175840] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.182112] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1696524.188733] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.195013] mdt_io01_109 S ffff9139f1f3a080 0 27760 2 0x00000080 [Wed Feb 6 09:40:24 2019][1696524.202292] Call Trace: [Wed Feb 6 09:40:24 2019][1696524.204927] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1696524.210118] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.217004] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1696524.222806] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.229289] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.236869] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1696524.241931] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.248206] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1696524.254826] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.261102] mdt_io01_110 S ffff9139f1f3d140 0 27761 2 0x00000080 [Wed Feb 6 09:40:24 2019][1696524.268390] Call Trace: [Wed Feb 6 09:40:24 2019][1696524.271028] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1696524.276219] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.283105] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1696524.288905] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.295401] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.302980] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1696524.308044] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.314321] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1696524.320935] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.327209] mdt_io01_111 S ffff9139f6034100 0 27762 2 0x00000080 [Wed Feb 6 09:40:24 2019][1696524.334491] Call Trace: [Wed Feb 6 09:40:24 2019][1696524.337129] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1696524.342322] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.349203] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1696524.354994] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.361473] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.369051] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1696524.374112] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.380395] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1696524.387020] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.393287] mdt_io02_072 S ffff9139f60330c0 0 27763 2 0x00000080 [Wed Feb 6 09:40:24 2019][1696524.400555] Call Trace: [Wed Feb 6 09:40:24 2019][1696524.403191] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1696524.408378] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.415263] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1696524.421053] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.427536] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.435114] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1696524.440181] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.446456] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1696524.453076] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.459350] mdt_io02_073 S ffff9139f6031040 0 27764 2 0x00000080 [Wed Feb 6 09:40:24 2019][1696524.466627] Call Trace: [Wed Feb 6 09:40:24 2019][1696524.469264] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1696524.474450] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.481336] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1696524.487137] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.493622] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.501202] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1696524.506265] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.512537] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1696524.519161] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.525432] mdt_io02_074 S ffff9139f66d8000 0 27765 2 0x00000080 [Wed Feb 6 09:40:24 2019][1696524.532709] Call Trace: [Wed Feb 6 09:40:24 2019][1696524.535340] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1696524.540529] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.547422] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1696524.553228] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.559715] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.567291] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1696524.572347] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.578624] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1696524.585246] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.591521] mdt_io02_075 S ffff9139f66dc100 0 27766 2 0x00000080 [Wed Feb 6 09:40:24 2019][1696524.598793] Call Trace: [Wed Feb 6 09:40:24 2019][1696524.601426] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1696524.606612] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.613490] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1696524.619280] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.625762] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.633337] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1696524.638396] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.644674] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1696524.651299] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.657575] mdt_io02_076 S ffff9139f66db0c0 0 27767 2 0x00000080 [Wed Feb 6 09:40:24 2019][1696524.664854] Call Trace: [Wed Feb 6 09:40:24 2019][1696524.667491] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1696524.672688] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.679569] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1696524.685359] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.691846] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.699429] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1696524.704490] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.710775] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1696524.717405] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.723680] mdt_io02_077 S ffff912aa98bb0c0 0 27768 2 0x00000080 [Wed Feb 6 09:40:24 2019][1696524.730956] Call Trace: [Wed Feb 6 09:40:24 2019][1696524.733591] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1696524.738785] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.745674] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1696524.751472] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.757959] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.765535] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1696524.770593] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.776870] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1696524.783494] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.789763] mdt_io02_078 S ffff9139f83a9040 0 27769 2 0x00000080 [Wed Feb 6 09:40:24 2019][1696524.797040] Call Trace: [Wed Feb 6 09:40:24 2019][1696524.799672] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1696524.804856] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.811738] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:24 2019][1696524.817527] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.824007] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:24 2019][1696524.831585] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:24 2019][1696524.836649] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.842918] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:24 2019][1696524.849540] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:24 2019][1696524.855815] mdt_io02_079 S ffff9139f83aa080 0 27770 2 0x00000080 [Wed Feb 6 09:40:24 2019][1696524.863092] Call Trace: [Wed Feb 6 09:40:24 2019][1696524.865730] [] schedule+0x29/0x70 [Wed Feb 6 09:40:24 2019][1696524.870922] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696524.877806] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1696524.883594] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696524.890074] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696524.897653] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1696524.902711] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696524.908986] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1696524.915607] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696524.921881] mdt_io02_080 S ffff9149ba351040 0 27771 2 0x00000080 [Wed Feb 6 09:40:25 2019][1696524.929149] Call Trace: [Wed Feb 6 09:40:25 2019][1696524.931781] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1696524.936970] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696524.943857] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1696524.949648] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696524.956128] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696524.963700] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1696524.968767] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696524.975046] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1696524.981672] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696524.987946] mdt_io02_081 S ffff9149ba356180 0 27772 2 0x00000080 [Wed Feb 6 09:40:25 2019][1696524.995217] Call Trace: [Wed Feb 6 09:40:25 2019][1696524.997850] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1696525.003040] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.009924] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1696525.015719] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.022206] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.029788] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1696525.034847] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.041125] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1696525.047741] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.054018] mdt_io02_082 S ffff9129c674e180 0 27773 2 0x00000080 [Wed Feb 6 09:40:25 2019][1696525.061293] Call Trace: [Wed Feb 6 09:40:25 2019][1696525.063923] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1696525.069115] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.075997] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1696525.081786] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.088275] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.095847] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1696525.100904] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.107187] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1696525.113803] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.120080] mdt_io02_083 S ffff9149ba350000 0 27774 2 0x00000080 [Wed Feb 6 09:40:25 2019][1696525.127365] Call Trace: [Wed Feb 6 09:40:25 2019][1696525.130001] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1696525.135188] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.142067] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1696525.147854] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.154342] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.161916] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1696525.166977] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.173263] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1696525.179884] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.186157] mdt_io02_084 S ffff9149ba3530c0 0 27775 2 0x00000080 [Wed Feb 6 09:40:25 2019][1696525.193431] Call Trace: [Wed Feb 6 09:40:25 2019][1696525.196067] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1696525.201264] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.208150] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1696525.213944] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.220432] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.228007] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1696525.233068] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.239346] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1696525.245966] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.252239] mdt_io02_085 S ffff9149ba355140 0 27776 2 0x00000080 [Wed Feb 6 09:40:25 2019][1696525.259513] Call Trace: [Wed Feb 6 09:40:25 2019][1696525.262144] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1696525.267338] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.274220] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1696525.280017] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.286505] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.294082] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1696525.299146] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.305424] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1696525.312056] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.318338] mdt_io02_086 S ffff9149f7214100 0 27777 2 0x00000080 [Wed Feb 6 09:40:25 2019][1696525.325608] Call Trace: [Wed Feb 6 09:40:25 2019][1696525.328241] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1696525.333430] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.340313] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1696525.346109] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.352591] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.360170] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1696525.365227] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.371504] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1696525.378124] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.384400] mdt_io02_087 S ffff913f51e16180 0 27778 2 0x00000080 [Wed Feb 6 09:40:25 2019][1696525.391676] Call Trace: [Wed Feb 6 09:40:25 2019][1696525.394307] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1696525.399494] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.406382] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1696525.412169] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.418653] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.426229] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1696525.431286] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.437554] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1696525.444173] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.450449] mdt_io02_088 S ffff913f51e11040 0 27779 2 0x00000080 [Wed Feb 6 09:40:25 2019][1696525.457725] Call Trace: [Wed Feb 6 09:40:25 2019][1696525.460357] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1696525.465544] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.472422] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1696525.478213] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.484694] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.492281] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1696525.497342] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.503615] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1696525.510235] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.516509] mdt_io02_089 S ffff9129c674a080 0 27780 2 0x00000080 [Wed Feb 6 09:40:25 2019][1696525.523784] Call Trace: [Wed Feb 6 09:40:25 2019][1696525.526415] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1696525.531607] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.538492] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1696525.544280] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.550762] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.558337] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1696525.563396] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.569676] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1696525.576296] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.582566] mdt_io02_090 S ffff9129c6748000 0 27781 2 0x00000080 [Wed Feb 6 09:40:25 2019][1696525.589835] Call Trace: [Wed Feb 6 09:40:25 2019][1696525.592465] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1696525.597652] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.604533] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1696525.610321] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.616807] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.624383] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1696525.629451] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.635723] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1696525.642344] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.648619] mdt_io02_091 S ffff9129c6749040 0 27782 2 0x00000080 [Wed Feb 6 09:40:25 2019][1696525.655902] Call Trace: [Wed Feb 6 09:40:25 2019][1696525.658533] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1696525.663720] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.670604] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1696525.676389] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.682870] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.690447] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1696525.695511] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.701782] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1696525.708406] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.714679] mdt_io00_064 S ffff9129c674d140 0 27783 2 0x00000080 [Wed Feb 6 09:40:25 2019][1696525.721953] Call Trace: [Wed Feb 6 09:40:25 2019][1696525.724582] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1696525.729771] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.736651] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:25 2019][1696525.742439] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.748922] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:25 2019][1696525.756509] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1696525.761563] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.767842] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1696525.774462] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.780742] kworker/23:2 S ffff91599d6aa080 0 28109 2 0x00000080 [Wed Feb 6 09:40:25 2019][1696525.788031] Call Trace: [Wed Feb 6 09:40:25 2019][1696525.790663] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:25 2019][1696525.796854] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1696525.801999] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:25 2019][1696525.807753] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:25 2019][1696525.814464] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:25 2019][1696525.819523] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.825799] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:25 2019][1696525.832419] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:25 2019][1696525.838695] kworker/u258:0 S ffff9130e37ee180 0 28453 2 0x00000080 [Wed Feb 6 09:40:25 2019][1696525.845993] Call Trace: [Wed Feb 6 09:40:25 2019][1696525.848631] [] ? __call_usermodehelper+0x4e/0x90 [Wed Feb 6 09:40:25 2019][1696525.855072] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:25 2019][1696525.861261] [] schedule+0x29/0x70 [Wed Feb 6 09:40:25 2019][1696525.866408] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:25 2019][1696525.872164] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:26 2019][1696525.878871] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696525.883931] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696525.890208] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1696525.896833] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696525.903103] ldlm_cn02_024 S ffff9129c3fd0000 0 28508 2 0x00000080 [Wed Feb 6 09:40:26 2019][1696525.910388] Call Trace: [Wed Feb 6 09:40:26 2019][1696525.913018] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1696525.918218] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696525.925104] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1696525.930892] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696525.937372] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696525.944951] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696525.950015] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696525.956286] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1696525.962908] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696525.969184] ldlm_cn02_025 S ffff9149f83fd140 0 28509 2 0x00000080 [Wed Feb 6 09:40:26 2019][1696525.976456] Call Trace: [Wed Feb 6 09:40:26 2019][1696525.979085] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1696525.984277] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696525.991161] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1696525.996953] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.003431] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.011009] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696526.016067] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.022342] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1696526.028965] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.035240] ldlm_cn03_023 S ffff9149f83fe180 0 28510 2 0x00000080 [Wed Feb 6 09:40:26 2019][1696526.042525] Call Trace: [Wed Feb 6 09:40:26 2019][1696526.045164] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1696526.050359] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.057239] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1696526.063030] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.069510] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.077091] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696526.082156] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.088429] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1696526.095050] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.101328] ldlm_cn03_024 S ffff9156a426b0c0 0 28512 2 0x00000080 [Wed Feb 6 09:40:26 2019][1696526.108598] Call Trace: [Wed Feb 6 09:40:26 2019][1696526.111230] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1696526.116417] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.123301] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1696526.129091] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.135582] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.143161] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696526.148224] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.154496] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1696526.161117] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.167394] ldlm_cn03_025 S ffff9149d5fc9040 0 28518 2 0x00000080 [Wed Feb 6 09:40:26 2019][1696526.174676] Call Trace: [Wed Feb 6 09:40:26 2019][1696526.177308] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1696526.182500] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.189383] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1696526.195175] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.201659] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.209248] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696526.214320] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.220594] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1696526.227221] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.233498] ldlm_cn02_026 S ffff9149d5fcc100 0 28520 2 0x00000080 [Wed Feb 6 09:40:26 2019][1696526.240769] Call Trace: [Wed Feb 6 09:40:26 2019][1696526.243403] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1696526.248591] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.255476] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1696526.261267] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.267752] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.275327] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696526.280391] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.286667] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1696526.293291] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.299565] ldlm_cn03_026 S ffff911a29b7b0c0 0 28521 2 0x00000080 [Wed Feb 6 09:40:26 2019][1696526.306839] Call Trace: [Wed Feb 6 09:40:26 2019][1696526.309468] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1696526.314661] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.321549] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1696526.327345] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.333846] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.341418] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696526.346479] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.352752] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1696526.359370] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.365638] ldlm_cn02_027 S ffff913a4cdfd140 0 28523 2 0x00000080 [Wed Feb 6 09:40:26 2019][1696526.372916] Call Trace: [Wed Feb 6 09:40:26 2019][1696526.375554] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1696526.380748] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.387639] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1696526.393433] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.399923] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.407506] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696526.412565] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.418840] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1696526.425459] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.431736] ldlm_cn03_027 S ffff9130e37eb0c0 0 28529 2 0x00000080 [Wed Feb 6 09:40:26 2019][1696526.439019] Call Trace: [Wed Feb 6 09:40:26 2019][1696526.441653] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1696526.446839] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.453731] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1696526.459532] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.466014] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.473591] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696526.478657] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.484933] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1696526.491550] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.497830] ldlm_cn02_028 S ffff9149f9a4e180 0 28532 2 0x00000080 [Wed Feb 6 09:40:26 2019][1696526.505102] Call Trace: [Wed Feb 6 09:40:26 2019][1696526.507732] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1696526.512922] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.519813] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1696526.525620] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.532113] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.539695] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696526.544762] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.551039] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1696526.557665] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.563938] ldlm_cn03_028 S ffff914889118000 0 28536 2 0x00000080 [Wed Feb 6 09:40:26 2019][1696526.571219] Call Trace: [Wed Feb 6 09:40:26 2019][1696526.573853] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1696526.579045] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.585928] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Wed Feb 6 09:40:26 2019][1696526.592901] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1696526.598700] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.605184] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.612763] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696526.617819] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.624097] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1696526.630717] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.636992] kworker/u264:0 S ffff913e0e7a4100 0 28537 2 0x00000080 [Wed Feb 6 09:40:26 2019][1696526.644282] Call Trace: [Wed Feb 6 09:40:26 2019][1696526.646940] [] ? ib_cq_poll_work+0x48/0x70 [ib_core] [Wed Feb 6 09:40:26 2019][1696526.653737] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:26 2019][1696526.659928] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1696526.665074] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:26 2019][1696526.670826] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:26 2019][1696526.677533] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696526.682597] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.688869] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1696526.695491] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.701766] ldlm_cn03_029 S ffff914661ae9040 0 28538 2 0x00000080 [Wed Feb 6 09:40:26 2019][1696526.709052] Call Trace: [Wed Feb 6 09:40:26 2019][1696526.711680] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1696526.716872] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.723756] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1696526.729548] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.736031] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.743604] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696526.748668] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.754941] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1696526.761561] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.767833] ldlm_cn03_030 S ffff9139f2ad9040 0 28542 2 0x00000080 [Wed Feb 6 09:40:26 2019][1696526.775108] Call Trace: [Wed Feb 6 09:40:26 2019][1696526.777739] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1696526.782927] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.789810] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:26 2019][1696526.795600] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.802087] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:26 2019][1696526.809663] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696526.814727] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.821000] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:26 2019][1696526.827620] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:26 2019][1696526.833896] kworker/u260:1 S ffff9156a4268000 0 28593 2 0x00000080 [Wed Feb 6 09:40:26 2019][1696526.841174] Call Trace: [Wed Feb 6 09:40:26 2019][1696526.843807] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:26 2019][1696526.849999] [] schedule+0x29/0x70 [Wed Feb 6 09:40:26 2019][1696526.855151] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:26 2019][1696526.860913] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:26 2019][1696526.867619] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:26 2019][1696526.872687] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696526.878971] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1696526.885595] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696526.891870] sssd_be S ffff9139f8b48000 0 28616 17535 0x00000080 [Wed Feb 6 09:40:27 2019][1696526.899150] Call Trace: [Wed Feb 6 09:40:27 2019][1696526.901785] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:40:27 2019][1696526.908492] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696526.913640] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:40:27 2019][1696526.920782] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:40:27 2019][1696526.926706] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:40:27 2019][1696526.934030] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:27 2019][1696526.940569] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:40:27 2019][1696526.945813] [] ? recalc_sigpending+0x1b/0x70 [Wed Feb 6 09:40:27 2019][1696526.951911] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1696526.957665] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:40:27 2019][1696526.963420] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:27 2019][1696526.969607] kworker/40:2 S ffff911a1a50e180 0 28621 2 0x00000080 [Wed Feb 6 09:40:27 2019][1696526.976895] Call Trace: [Wed Feb 6 09:40:27 2019][1696526.979527] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:27 2019][1696526.985723] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696526.990890] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:27 2019][1696526.996647] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:27 2019][1696527.003362] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1696527.008421] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.014692] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1696527.021314] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.027591] kworker/17:0 S ffff912be6b9c100 0 28831 2 0x00000080 [Wed Feb 6 09:40:27 2019][1696527.034867] Call Trace: [Wed Feb 6 09:40:27 2019][1696527.037494] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:27 2019][1696527.043672] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696527.048815] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:27 2019][1696527.054568] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:27 2019][1696527.061279] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1696527.066342] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.072613] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1696527.079236] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.085511] kworker/11:1 S ffff9159eb755140 0 28842 2 0x00000080 [Wed Feb 6 09:40:27 2019][1696527.092789] Call Trace: [Wed Feb 6 09:40:27 2019][1696527.095422] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:27 2019][1696527.101607] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696527.106755] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:27 2019][1696527.112506] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:27 2019][1696527.119216] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1696527.124275] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.130545] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1696527.137166] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.143443] kworker/9:2 S ffff9139f3760000 0 28872 2 0x00000080 [Wed Feb 6 09:40:27 2019][1696527.150721] Call Trace: [Wed Feb 6 09:40:27 2019][1696527.153356] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:27 2019][1696527.159540] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696527.164683] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:27 2019][1696527.170447] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:27 2019][1696527.177164] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1696527.182222] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.188491] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1696527.195114] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.201386] kworker/15:0 S ffff9159fc2ce180 0 28874 2 0x00000080 [Wed Feb 6 09:40:27 2019][1696527.208675] Call Trace: [Wed Feb 6 09:40:27 2019][1696527.211313] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:27 2019][1696527.217503] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696527.222645] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:27 2019][1696527.228401] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:27 2019][1696527.235107] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1696527.240162] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.246435] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1696527.253074] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.259355] kworker/47:0 S ffff9159fa65d140 0 28887 2 0x00000080 [Wed Feb 6 09:40:27 2019][1696527.266642] Call Trace: [Wed Feb 6 09:40:27 2019][1696527.269274] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:27 2019][1696527.275463] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696527.280610] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:27 2019][1696527.286364] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:27 2019][1696527.293075] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1696527.298138] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.304411] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1696527.311030] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.317309] kworker/7:2 S ffff9159fa658000 0 28894 2 0x00000080 [Wed Feb 6 09:40:27 2019][1696527.324585] Call Trace: [Wed Feb 6 09:40:27 2019][1696527.327220] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696527.332376] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:27 2019][1696527.338131] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:27 2019][1696527.344840] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1696527.349906] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.356181] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1696527.362811] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.369086] kworker/39:0 S ffff91599ff9b0c0 0 28897 2 0x00000080 [Wed Feb 6 09:40:27 2019][1696527.376378] Call Trace: [Wed Feb 6 09:40:27 2019][1696527.379017] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:27 2019][1696527.385206] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696527.390353] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:27 2019][1696527.396107] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:27 2019][1696527.402813] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1696527.407873] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.414150] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1696527.420775] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.427048] kworker/u258:2 S ffff9139f3762080 0 28943 2 0x00000080 [Wed Feb 6 09:40:27 2019][1696527.434326] Call Trace: [Wed Feb 6 09:40:27 2019][1696527.436959] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:27 2019][1696527.443141] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696527.448288] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:27 2019][1696527.454043] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:27 2019][1696527.460749] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1696527.465809] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.472081] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1696527.478701] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.484975] ldlm_cn03_031 S ffff914487475140 0 28944 2 0x00000080 [Wed Feb 6 09:40:27 2019][1696527.492259] Call Trace: [Wed Feb 6 09:40:27 2019][1696527.494891] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696527.500085] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.506968] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1696527.512757] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.519248] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.526822] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1696527.531882] [] ? finish_task_switch+0x54/0x1c0 [Wed Feb 6 09:40:27 2019][1696527.538165] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.544444] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1696527.551069] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.557347] ldlm_cn02_029 S ffff913ac8551040 0 28945 2 0x00000080 [Wed Feb 6 09:40:27 2019][1696527.564628] Call Trace: [Wed Feb 6 09:40:27 2019][1696527.567262] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696527.572447] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.579335] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1696527.585125] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.591605] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.599181] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1696527.604245] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.610521] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1696527.617150] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.623422] ldlm_cn02_030 S ffff914661a330c0 0 28946 2 0x00000080 [Wed Feb 6 09:40:27 2019][1696527.630713] Call Trace: [Wed Feb 6 09:40:27 2019][1696527.633343] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696527.638537] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.645423] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1696527.651223] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.657710] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.665284] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1696527.670343] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.676619] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1696527.683238] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.689514] mdt_io03_029 S ffff913ac8554100 0 28948 2 0x00000080 [Wed Feb 6 09:40:27 2019][1696527.696789] Call Trace: [Wed Feb 6 09:40:27 2019][1696527.699420] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696527.704606] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.711485] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1696527.717273] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.723757] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.731336] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1696527.736392] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.742669] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1696527.749288] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.755564] ldlm_cn02_031 S ffff914661a32080 0 28949 2 0x00000080 [Wed Feb 6 09:40:27 2019][1696527.762839] Call Trace: [Wed Feb 6 09:40:27 2019][1696527.765469] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696527.770659] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.777546] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1696527.783335] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.789814] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.797393] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1696527.802452] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.808733] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:27 2019][1696527.815351] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.821622] ll_mgs_0004 S ffff913be6bb8000 0 28952 2 0x00000080 [Wed Feb 6 09:40:27 2019][1696527.828899] Call Trace: [Wed Feb 6 09:40:27 2019][1696527.831529] [] schedule+0x29/0x70 [Wed Feb 6 09:40:27 2019][1696527.836715] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.843598] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:27 2019][1696527.849386] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.855865] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:27 2019][1696527.863444] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:27 2019][1696527.868502] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:27 2019][1696527.874783] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696527.881402] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696527.887676] mdt_rdpg01_013 S ffff913be6bb9040 0 28954 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696527.894949] Call Trace: [Wed Feb 6 09:40:28 2019][1696527.897580] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1696527.902771] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696527.909656] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1696527.915445] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696527.921931] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696527.929508] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1696527.934569] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696527.940850] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696527.947470] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696527.953744] ll_mgs_0005 S ffff9139f1f3c100 0 28956 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696527.961016] Call Trace: [Wed Feb 6 09:40:28 2019][1696527.963647] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1696527.968836] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696527.975714] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1696527.981505] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696527.987986] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696527.995563] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1696528.000625] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.006900] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696528.013529] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.019802] ll_mgs_0006 S ffff9149f62b2080 0 28957 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696528.027074] Call Trace: [Wed Feb 6 09:40:28 2019][1696528.029706] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1696528.034892] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.041772] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1696528.047563] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.054042] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.061621] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1696528.066679] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.072959] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696528.079579] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.085851] ll_mgs_0007 S ffff9149f62b30c0 0 28958 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696528.093135] Call Trace: [Wed Feb 6 09:40:28 2019][1696528.095769] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1696528.100962] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.107844] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1696528.113639] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.120119] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.127696] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1696528.132755] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.139033] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696528.145650] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.151930] mdt_io03_030 S ffff9149f62b5140 0 28960 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696528.159202] Call Trace: [Wed Feb 6 09:40:28 2019][1696528.161831] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1696528.167019] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.173899] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1696528.179688] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.186170] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.193747] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1696528.198806] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.205086] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696528.211702] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.217977] ll_mgs_0008 S ffff912be6b99040 0 28965 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696528.225253] Call Trace: [Wed Feb 6 09:40:28 2019][1696528.227882] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1696528.233072] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.239950] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1696528.245742] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.252228] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.259812] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1696528.264877] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.271154] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696528.277768] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.284040] ll_mgs_0009 S ffff912be6b9a080 0 28966 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696528.291312] Call Trace: [Wed Feb 6 09:40:28 2019][1696528.293942] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1696528.299130] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.306010] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1696528.311800] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.318278] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.325862] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1696528.330924] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.337201] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696528.343819] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.350095] mdt_io03_031 S ffff912be6b98000 0 28969 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696528.357380] Call Trace: [Wed Feb 6 09:40:28 2019][1696528.360011] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1696528.365199] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.372084] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1696528.377874] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.384359] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.391933] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1696528.396991] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.403270] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696528.409892] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.416162] kworker/46:0 S ffff9141bfb85140 0 28975 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696528.423444] Call Trace: [Wed Feb 6 09:40:28 2019][1696528.426076] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:28 2019][1696528.432269] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1696528.437417] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:28 2019][1696528.443169] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:28 2019][1696528.449875] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1696528.454928] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.461205] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696528.467825] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.474099] ll_mgs_0010 S ffff9141bfb84100 0 28978 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696528.481369] Call Trace: [Wed Feb 6 09:40:28 2019][1696528.484005] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1696528.489200] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.496080] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1696528.501878] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.508364] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.515939] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1696528.521002] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.527277] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696528.533893] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.540169] ll_mgs_0011 S ffff912aa9a29040 0 28979 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696528.547445] Call Trace: [Wed Feb 6 09:40:28 2019][1696528.550075] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1696528.555263] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.562143] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1696528.567931] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.574425] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.582002] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1696528.587060] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.593333] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696528.599951] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.606231] ll_mgs_0012 S ffff91370ca3a080 0 28980 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696528.613503] Call Trace: [Wed Feb 6 09:40:28 2019][1696528.616135] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1696528.621322] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.628217] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1696528.634013] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.640502] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.648074] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1696528.653133] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.659409] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696528.666030] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.672303] ll_mgs_0013 S ffff91370ca39040 0 28981 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696528.679589] Call Trace: [Wed Feb 6 09:40:28 2019][1696528.682223] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1696528.687411] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.694294] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1696528.700083] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.706574] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.714151] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1696528.719214] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.725483] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696528.732103] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.738372] ll_mgs_0014 S ffff91370ca3e180 0 28982 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696528.745651] Call Trace: [Wed Feb 6 09:40:28 2019][1696528.748289] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1696528.753474] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.760352] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1696528.766142] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.772623] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.780210] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1696528.785272] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.791553] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696528.798178] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.804448] ll_mgs_0015 S ffff91370ca3c100 0 28983 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696528.811723] Call Trace: [Wed Feb 6 09:40:28 2019][1696528.814356] [] schedule+0x29/0x70 [Wed Feb 6 09:40:28 2019][1696528.819549] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.826438] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:28 2019][1696528.832236] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.838715] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:28 2019][1696528.846292] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:28 2019][1696528.851365] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.857644] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:28 2019][1696528.864270] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:28 2019][1696528.870545] ll_mgs_0016 S ffff91370ca3d140 0 28984 2 0x00000080 [Wed Feb 6 09:40:28 2019][1696528.877825] Call Trace: [Wed Feb 6 09:40:28 2019][1696528.880461] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1696528.885653] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696528.892532] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1696528.898323] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696528.904807] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696528.912379] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1696528.917442] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696528.923716] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1696528.930339] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696528.936609] ll_mgs_0017 S ffff91370ca38000 0 28985 2 0x00000080 [Wed Feb 6 09:40:29 2019][1696528.943893] Call Trace: [Wed Feb 6 09:40:29 2019][1696528.946526] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1696528.951722] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696528.958606] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1696528.964417] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696528.970910] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696528.978490] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1696528.983550] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696528.989826] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1696528.996445] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.002721] ll_mgs_0018 S ffff912aa995a080 0 28986 2 0x00000080 [Wed Feb 6 09:40:29 2019][1696529.010000] Call Trace: [Wed Feb 6 09:40:29 2019][1696529.012636] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1696529.017824] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.024711] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1696529.030512] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.037007] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.044585] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1696529.049643] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.055920] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1696529.062540] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.068814] ll_mgs_0019 S ffff912da3d24100 0 28987 2 0x00000080 [Wed Feb 6 09:40:29 2019][1696529.076090] Call Trace: [Wed Feb 6 09:40:29 2019][1696529.078720] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1696529.083914] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.090797] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1696529.096587] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.103071] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.110643] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1696529.115701] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.121976] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1696529.128597] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.134873] ll_mgs_0020 S ffff912da3d230c0 0 28988 2 0x00000080 [Wed Feb 6 09:40:29 2019][1696529.142147] Call Trace: [Wed Feb 6 09:40:29 2019][1696529.144778] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1696529.149965] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.156847] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1696529.162634] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.169116] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.176693] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1696529.181752] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.188028] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1696529.194649] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.200922] ll_mgs_0021 S ffff912da3d25140 0 28989 2 0x00000080 [Wed Feb 6 09:40:29 2019][1696529.208197] Call Trace: [Wed Feb 6 09:40:29 2019][1696529.210830] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1696529.216017] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.222895] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1696529.228686] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.235167] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.242745] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1696529.247808] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.254077] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1696529.260702] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.266977] ll_mgs_0022 S ffff912c13620000 0 28990 2 0x00000080 [Wed Feb 6 09:40:29 2019][1696529.274250] Call Trace: [Wed Feb 6 09:40:29 2019][1696529.276881] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1696529.282069] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.288957] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1696529.294746] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.301230] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.308803] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1696529.313865] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.320147] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1696529.326771] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.333040] ll_mgs_0023 S ffff912c136230c0 0 28991 2 0x00000080 [Wed Feb 6 09:40:29 2019][1696529.340317] Call Trace: [Wed Feb 6 09:40:29 2019][1696529.342947] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1696529.348139] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.355025] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1696529.360819] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.367305] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.374883] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1696529.379940] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.386220] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1696529.392845] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.399121] ll_mgs_0024 S ffff912c13624100 0 28992 2 0x00000080 [Wed Feb 6 09:40:29 2019][1696529.406396] Call Trace: [Wed Feb 6 09:40:29 2019][1696529.409031] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1696529.414217] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.421100] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1696529.426902] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.433392] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.440971] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1696529.446032] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.452311] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1696529.458939] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.465215] ll_mgs_0025 S ffff912c13622080 0 28993 2 0x00000080 [Wed Feb 6 09:40:29 2019][1696529.472492] Call Trace: [Wed Feb 6 09:40:29 2019][1696529.475127] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1696529.480317] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.487202] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1696529.492994] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.499472] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.507049] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1696529.512106] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.518375] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1696529.524994] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.531271] ll_mgs_0026 S ffff912c13626180 0 28994 2 0x00000080 [Wed Feb 6 09:40:29 2019][1696529.538546] Call Trace: [Wed Feb 6 09:40:29 2019][1696529.541177] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1696529.546363] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.553242] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1696529.559031] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.565513] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.573092] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1696529.578149] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.584426] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1696529.591046] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.597327] ll_mgs_0027 S ffff91383a612080 0 28995 2 0x00000080 [Wed Feb 6 09:40:29 2019][1696529.604607] Call Trace: [Wed Feb 6 09:40:29 2019][1696529.607238] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1696529.612427] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.619312] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1696529.625105] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.631589] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.639174] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1696529.644235] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.650517] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1696529.657142] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.663416] ll_mgs_0028 S ffff91383a615140 0 28996 2 0x00000080 [Wed Feb 6 09:40:29 2019][1696529.670698] Call Trace: [Wed Feb 6 09:40:29 2019][1696529.673334] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1696529.678529] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.685421] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1696529.691217] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.697704] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.705283] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1696529.710344] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.716613] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1696529.723237] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.729513] ll_mgs_0029 S ffff91383a614100 0 28997 2 0x00000080 [Wed Feb 6 09:40:29 2019][1696529.736796] Call Trace: [Wed Feb 6 09:40:29 2019][1696529.739436] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1696529.744625] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.751508] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1696529.757295] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.763777] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.771354] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1696529.776413] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.782689] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1696529.789308] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.795589] ll_mgs_0030 S ffff9159eb750000 0 29001 2 0x00000080 [Wed Feb 6 09:40:29 2019][1696529.802859] Call Trace: [Wed Feb 6 09:40:29 2019][1696529.805491] [] schedule+0x29/0x70 [Wed Feb 6 09:40:29 2019][1696529.810678] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.817561] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:29 2019][1696529.823345] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.829832] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:29 2019][1696529.837406] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:29 2019][1696529.842465] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.848741] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:29 2019][1696529.855365] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:29 2019][1696529.861639] ll_mgs_0031 S ffff91218e9d5140 0 29002 2 0x00000080 [Wed Feb 6 09:40:30 2019][1696529.868927] Call Trace: [Wed Feb 6 09:40:30 2019][1696529.871562] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696529.876752] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Wed Feb 6 09:40:30 2019][1696529.883634] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1696529.889427] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Wed Feb 6 09:40:30 2019][1696529.895918] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:40:30 2019][1696529.903494] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1696529.908547] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696529.914825] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1696529.921444] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696529.927722] kworker/27:0 S ffff9159fa6d1040 0 29045 2 0x00000080 [Wed Feb 6 09:40:30 2019][1696529.935012] Call Trace: [Wed Feb 6 09:40:30 2019][1696529.937644] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:30 2019][1696529.943830] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696529.948980] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:30 2019][1696529.954235] LustreError: 17223:0:(upcall_cache.c:233:upcall_cache_get_entry()) acquire for key 290397: error -110 [Wed Feb 6 09:40:30 2019][1696529.965167] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:30 2019][1696529.971869] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1696529.976932] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696529.983205] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1696529.989830] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696529.996103] systemd-logind S ffff9156a426a080 0 29049 1 0x00000080 [Wed Feb 6 09:40:30 2019][1696530.003390] Call Trace: [Wed Feb 6 09:40:30 2019][1696530.006027] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696530.011175] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Wed Feb 6 09:40:30 2019][1696530.018409] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Wed Feb 6 09:40:30 2019][1696530.025382] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:30 2019][1696530.031915] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:40:30 2019][1696530.037149] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1696530.042902] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:40:30 2019][1696530.048659] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:30 2019][1696530.054846] kworker/14:2 S ffff913c8a662080 0 29068 2 0x00000080 [Wed Feb 6 09:40:30 2019][1696530.062128] Call Trace: [Wed Feb 6 09:40:30 2019][1696530.064760] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:30 2019][1696530.070947] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696530.076098] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:30 2019][1696530.081851] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:30 2019][1696530.088564] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1696530.093626] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.099901] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1696530.106526] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.112804] kworker/29:2 S ffff9139f6035140 0 29130 2 0x00000080 [Wed Feb 6 09:40:30 2019][1696530.120093] Call Trace: [Wed Feb 6 09:40:30 2019][1696530.122724] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:30 2019][1696530.128914] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696530.134066] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:30 2019][1696530.139821] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:30 2019][1696530.146527] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1696530.151588] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.157861] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1696530.164487] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.170759] kworker/19:2 S ffff9159f7d14100 0 29155 2 0x00000080 [Wed Feb 6 09:40:30 2019][1696530.178052] Call Trace: [Wed Feb 6 09:40:30 2019][1696530.180689] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:30 2019][1696530.186883] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696530.192028] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:30 2019][1696530.197791] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:30 2019][1696530.204501] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1696530.209560] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.215839] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1696530.222458] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.228728] kworker/41:1 S ffff9139f6030000 0 29345 2 0x00000080 [Wed Feb 6 09:40:30 2019][1696530.236012] Call Trace: [Wed Feb 6 09:40:30 2019][1696530.238645] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696530.243793] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:30 2019][1696530.249547] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:30 2019][1696530.256254] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1696530.261319] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.267593] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1696530.274215] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.280486] kworker/u259:0 S ffff9149f62b1040 0 29378 2 0x00000080 [Wed Feb 6 09:40:30 2019][1696530.287761] Call Trace: [Wed Feb 6 09:40:30 2019][1696530.290402] [] ? start_worker+0x27/0x30 [Wed Feb 6 09:40:30 2019][1696530.296071] [] ? manage_workers.isra.25+0x189/0x2a0 [Wed Feb 6 09:40:30 2019][1696530.302781] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696530.307931] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:30 2019][1696530.313680] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:30 2019][1696530.320385] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1696530.325445] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.331719] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1696530.338339] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.344613] kworker/13:0 S ffff912aa9a5b0c0 0 29471 2 0x00000080 [Wed Feb 6 09:40:30 2019][1696530.351894] Call Trace: [Wed Feb 6 09:40:30 2019][1696530.354528] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:30 2019][1696530.360721] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696530.365864] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:30 2019][1696530.371619] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:30 2019][1696530.378331] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1696530.383393] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.389667] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1696530.396291] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.402562] kworker/36:0 S ffff911b2974a080 0 29485 2 0x00000080 [Wed Feb 6 09:40:30 2019][1696530.409844] Call Trace: [Wed Feb 6 09:40:30 2019][1696530.412480] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:30 2019][1696530.418665] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696530.423811] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:30 2019][1696530.429568] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:30 2019][1696530.436282] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1696530.441340] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.447614] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1696530.454237] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.460514] pickup S ffff914ad73c5140 0 29530 19477 0x00000080 [Wed Feb 6 09:40:30 2019][1696530.467791] Call Trace: [Wed Feb 6 09:40:30 2019][1696530.470427] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:40:30 2019][1696530.477131] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696530.482283] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:40:30 2019][1696530.489419] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:40:30 2019][1696530.495340] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:40:30 2019][1696530.502653] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:30 2019][1696530.509184] [] ep_poll+0x23e/0x360 [Wed Feb 6 09:40:30 2019][1696530.514416] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:30 2019][1696530.520173] [] SyS_epoll_wait+0xed/0x120 [Wed Feb 6 09:40:30 2019][1696530.525934] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:30 2019][1696530.532121] kworker/26:2 S ffff913a4cdf8000 0 29612 2 0x00000080 [Wed Feb 6 09:40:30 2019][1696530.539406] Call Trace: [Wed Feb 6 09:40:30 2019][1696530.542037] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696530.547191] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:30 2019][1696530.552939] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:30 2019][1696530.559652] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1696530.564711] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.570985] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1696530.577606] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.583881] kworker/43:0 S ffff9156a4269040 0 29622 2 0x00000080 [Wed Feb 6 09:40:30 2019][1696530.591164] Call Trace: [Wed Feb 6 09:40:30 2019][1696530.593799] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:30 2019][1696530.599983] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696530.605132] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:30 2019][1696530.610886] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:30 2019][1696530.617594] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1696530.622657] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.628934] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1696530.635555] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.641832] kworker/35:0 S ffff915365740000 0 29665 2 0x00000080 [Wed Feb 6 09:40:30 2019][1696530.649108] Call Trace: [Wed Feb 6 09:40:30 2019][1696530.651741] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:30 2019][1696530.657930] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696530.663089] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:30 2019][1696530.668843] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:30 2019][1696530.675551] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1696530.680613] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.686890] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1696530.693508] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.699787] kworker/3:0 S ffff914aa94bb0c0 0 29739 2 0x00000080 [Wed Feb 6 09:40:30 2019][1696530.707071] Call Trace: [Wed Feb 6 09:40:30 2019][1696530.709713] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:30 2019][1696530.715900] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696530.721042] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:30 2019][1696530.726800] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:30 2019][1696530.733507] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1696530.738568] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.744847] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1696530.751466] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.757743] kworker/2:2 S ffff913ccad9d140 0 29774 2 0x00000080 [Wed Feb 6 09:40:30 2019][1696530.765027] Call Trace: [Wed Feb 6 09:40:30 2019][1696530.767661] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696530.772810] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:30 2019][1696530.778566] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:30 2019][1696530.785276] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1696530.790335] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.796609] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1696530.803231] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.809505] kworker/10:1 S ffff913ccad9b0c0 0 29787 2 0x00000080 [Wed Feb 6 09:40:30 2019][1696530.816776] Call Trace: [Wed Feb 6 09:40:30 2019][1696530.819403] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:30 2019][1696530.825594] [] schedule+0x29/0x70 [Wed Feb 6 09:40:30 2019][1696530.830748] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:30 2019][1696530.836509] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:30 2019][1696530.843219] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:30 2019][1696530.848285] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:30 2019][1696530.854558] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:30 2019][1696530.861176] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696530.867444] kworker/30:0 S ffff9149fec2e180 0 29820 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696530.874728] Call Trace: [Wed Feb 6 09:40:31 2019][1696530.877359] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696530.883546] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696530.888695] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696530.894450] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696530.901163] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696530.906219] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696530.912498] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696530.919115] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696530.925389] kworker/6:2 S ffff9149ca741040 0 29831 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696530.932672] Call Trace: [Wed Feb 6 09:40:31 2019][1696530.935312] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696530.941502] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696530.946649] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696530.952404] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696530.959115] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696530.964173] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696530.970441] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696530.977062] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696530.983336] kworker/22:0 S ffff91488911b0c0 0 29879 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696530.990610] Call Trace: [Wed Feb 6 09:40:31 2019][1696530.993244] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696530.999433] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696531.004582] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696531.010339] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696531.017042] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696531.022105] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.028380] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696531.035001] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.041276] kworker/32:1 S ffff9129fc1a9040 0 29908 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696531.048557] Call Trace: [Wed Feb 6 09:40:31 2019][1696531.051189] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696531.057378] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696531.062530] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696531.068283] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696531.074989] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696531.080052] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.086326] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696531.092952] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.099221] kworker/16:0 S ffff9129fc1ae180 0 29942 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696531.106494] Call Trace: [Wed Feb 6 09:40:31 2019][1696531.109133] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696531.115317] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696531.120466] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696531.126219] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696531.132931] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696531.137988] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.144264] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696531.150886] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.157161] kworker/37:2 S ffff9138d62b30c0 0 30074 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696531.164440] Call Trace: [Wed Feb 6 09:40:31 2019][1696531.167076] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696531.173266] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696531.178411] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696531.184166] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696531.190875] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696531.195939] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.202215] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696531.208833] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.215106] kworker/4:0 S ffff91218e9d30c0 0 30083 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696531.222386] Call Trace: [Wed Feb 6 09:40:31 2019][1696531.225013] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696531.231201] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696531.236355] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696531.242108] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696531.248812] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696531.253873] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.260148] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696531.266762] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.273037] kworker/u256:1 S ffff9149ca782080 0 30143 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696531.280318] Call Trace: [Wed Feb 6 09:40:31 2019][1696531.282953] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696531.289143] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696531.294291] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696531.300043] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696531.306750] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696531.311812] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.318086] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696531.324715] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.330987] kworker/42:1 S ffff9149ca781040 0 30164 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696531.338271] Call Trace: [Wed Feb 6 09:40:31 2019][1696531.340898] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696531.347090] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696531.352238] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696531.357992] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696531.364700] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696531.369764] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.376037] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696531.382656] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.388930] kworker/38:0 S ffff9149ccb78000 0 30176 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696531.396211] Call Trace: [Wed Feb 6 09:40:31 2019][1696531.398845] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696531.405034] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696531.410185] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696531.415942] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696531.422659] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696531.427717] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.433994] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696531.440609] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.446877] kworker/8:1 S ffff911b2902d140 0 30223 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696531.454159] Call Trace: [Wed Feb 6 09:40:31 2019][1696531.456791] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696531.462971] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696531.468123] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696531.473874] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696531.480581] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696531.485636] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.491916] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696531.498534] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.504808] kworker/34:0 S ffff9149ccb7e180 0 30225 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696531.512088] Call Trace: [Wed Feb 6 09:40:31 2019][1696531.514724] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696531.520909] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696531.526059] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696531.531813] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696531.538523] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696531.543583] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.549858] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696531.556478] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.562751] kworker/5:0 S ffff913766150000 0 30238 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696531.570038] Call Trace: [Wed Feb 6 09:40:31 2019][1696531.572668] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696531.578860] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696531.584008] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696531.589762] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696531.596471] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696531.601531] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.607802] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696531.614419] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.620693] kworker/u259:2 S ffff9149ccb79040 0 30246 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696531.627970] Call Trace: [Wed Feb 6 09:40:31 2019][1696531.630596] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696531.636777] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696531.641928] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696531.647682] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696531.654388] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696531.659450] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.665726] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696531.672350] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.678621] kworker/33:2 S ffff913766151040 0 30273 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696531.685914] Call Trace: [Wed Feb 6 09:40:31 2019][1696531.688548] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696531.694734] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696531.699881] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696531.705637] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696531.712346] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696531.717404] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.723684] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696531.730306] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.736576] kworker/u256:0 S ffff913766156180 0 30284 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696531.743864] Call Trace: [Wed Feb 6 09:40:31 2019][1696531.746500] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696531.752688] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696531.757837] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696531.763591] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696531.770300] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696531.775359] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.781627] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696531.788248] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.794522] kworker/1:2 S ffff913766155140 0 30286 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696531.801794] Call Trace: [Wed Feb 6 09:40:31 2019][1696531.804420] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:31 2019][1696531.810609] [] schedule+0x29/0x70 [Wed Feb 6 09:40:31 2019][1696531.815756] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:31 2019][1696531.821513] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:31 2019][1696531.828225] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:31 2019][1696531.833282] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.839555] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:31 2019][1696531.846178] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:31 2019][1696531.852453] kworker/18:0 S ffff913a4cdfa080 0 30339 2 0x00000080 [Wed Feb 6 09:40:31 2019][1696531.859735] Call Trace: [Wed Feb 6 09:40:31 2019][1696531.862372] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:32 2019][1696531.868560] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1696531.873708] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:32 2019][1696531.879459] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:32 2019][1696531.886166] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1696531.891233] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1696531.897521] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1696531.904143] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1696531.910417] sshd D ffff9137661530c0 0 30345 19077 0x00000080 [Wed Feb 6 09:40:32 2019][1696531.917705] Call Trace: [Wed Feb 6 09:40:32 2019][1696531.920341] [] ? account_entity_dequeue+0xae/0xd0 [Wed Feb 6 09:40:32 2019][1696531.926877] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1696531.932023] [] schedule_timeout+0x221/0x2d0 [Wed Feb 6 09:40:32 2019][1696531.938041] [] ? __switch_to+0x151/0x580 [Wed Feb 6 09:40:32 2019][1696531.943795] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:40:32 2019][1696531.949807] [] wait_for_completion+0xfd/0x140 [Wed Feb 6 09:40:32 2019][1696531.955995] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:40:32 2019][1696531.961743] [] flush_work+0xfd/0x190 [Wed Feb 6 09:40:32 2019][1696531.967154] [] ? move_linked_works+0x90/0x90 [Wed Feb 6 09:40:32 2019][1696531.973257] [] tty_flush_to_ldisc+0x29/0x30 [Wed Feb 6 09:40:32 2019][1696531.979269] [] n_tty_poll+0x95/0x1d0 [Wed Feb 6 09:40:32 2019][1696531.984680] [] tty_poll+0x75/0x90 [Wed Feb 6 09:40:32 2019][1696531.989828] [] do_select+0x3a8/0x7c0 [Wed Feb 6 09:40:32 2019][1696531.995235] [] ? ip_finish_output+0x284/0x7b0 [Wed Feb 6 09:40:32 2019][1696532.001434] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:32 2019][1696532.006926] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:32 2019][1696532.012416] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:32 2019][1696532.017912] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:32 2019][1696532.023413] [] ? check_preempt_wakeup+0x148/0x250 [Wed Feb 6 09:40:32 2019][1696532.029946] [] ? tracing_is_on+0x15/0x30 [Wed Feb 6 09:40:32 2019][1696532.035697] [] ? tracing_record_cmdline+0x1d/0x120 [Wed Feb 6 09:40:32 2019][1696532.042319] [] ? probe_sched_wakeup+0x2b/0xa0 [Wed Feb 6 09:40:32 2019][1696532.048511] [] ? tracing_is_on+0x15/0x30 [Wed Feb 6 09:40:32 2019][1696532.054264] [] ? tracing_record_cmdline+0x1d/0x120 [Wed Feb 6 09:40:32 2019][1696532.060889] [] ? probe_sched_wakeup+0x2b/0xa0 [Wed Feb 6 09:40:32 2019][1696532.067077] [] ? ttwu_do_wakeup+0xb5/0xe0 [Wed Feb 6 09:40:32 2019][1696532.072921] [] ? try_to_wake_up+0x190/0x390 [Wed Feb 6 09:40:32 2019][1696532.078935] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:40:32 2019][1696532.085384] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:32 2019][1696532.091400] [] core_sys_select+0x1ff/0x340 [Wed Feb 6 09:40:32 2019][1696532.097329] [] ? __wake_up+0x44/0x50 [Wed Feb 6 09:40:32 2019][1696532.102750] [] ? tty_write_unlock+0x31/0x40 [Wed Feb 6 09:40:32 2019][1696532.108770] [] ? tty_ldisc_deref+0x19/0x20 [Wed Feb 6 09:40:32 2019][1696532.114695] [] ? tty_write+0x1eb/0x2c0 [Wed Feb 6 09:40:32 2019][1696532.120280] [] ? n_tty_ioctl+0xe0/0xe0 [Wed Feb 6 09:40:32 2019][1696532.125859] [] SyS_select+0xca/0x120 [Wed Feb 6 09:40:32 2019][1696532.131269] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:32 2019][1696532.137459] bash R running task 0 30347 30345 0x00000088 [Wed Feb 6 09:40:32 2019][1696532.144744] Call Trace: [Wed Feb 6 09:40:32 2019][1696532.147380] [] sched_show_task+0xa8/0x110 [Wed Feb 6 09:40:32 2019][1696532.153223] [] show_state_filter+0x82/0xd0 [Wed Feb 6 09:40:32 2019][1696532.159150] [] sysrq_handle_showstate+0x10/0x20 [Wed Feb 6 09:40:32 2019][1696532.165511] [] __handle_sysrq+0x10d/0x170 [Wed Feb 6 09:40:32 2019][1696532.171353] [] write_sysrq_trigger+0x28/0x40 [Wed Feb 6 09:40:32 2019][1696532.177453] [] proc_reg_write+0x40/0x80 [Wed Feb 6 09:40:32 2019][1696532.183121] [] vfs_write+0xc0/0x1f0 [Wed Feb 6 09:40:32 2019][1696532.188445] [] SyS_write+0x7f/0xf0 [Wed Feb 6 09:40:32 2019][1696532.193680] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:32 2019][1696532.199864] kworker/u256:2 S ffff913a4cdfe180 0 30376 2 0x00000080 [Wed Feb 6 09:40:32 2019][1696532.207140] Call Trace: [Wed Feb 6 09:40:32 2019][1696532.209774] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 09:40:32 2019][1696532.216398] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:32 2019][1696532.222582] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1696532.227732] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:32 2019][1696532.233482] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:32 2019][1696532.240187] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1696532.245249] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1696532.251523] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1696532.258147] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1696532.264421] kworker/38:2 S ffff913a4cdf9040 0 30385 2 0x00000080 [Wed Feb 6 09:40:32 2019][1696532.271707] Call Trace: [Wed Feb 6 09:40:32 2019][1696532.274344] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:32 2019][1696532.280530] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1696532.285683] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:32 2019][1696532.291434] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:32 2019][1696532.298140] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1696532.303204] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1696532.309478] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1696532.316095] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1696532.322363] kworker/1:0 S ffff9135b0d75140 0 30400 2 0x00000080 [Wed Feb 6 09:40:32 2019][1696532.329654] Call Trace: [Wed Feb 6 09:40:32 2019][1696532.332289] [] ? process_one_work+0x21a/0x440 [Wed Feb 6 09:40:32 2019][1696532.338473] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1696532.343623] [] worker_thread+0x1d9/0x3c0 [Wed Feb 6 09:40:32 2019][1696532.349376] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Feb 6 09:40:32 2019][1696532.356088] [] kthread+0xd1/0xe0 [Wed Feb 6 09:40:32 2019][1696532.361150] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1696532.367421] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:40:32 2019][1696532.374047] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:40:32 2019][1696532.380323] l_getidentity S ffff9159f65dc100 0 30457 28943 0x00000080 [Wed Feb 6 09:40:32 2019][1696532.387611] Call Trace: [Wed Feb 6 09:40:32 2019][1696532.390247] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:40:32 2019][1696532.396952] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1696532.402097] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:40:32 2019][1696532.409231] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:40:32 2019][1696532.415157] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:40:32 2019][1696532.422469] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:32 2019][1696532.429008] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:40:32 2019][1696532.435292] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:40:32 2019][1696532.440879] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:40:32 2019][1696532.446891] [] ? __wake_up_sync_key+0x4f/0x60 [Wed Feb 6 09:40:32 2019][1696532.453080] [] ? sock_def_readable+0x3a/0x70 [Wed Feb 6 09:40:32 2019][1696532.459181] [] ? unix_stream_sendmsg+0x3fd/0x430 [Wed Feb 6 09:40:32 2019][1696532.465627] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:32 2019][1696532.471121] [] ? unix_stream_recvmsg+0x54/0x70 [Wed Feb 6 09:40:32 2019][1696532.477398] [] ? sock_aio_read.part.9+0x14c/0x170 [Wed Feb 6 09:40:32 2019][1696532.483932] [] ? SYSC_sendto+0x121/0x1c0 [Wed Feb 6 09:40:32 2019][1696532.489691] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:40:32 2019][1696532.495528] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:40:32 2019][1696532.500771] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:32 2019][1696532.506961] l_getidentity S ffff9129f375b0c0 0 30458 16011 0x00000080 [Wed Feb 6 09:40:32 2019][1696532.514253] Call Trace: [Wed Feb 6 09:40:32 2019][1696532.516885] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:40:32 2019][1696532.523596] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1696532.528741] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:40:32 2019][1696532.535879] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:40:32 2019][1696532.541808] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:40:32 2019][1696532.549122] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:32 2019][1696532.555659] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:40:32 2019][1696532.561939] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:40:32 2019][1696532.567527] [] ? skb_copy_datagram_from_iovec+0x5c/0x280 [Wed Feb 6 09:40:32 2019][1696532.574669] [] ? skb_queue_tail+0x43/0x50 [Wed Feb 6 09:40:32 2019][1696532.580511] [] ? unix_stream_sendmsg+0x3fd/0x430 [Wed Feb 6 09:40:32 2019][1696532.586966] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:32 2019][1696532.592464] [] ? do_read_fault.isra.61+0x139/0x1b0 [Wed Feb 6 09:40:32 2019][1696532.599092] [] ? SYSC_sendto+0x121/0x1c0 [Wed Feb 6 09:40:32 2019][1696532.604843] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:40:32 2019][1696532.610694] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:40:32 2019][1696532.615935] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:32 2019][1696532.622127] sshd S ffff9129c3fd30c0 0 30459 19077 0x00000080 [Wed Feb 6 09:40:32 2019][1696532.629404] Call Trace: [Wed Feb 6 09:40:32 2019][1696532.632047] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:40:32 2019][1696532.638760] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1696532.643908] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:40:32 2019][1696532.651045] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:40:32 2019][1696532.656977] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:40:32 2019][1696532.664297] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:32 2019][1696532.670835] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:40:32 2019][1696532.677112] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:40:32 2019][1696532.682702] [] ? skb_copy_datagram_from_iovec+0x5c/0x280 [Wed Feb 6 09:40:32 2019][1696532.689848] [] ? legitimize_mnt+0x16/0x40 [Wed Feb 6 09:40:32 2019][1696532.695688] [] ? skb_queue_tail+0x43/0x50 [Wed Feb 6 09:40:32 2019][1696532.701533] [] ? unix_stream_sendmsg+0x3fd/0x430 [Wed Feb 6 09:40:32 2019][1696532.707982] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:32 2019][1696532.713473] [] ? kern_path+0x60/0xb0 [Wed Feb 6 09:40:32 2019][1696532.718883] [] ? ep_poll_callback+0xee/0x210 [Wed Feb 6 09:40:32 2019][1696532.724985] [] ? SYSC_sendto+0x121/0x1c0 [Wed Feb 6 09:40:32 2019][1696532.730739] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:40:32 2019][1696532.736581] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:40:32 2019][1696532.741817] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:32 2019][1696532.748006] sshd S ffff9139f66da080 0 30460 30459 0x000001a0 [Wed Feb 6 09:40:32 2019][1696532.755279] Call Trace: [Wed Feb 6 09:40:32 2019][1696532.757913] [] ? __enqueue_entity+0x78/0x80 [Wed Feb 6 09:40:32 2019][1696532.763924] [] schedule+0x29/0x70 [Wed Feb 6 09:40:32 2019][1696532.769070] [] schedule_timeout+0x221/0x2d0 [Wed Feb 6 09:40:32 2019][1696532.775089] [] ? __kmalloc_node_track_caller+0x58/0x290 [Wed Feb 6 09:40:32 2019][1696532.782145] [] ? probe_sched_wakeup+0x2b/0xa0 [Wed Feb 6 09:40:32 2019][1696532.788329] [] ? __alloc_skb+0x8d/0x2d0 [Wed Feb 6 09:40:32 2019][1696532.793998] [] ? prepare_to_wait+0x56/0x90 [Wed Feb 6 09:40:32 2019][1696532.799928] [] unix_stream_read_generic+0x309/0x8e0 [Wed Feb 6 09:40:32 2019][1696532.806634] [] ? wake_up_atomic_t+0x30/0x30 [Wed Feb 6 09:40:32 2019][1696532.812649] [] ? __wake_up_sync_key+0x4f/0x60 [Wed Feb 6 09:40:32 2019][1696532.818836] [] ? sock_def_readable+0x3a/0x70 [Wed Feb 6 09:40:32 2019][1696532.824939] [] ? unix_stream_sendmsg+0x3fd/0x430 [Wed Feb 6 09:40:32 2019][1696532.831390] [] unix_stream_recvmsg+0x54/0x70 [Wed Feb 6 09:40:32 2019][1696532.837487] [] ? unix_ioctl+0x70/0x70 [Wed Feb 6 09:40:32 2019][1696532.842982] [] sock_aio_read.part.9+0x14c/0x170 [Wed Feb 6 09:40:32 2019][1696532.849344] [] sock_aio_read+0x21/0x30 [Wed Feb 6 09:40:32 2019][1696532.854926] [] do_sync_read+0x93/0xe0 [Wed Feb 6 09:40:32 2019][1696532.860420] [] vfs_read+0x145/0x170 [Wed Feb 6 09:40:32 2019][1696532.865746] [] SyS_read+0x7f/0xf0 [Wed Feb 6 09:40:33 2019][1696532.870889] [] tracesys+0xa3/0xc9 [Wed Feb 6 09:40:33 2019][1696532.876038] l_getidentity S ffff9144874730c0 0 30461 28593 0x00000080 [Wed Feb 6 09:40:33 2019][1696532.883332] Call Trace: [Wed Feb 6 09:40:33 2019][1696532.885969] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:40:33 2019][1696532.892683] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1696532.897827] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:40:33 2019][1696532.904973] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:40:33 2019][1696532.910899] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:40:33 2019][1696532.918213] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:33 2019][1696532.924747] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:40:33 2019][1696532.931021] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:40:33 2019][1696532.936605] [] ? skb_copy_datagram_from_iovec+0x5c/0x280 [Wed Feb 6 09:40:33 2019][1696532.943744] [] ? skb_queue_tail+0x43/0x50 [Wed Feb 6 09:40:33 2019][1696532.949587] [] ? unix_stream_sendmsg+0x3fd/0x430 [Wed Feb 6 09:40:33 2019][1696532.956034] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:33 2019][1696532.961533] [] ? do_read_fault.isra.61+0x139/0x1b0 [Wed Feb 6 09:40:33 2019][1696532.968151] [] ? SYSC_sendto+0x121/0x1c0 [Wed Feb 6 09:40:33 2019][1696532.973907] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:40:33 2019][1696532.979745] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:40:33 2019][1696532.984975] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:33 2019][1696532.991161] l_getidentity S ffff9139f65e2080 0 30462 16011 0x00000080 [Wed Feb 6 09:40:33 2019][1696532.998436] Call Trace: [Wed Feb 6 09:40:33 2019][1696533.001067] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:40:33 2019][1696533.007775] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1696533.012923] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:40:33 2019][1696533.020064] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:40:33 2019][1696533.025994] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:40:33 2019][1696533.033308] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:33 2019][1696533.039841] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:40:33 2019][1696533.046118] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:40:33 2019][1696533.051701] [] ? skb_copy_datagram_from_iovec+0x5c/0x280 [Wed Feb 6 09:40:33 2019][1696533.058841] [] ? skb_queue_tail+0x43/0x50 [Wed Feb 6 09:40:33 2019][1696533.064685] [] ? unix_stream_sendmsg+0x3fd/0x430 [Wed Feb 6 09:40:33 2019][1696533.071130] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:33 2019][1696533.076630] [] ? do_read_fault.isra.61+0x139/0x1b0 [Wed Feb 6 09:40:33 2019][1696533.083248] [] ? SYSC_sendto+0x121/0x1c0 [Wed Feb 6 09:40:33 2019][1696533.089003] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:40:33 2019][1696533.094847] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:40:33 2019][1696533.100077] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:33 2019][1696533.106268] l_getidentity S ffff9139f2ada080 0 30463 28943 0x00000080 [Wed Feb 6 09:40:33 2019][1696533.113549] Call Trace: [Wed Feb 6 09:40:33 2019][1696533.116184] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:40:33 2019][1696533.122891] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1696533.128040] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:40:33 2019][1696533.135177] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:40:33 2019][1696533.141101] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:40:33 2019][1696533.148414] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:33 2019][1696533.154952] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:40:33 2019][1696533.161228] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:40:33 2019][1696533.166806] [] ? skb_copy_datagram_from_iovec+0x5c/0x280 [Wed Feb 6 09:40:33 2019][1696533.173948] [] ? skb_queue_tail+0x43/0x50 [Wed Feb 6 09:40:33 2019][1696533.179791] [] ? unix_stream_sendmsg+0x3fd/0x430 [Wed Feb 6 09:40:33 2019][1696533.186238] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:33 2019][1696533.191735] [] ? do_read_fault.isra.61+0x139/0x1b0 [Wed Feb 6 09:40:33 2019][1696533.198366] [] ? SYSC_sendto+0x121/0x1c0 [Wed Feb 6 09:40:33 2019][1696533.204119] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:40:33 2019][1696533.209959] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:40:33 2019][1696533.215193] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:33 2019][1696533.221383] l_getidentity S ffff9139f65e30c0 0 30467 16011 0x00000080 [Wed Feb 6 09:40:33 2019][1696533.228656] Call Trace: [Wed Feb 6 09:40:33 2019][1696533.231285] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:40:33 2019][1696533.237997] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1696533.243145] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:40:33 2019][1696533.250281] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:40:33 2019][1696533.256212] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:40:33 2019][1696533.263527] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:33 2019][1696533.270060] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:40:33 2019][1696533.276336] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:40:33 2019][1696533.281921] [] ? skb_copy_datagram_from_iovec+0x5c/0x280 [Wed Feb 6 09:40:33 2019][1696533.289058] [] ? skb_queue_tail+0x43/0x50 [Wed Feb 6 09:40:33 2019][1696533.294904] [] ? unix_stream_sendmsg+0x3fd/0x430 [Wed Feb 6 09:40:33 2019][1696533.301347] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:33 2019][1696533.306842] [] ? do_read_fault.isra.61+0x139/0x1b0 [Wed Feb 6 09:40:33 2019][1696533.313465] [] ? SYSC_sendto+0x121/0x1c0 [Wed Feb 6 09:40:33 2019][1696533.319217] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:40:33 2019][1696533.325052] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:40:33 2019][1696533.330288] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:33 2019][1696533.336474] crond S ffff91599ff98000 0 30469 19272 0x00000080 [Wed Feb 6 09:40:33 2019][1696533.343751] Call Trace: [Wed Feb 6 09:40:33 2019][1696533.346383] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:40:33 2019][1696533.353092] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1696533.358236] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:40:33 2019][1696533.365371] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:40:33 2019][1696533.371299] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:40:33 2019][1696533.378617] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:33 2019][1696533.385147] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:40:33 2019][1696533.391427] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:40:33 2019][1696533.397005] [] ? skb_copy_datagram_from_iovec+0x5c/0x280 [Wed Feb 6 09:40:33 2019][1696533.404146] [] ? legitimize_mnt+0x16/0x40 [Wed Feb 6 09:40:33 2019][1696533.409988] [] ? skb_queue_tail+0x43/0x50 [Wed Feb 6 09:40:33 2019][1696533.415830] [] ? unix_stream_sendmsg+0x3fd/0x430 [Wed Feb 6 09:40:33 2019][1696533.422278] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:33 2019][1696533.427772] [] ? kern_path+0x60/0xb0 [Wed Feb 6 09:40:33 2019][1696533.433182] [] ? ep_poll_callback+0xee/0x210 [Wed Feb 6 09:40:33 2019][1696533.439286] [] ? SYSC_sendto+0x121/0x1c0 [Wed Feb 6 09:40:33 2019][1696533.445040] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:40:33 2019][1696533.450884] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:40:33 2019][1696533.456112] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:33 2019][1696533.462300] l_getidentity S ffff9159fa65e180 0 30470 28943 0x00000080 [Wed Feb 6 09:40:33 2019][1696533.469576] Call Trace: [Wed Feb 6 09:40:33 2019][1696533.472206] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:40:33 2019][1696533.478906] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1696533.484054] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:40:33 2019][1696533.491195] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:40:33 2019][1696533.497124] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:40:33 2019][1696533.504439] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:33 2019][1696533.510974] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:40:33 2019][1696533.517251] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:40:33 2019][1696533.522829] [] ? skb_copy_datagram_from_iovec+0x5c/0x280 [Wed Feb 6 09:40:33 2019][1696533.529971] [] ? skb_queue_tail+0x43/0x50 [Wed Feb 6 09:40:33 2019][1696533.535814] [] ? unix_stream_sendmsg+0x3fd/0x430 [Wed Feb 6 09:40:33 2019][1696533.542261] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:33 2019][1696533.547761] [] ? do_read_fault.isra.61+0x139/0x1b0 [Wed Feb 6 09:40:33 2019][1696533.554376] [] ? SYSC_sendto+0x121/0x1c0 [Wed Feb 6 09:40:33 2019][1696533.560124] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:40:33 2019][1696533.565964] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:40:33 2019][1696533.571199] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:33 2019][1696533.577379] l_getidentity S ffff9159fa659040 0 30471 28943 0x00000080 [Wed Feb 6 09:40:33 2019][1696533.584674] Call Trace: [Wed Feb 6 09:40:33 2019][1696533.587303] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:40:33 2019][1696533.594012] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1696533.599158] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:40:33 2019][1696533.606292] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:40:33 2019][1696533.612220] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:40:33 2019][1696533.619535] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:33 2019][1696533.626070] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:40:33 2019][1696533.632348] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:40:33 2019][1696533.637927] [] ? skb_copy_datagram_from_iovec+0x5c/0x280 [Wed Feb 6 09:40:33 2019][1696533.645070] [] ? skb_queue_tail+0x43/0x50 [Wed Feb 6 09:40:33 2019][1696533.650909] [] ? unix_stream_sendmsg+0x3fd/0x430 [Wed Feb 6 09:40:33 2019][1696533.657359] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:33 2019][1696533.662856] [] ? do_read_fault.isra.61+0x139/0x1b0 [Wed Feb 6 09:40:33 2019][1696533.669476] [] ? SYSC_sendto+0x121/0x1c0 [Wed Feb 6 09:40:33 2019][1696533.675229] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:40:33 2019][1696533.681073] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:40:33 2019][1696533.686306] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:33 2019][1696533.692492] l_getidentity S ffff9159fa65c100 0 30472 28943 0x00000080 [Wed Feb 6 09:40:33 2019][1696533.699761] Call Trace: [Wed Feb 6 09:40:33 2019][1696533.702389] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Wed Feb 6 09:40:33 2019][1696533.709093] [] schedule+0x29/0x70 [Wed Feb 6 09:40:33 2019][1696533.714243] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Wed Feb 6 09:40:33 2019][1696533.721382] [] ? hrtimer_get_res+0x50/0x50 [Wed Feb 6 09:40:33 2019][1696533.727307] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Wed Feb 6 09:40:33 2019][1696533.734622] [] schedule_hrtimeout_range+0x13/0x20 [Wed Feb 6 09:40:33 2019][1696533.741161] [] poll_schedule_timeout+0x55/0xb0 [Wed Feb 6 09:40:33 2019][1696533.747437] [] do_sys_poll+0x48d/0x590 [Wed Feb 6 09:40:33 2019][1696533.753021] [] ? skb_copy_datagram_from_iovec+0x5c/0x280 [Wed Feb 6 09:40:33 2019][1696533.760168] [] ? skb_queue_tail+0x43/0x50 [Wed Feb 6 09:40:33 2019][1696533.766026] [] ? unix_stream_sendmsg+0x3fd/0x430 [Wed Feb 6 09:40:33 2019][1696533.772482] [] ? __pollwait+0xf0/0xf0 [Wed Feb 6 09:40:33 2019][1696533.777980] [] ? do_read_fault.isra.61+0x139/0x1b0 [Wed Feb 6 09:40:33 2019][1696533.784611] [] ? SYSC_sendto+0x121/0x1c0 [Wed Feb 6 09:40:33 2019][1696533.790361] [] ? ktime_get_ts64+0x52/0xf0 [Wed Feb 6 09:40:33 2019][1696533.796205] [] SyS_poll+0x74/0x110 [Wed Feb 6 09:40:33 2019][1696533.801438] [] system_call_fastpath+0x22/0x27 [Wed Feb 6 09:40:33 2019][1696533.807634] Sched Debug Version: v0.11, 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 09:40:33 2019][1696533.814775] ktime : 1696490392.319735 [Wed Feb 6 09:40:33 2019][1696533.821572] sched_clk : 1696533807.633243 [Wed Feb 6 09:40:33 2019][1696533.828366] cpu_clk : 1696533807.633273 [Wed Feb 6 09:40:33 2019][1696533.835161] jiffies : 5991157716 [Wed Feb 6 09:40:33 2019][1696533.841346] sched_clock_stable() : 1 [Wed Feb 6 09:40:33 2019][1696533.846754] [Wed Feb 6 09:40:33 2019][1696533.848436] sysctl_sched [Wed Feb 6 09:40:33 2019][1696533.851157] .sysctl_sched_latency : 24.000000 [Wed Feb 6 09:40:33 2019][1696533.857518] .sysctl_sched_min_granularity : 10.000000 [Wed Feb 6 09:40:33 2019][1696533.863871] .sysctl_sched_wakeup_granularity : 4.000000 [Wed Feb 6 09:40:34 2019][1696533.870147] .sysctl_sched_child_runs_first : 0 [Wed Feb 6 09:40:34 2019][1696533.875814] .sysctl_sched_features : 56955 [Wed Feb 6 09:40:34 2019][1696533.881833] .sysctl_sched_tunable_scaling : 1 (logaritmic) [Wed Feb 6 09:40:34 2019][1696533.888627] [Wed Feb 6 09:40:34 2019][1696533.890307] cpu#0, 1996.199 MHz [Wed Feb 6 09:40:34 2019][1696533.893637] .nr_running : 0 [Wed Feb 6 09:40:34 2019][1696533.898433] .load : 0 [Wed Feb 6 09:40:34 2019][1696533.903229] .nr_switches : 4222411519 [Wed Feb 6 09:40:34 2019][1696533.908810] .nr_load_updates : 503377273 [Wed Feb 6 09:40:34 2019][1696533.914304] .nr_uninterruptible : 548517 [Wed Feb 6 09:40:34 2019][1696533.919538] .next_balance : 5991.157804 [Wed Feb 6 09:40:34 2019][1696533.925211] .curr->pid : 0 [Wed Feb 6 09:40:34 2019][1696533.930012] .clock : 1696533929.423345 [Wed Feb 6 09:40:34 2019][1696533.936201] .cpu_load[0] : 0 [Wed Feb 6 09:40:34 2019][1696533.940996] .cpu_load[1] : 0 [Wed Feb 6 09:40:34 2019][1696533.945789] .cpu_load[2] : 0 [Wed Feb 6 09:40:34 2019][1696533.950582] .cpu_load[3] : 0 [Wed Feb 6 09:40:34 2019][1696533.955375] .cpu_load[4] : 0 [Wed Feb 6 09:40:34 2019][1696533.960172] .avg_idle : 2072343 [Wed Feb 6 09:40:34 2019][1696533.965489] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:34 2019][1696533.970806] [Wed Feb 6 09:40:34 2019][1696533.972482] cfs_rq[0]:/ [Wed Feb 6 09:40:34 2019][1696533.975106] .exec_clock : 0.000000 [Wed Feb 6 09:40:34 2019][1696533.980505] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:34 2019][1696533.985903] .min_vruntime : 28125570.196552 [Wed Feb 6 09:40:34 2019][1696533.991911] .max_vruntime : 0.000001 [Wed Feb 6 09:40:34 2019][1696533.997309] .spread : 0.000000 [Wed Feb 6 09:40:34 2019][1696534.002710] .spread0 : 0.000000 [Wed Feb 6 09:40:34 2019][1696534.008109] .nr_spread_over : 0 [Wed Feb 6 09:40:34 2019][1696534.012901] .nr_running : 0 [Wed Feb 6 09:40:34 2019][1696534.017695] .load : 0 [Wed Feb 6 09:40:34 2019][1696534.022486] .runnable_load_avg : 0 [Wed Feb 6 09:40:34 2019][1696534.027280] .blocked_load_avg : 0 [Wed Feb 6 09:40:34 2019][1696534.032074] .tg_load_avg : 0 [Wed Feb 6 09:40:34 2019][1696534.036867] .tg_load_contrib : 0 [Wed Feb 6 09:40:34 2019][1696534.041657] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:34 2019][1696534.046451] .tg->runnable_avg : 0 [Wed Feb 6 09:40:34 2019][1696534.051244] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:34 2019][1696534.056036] .throttled : 0 [Wed Feb 6 09:40:34 2019][1696534.060830] .throttle_count : 0 [Wed Feb 6 09:40:34 2019][1696534.065621] .avg->runnable_avg_sum : 103 [Wed Feb 6 09:40:34 2019][1696534.070588] .avg->runnable_avg_period : 47315 [Wed Feb 6 09:40:34 2019][1696534.075737] [Wed Feb 6 09:40:34 2019][1696534.077410] rt_rq[0]:/ [Wed Feb 6 09:40:34 2019][1696534.079948] .rt_nr_running : 0 [Wed Feb 6 09:40:34 2019][1696534.084742] .rt_throttled : 0 [Wed Feb 6 09:40:34 2019][1696534.089533] .rt_time : 0.000000 [Wed Feb 6 09:40:34 2019][1696534.094934] .rt_runtime : 982.997715 [Wed Feb 6 09:40:34 2019][1696534.100507] [Wed Feb 6 09:40:34 2019][1696534.102178] runnable tasks: [Wed Feb 6 09:40:34 2019][1696534.105150] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:34 2019][1696534.116097] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:34 2019][1696534.126958] ksoftirqd/0 3 28125390.046025 12377353 120 0.000000 136851.374194 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.138293] kworker/0:0H 5 5293.750568 8 100 0.000000 0.028064 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.149630] migration/0 8 0.000000 3402 0 0.000000 602297.459112 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.160966] rcu_bh 9 513.633045 2 120 0.000000 0.002045 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.172302] rcu_sched 10 28125567.150505 218587805 120 0.000000 2941587.236153 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.183639] lru-add-drain 11 517.635053 2 100 0.000000 0.002545 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.194975] watchdog/0 12 0.000000 424090 0 0.000000 145233.916582 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.206346] irq/26-AMD-Vi 268 0.000000 3 49 0.000000 0.042521 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.217682] kswapd0 276 20390107.225654 339 120 0.000000 459.609799 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.229023] scsi_eh_0 2282 3523.086309 2 120 0.000000 0.031960 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.240361] kworker/u262:0 7440 4562.640519 4 100 0.000000 0.015480 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.251702] kworker/0:1H 7529 28125389.900986 11799702 100 0.000000 49700.986967 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.263075] kiblnd_sd_00_01 5876 28125570.533796 4154605887 120 0.000000 27485478.182249 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.274500] ptlrpcd_00_06 5902 28125556.109269 6012173 120 0.000000 93944.376342 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.285856] mdt00_001 6168 27579456.227270 234608048 120 0.000000 7815688.496953 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.297198] mdt_seqm_0001 6206 27953933.962392 58 120 0.000000 1.212133 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.308540] osp-syn-33-0 6300 28081641.235660 789050 120 0.000000 20842.655998 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.319904] ldlm_cn00_006 6777 28125495.126527 12480774 120 0.000000 579283.596030 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.331254] mdt_io00_014 8974 27955671.161049 6465576 120 0.000000 256771.203199 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.342601] ldlm_cn00_009 9226 28125558.624150 15746508 120 0.000000 628016.166106 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.353938] ldlm_cn00_015 14886 27954222.528430 12090912 120 0.000000 617546.051065 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.365295] kworker/0:1 77924 28125554.983387 155326 120 0.000000 9486.776609 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.376654] ldlm_bl_26 15989 28125501.444715 2324805 120 0.000000 19651.314187 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.387994] mdt_rdpg00_011 17451 27140409.486969 2645696 120 0.000000 97323.569530 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.399338] kworker/0:2 27671 24817112.122116 5 120 0.000000 0.034506 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.410677] mdt_io00_041 27675 24817208.487676 174 120 0.000000 6.561695 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.422029] mdt_io00_064 27783 24817219.243855 4 120 0.000000 0.059244 0.000000 0 / [Wed Feb 6 09:40:34 2019][1696534.433406] [Wed Feb 6 09:40:34 2019][1696534.435081] cpu#1, 1996.199 MHz [Wed Feb 6 09:40:34 2019][1696534.438408] .nr_running : 0 [Wed Feb 6 09:40:34 2019][1696534.443204] .load : 0 [Wed Feb 6 09:40:34 2019][1696534.448006] .nr_switches : 2808094256 [Wed Feb 6 09:40:34 2019][1696534.453585] .nr_load_updates : 434276718 [Wed Feb 6 09:40:34 2019][1696534.459081] .nr_uninterruptible : -153626 [Wed Feb 6 09:40:34 2019][1696534.464405] .next_balance : 5991.158348 [Wed Feb 6 09:40:34 2019][1696534.470067] .curr->pid : 0 [Wed Feb 6 09:40:34 2019][1696534.474860] .clock : 1696534474.331904 [Wed Feb 6 09:40:34 2019][1696534.481049] .cpu_load[0] : 0 [Wed Feb 6 09:40:34 2019][1696534.485855] .cpu_load[1] : 0 [Wed Feb 6 09:40:34 2019][1696534.490655] .cpu_load[2] : 0 [Wed Feb 6 09:40:34 2019][1696534.495455] .cpu_load[3] : 0 [Wed Feb 6 09:40:34 2019][1696534.500259] .cpu_load[4] : 0 [Wed Feb 6 09:40:34 2019][1696534.505055] .avg_idle : 3135293 [Wed Feb 6 09:40:34 2019][1696534.510369] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:34 2019][1696534.515691] [Wed Feb 6 09:40:34 2019][1696534.517367] cfs_rq[1]:/ [Wed Feb 6 09:40:34 2019][1696534.519992] .exec_clock : 0.000000 [Wed Feb 6 09:40:34 2019][1696534.525392] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:34 2019][1696534.530792] .min_vruntime : 19581022.310706 [Wed Feb 6 09:40:34 2019][1696534.536798] .max_vruntime : 0.000001 [Wed Feb 6 09:40:34 2019][1696534.542197] .spread : 0.000000 [Wed Feb 6 09:40:34 2019][1696534.547596] .spread0 : -8544548.588470 [Wed Feb 6 09:40:34 2019][1696534.553603] .nr_spread_over : 0 [Wed Feb 6 09:40:34 2019][1696534.558397] .nr_running : 0 [Wed Feb 6 09:40:34 2019][1696534.563188] .load : 0 [Wed Feb 6 09:40:34 2019][1696534.567980] .runnable_load_avg : 0 [Wed Feb 6 09:40:34 2019][1696534.572775] .blocked_load_avg : 0 [Wed Feb 6 09:40:34 2019][1696534.577567] .tg_load_avg : 0 [Wed Feb 6 09:40:34 2019][1696534.582360] .tg_load_contrib : 0 [Wed Feb 6 09:40:34 2019][1696534.587152] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:34 2019][1696534.591946] .tg->runnable_avg : 0 [Wed Feb 6 09:40:34 2019][1696534.596737] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:34 2019][1696534.601532] .throttled : 0 [Wed Feb 6 09:40:34 2019][1696534.606322] .throttle_count : 0 [Wed Feb 6 09:40:34 2019][1696534.611117] .avg->runnable_avg_sum : 73 [Wed Feb 6 09:40:34 2019][1696534.615996] .avg->runnable_avg_period : 46229 [Wed Feb 6 09:40:34 2019][1696534.621144] [Wed Feb 6 09:40:34 2019][1696534.622818] rt_rq[1]:/ [Wed Feb 6 09:40:34 2019][1696534.625355] .rt_nr_running : 0 [Wed Feb 6 09:40:34 2019][1696534.630150] .rt_throttled : 0 [Wed Feb 6 09:40:34 2019][1696534.634942] .rt_time : 0.000000 [Wed Feb 6 09:40:34 2019][1696534.640340] .rt_runtime : 634.639237 [Wed Feb 6 09:40:34 2019][1696534.645915] [Wed Feb 6 09:40:34 2019][1696534.647587] runnable tasks: [Wed Feb 6 09:40:34 2019][1696534.650559] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:34 2019][1696534.661506] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:34 2019][1696534.672367] watchdog/1 13 -2.962990 424068 0 0.000000 156226.458148 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.683702] migration/1 14 0.000000 2506 0 0.000000 566894.638011 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.695037] ksoftirqd/1 15 19580981.593085 17554333 120 0.000000 170981.085526 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.706375] kworker/1:0H 17 6764.933380 10 100 0.000000 0.039033 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.717739] writeback 258 19392964.994993 3 100 0.000000 42.522634 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.729080] scsi_eh_2 7338 2304.439178 22 120 0.000000 5.646079 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.740420] ib_mad1 7482 19237139.559770 3 100 0.000000 0.017764 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.751757] ccp-2-q4 10557 2925.655839 2 120 0.000000 0.013447 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.763104] dsm_sa_datamgrd 18343 6776.054770 1778 120 0.000000 1315.203006 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.774444] dsm_sa_datamgrd 18869 19581009.557884 12526143 120 0.000000 325784.684415 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.785780] dsm_sa_datamgrd 18872 19581009.547826 2101950 120 0.000000 40819.053365 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.797117] kworker/1:1H 18991 19580970.321717 13462835 100 0.000000 62301.753698 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.808474] kiblnd_sd_01_02 5881 19581022.736572 2884831008 120 0.000000 21186259.003175 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.819897] ptlrpcd_01_01 5909 19581009.758185 7003452 120 0.000000 98870.626209 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.831230] ptlrpcd_01_07 5915 19581004.549952 7013043 120 0.000000 107395.737546 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.842572] ldlm_cb01_000 6022 19578917.993760 14049428 120 0.000000 259980.067815 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.853914] ldlm_cn01_003 6071 19562407.725183 40052710 120 0.000000 1009685.046612 0.000000 1 / [Wed Feb 6 09:40:34 2019][1696534.865252] mdt_attr01_000 6189 185090.912814 3 120 0.000000 0.064583 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696534.876583] mdt_attr01_001 6190 185090.937441 3 120 0.000000 0.034114 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696534.887921] mdt_out01_001 6198 19581007.827757 28379065 120 0.000000 556520.507201 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696534.899260] osp-pre-26-0 6263 19581004.495154 247836 120 0.000000 4832.351450 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696534.910607] osp-pre-25-2 6469 19581009.736082 234953 120 0.000000 6883.835548 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696534.921953] lquota_wb_fir-M 6595 19580968.843316 17159 120 0.000000 94.571651 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696534.933293] mdt01_005 6683 19581004.188523 112880547 120 0.000000 4529069.261355 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696534.944629] mdt_io01_003 6774 19393290.278598 5818741 120 0.000000 225650.818179 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696534.955969] mdt_rdpg01_002 6870 19581002.967697 9798910 120 0.000000 426551.924144 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696534.967309] mdt_fld_0002 7088 19580859.908166 3141 120 0.000000 99.981211 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696534.978643] ldlm_cb01_002 7091 15923409.886337 17198723 120 0.000000 317407.183489 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696534.989989] mdt01_017 8970 19581004.226225 115000684 120 0.000000 4706086.149407 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.001324] mdt01_018 8971 19580945.849350 109316645 120 0.000000 4767727.526879 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.012666] mdt01_029 9013 19581004.109974 112991200 120 0.000000 4296467.717915 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.023999] mdt01_035 9019 19581013.389880 88898327 120 0.000000 3679892.681881 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.035338] mdt_io01_029 9840 19580940.694948 9041458 120 0.000000 326829.244900 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.046674] mdt_io01_031 14973 18867120.207234 6416845 120 0.000000 245104.005604 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.058021] ldlm_cn01_017 23315 19578227.775517 24239060 120 0.000000 727234.581439 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.069353] ldlm_cn01_019 23323 19523273.515129 10937467 120 0.000000 475412.554627 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.080690] ldlm_cn01_023 28088 19578227.352159 16933387 120 0.000000 525926.788222 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.092023] ldlm_cn01_024 28325 19562416.333955 10268831 120 0.000000 345237.894762 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.103359] ldlm_cn01_025 28326 19562405.599063 15076372 120 0.000000 421330.309668 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.114704] kworker/1:1 81186 19581001.238853 208564 120 0.000000 14374.922132 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.126045] mdt01_043 15170 19581004.115634 4416623 120 0.000000 752519.787184 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.137382] mdt01_052 15239 19561371.083598 9703060 120 0.000000 858010.871073 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.148720] mdt01_068 15382 19581010.157164 5709466 120 0.000000 367072.565742 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.160050] mdt01_075 15392 19581008.001521 7161497 120 0.000000 459242.109426 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.171393] ldlm_bl_18 15975 19580931.545770 2330615 120 0.000000 19114.116749 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.182751] mdt_io01_093 27704 19393289.666454 897 120 0.000000 35.184322 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.194089] mdt_io01_102 27744 19393290.384409 19994 120 0.000000 1232.020006 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.205435] ll_mgs_0008 28965 19418690.226032 4526 120 0.000000 108.645903 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.216777] kworker/1:2 30286 19579554.692685 6 120 0.000000 0.057750 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.228112] kworker/1:0 30400 19579566.693175 6 120 0.000000 0.032221 0.000000 1 / [Wed Feb 6 09:40:35 2019][1696535.239468] [Wed Feb 6 09:40:35 2019][1696535.241146] cpu#2, 1996.199 MHz [Wed Feb 6 09:40:35 2019][1696535.244475] .nr_running : 1 [Wed Feb 6 09:40:35 2019][1696535.249275] .load : 1024 [Wed Feb 6 09:40:35 2019][1696535.254351] .nr_switches : 1896961155 [Wed Feb 6 09:40:35 2019][1696535.259938] .nr_load_updates : 360456923 [Wed Feb 6 09:40:35 2019][1696535.265432] .nr_uninterruptible : -41659 [Wed Feb 6 09:40:35 2019][1696535.270665] .next_balance : 5991.159162 [Wed Feb 6 09:40:35 2019][1696535.276329] .curr->pid : 15988 [Wed Feb 6 09:40:35 2019][1696535.281475] .clock : 1696535281.352817 [Wed Feb 6 09:40:35 2019][1696535.287664] .cpu_load[0] : 1024 [Wed Feb 6 09:40:35 2019][1696535.292723] .cpu_load[1] : 1024 [Wed Feb 6 09:40:35 2019][1696535.297783] .cpu_load[2] : 1024 [Wed Feb 6 09:40:35 2019][1696535.302844] .cpu_load[3] : 1024 [Wed Feb 6 09:40:35 2019][1696535.307907] .cpu_load[4] : 1024 [Wed Feb 6 09:40:35 2019][1696535.312968] .avg_idle : 10000000 [Wed Feb 6 09:40:35 2019][1696535.318376] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:35 2019][1696535.323700] [Wed Feb 6 09:40:35 2019][1696535.325372] cfs_rq[2]:/ [Wed Feb 6 09:40:35 2019][1696535.327999] .exec_clock : 0.000000 [Wed Feb 6 09:40:35 2019][1696535.333399] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:35 2019][1696535.338797] .min_vruntime : 14455287.972618 [Wed Feb 6 09:40:35 2019][1696535.344804] .max_vruntime : 0.000001 [Wed Feb 6 09:40:35 2019][1696535.350202] .spread : 0.000000 [Wed Feb 6 09:40:35 2019][1696535.355601] .spread0 : -13670284.089093 [Wed Feb 6 09:40:35 2019][1696535.361694] .nr_spread_over : 0 [Wed Feb 6 09:40:35 2019][1696535.366488] .nr_running : 1 [Wed Feb 6 09:40:35 2019][1696535.371280] .load : 1024 [Wed Feb 6 09:40:35 2019][1696535.376333] .runnable_load_avg : 1023 [Wed Feb 6 09:40:35 2019][1696535.381386] .blocked_load_avg : 0 [Wed Feb 6 09:40:35 2019][1696535.386178] .tg_load_avg : 0 [Wed Feb 6 09:40:35 2019][1696535.390971] .tg_load_contrib : 0 [Wed Feb 6 09:40:35 2019][1696535.395765] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:35 2019][1696535.400558] .tg->runnable_avg : 0 [Wed Feb 6 09:40:35 2019][1696535.405349] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:35 2019][1696535.410142] .throttled : 0 [Wed Feb 6 09:40:35 2019][1696535.414934] .throttle_count : 0 [Wed Feb 6 09:40:35 2019][1696535.419729] .avg->runnable_avg_sum : 46925 [Wed Feb 6 09:40:35 2019][1696535.424868] .avg->runnable_avg_period : 46755 [Wed Feb 6 09:40:35 2019][1696535.430013] [Wed Feb 6 09:40:35 2019][1696535.431689] rt_rq[2]:/ [Wed Feb 6 09:40:35 2019][1696535.434228] .rt_nr_running : 0 [Wed Feb 6 09:40:35 2019][1696535.439021] .rt_throttled : 0 [Wed Feb 6 09:40:35 2019][1696535.443812] .rt_time : 0.000000 [Wed Feb 6 09:40:35 2019][1696535.449212] .rt_runtime : 908.687510 [Wed Feb 6 09:40:35 2019][1696535.454785] [Wed Feb 6 09:40:35 2019][1696535.456456] runnable tasks: [Wed Feb 6 09:40:35 2019][1696535.459431] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:35 2019][1696535.470376] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:35 2019][1696535.481240] watchdog/2 19 -5.960495 424092 0 0.000000 139231.494547 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.492573] migration/2 20 0.000000 2390 0 0.000000 516935.257908 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.503910] ksoftirqd/2 21 14386028.479704 9151913 120 0.000000 101157.460966 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.515244] kworker/2:0H 23 4824.612130 9 100 0.000000 0.032612 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.526612] irq/28-AMD-Vi 270 0.000000 3 49 0.000000 0.030768 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.537944] kswapd2 278 11721195.148466 8 120 0.000000 0.652149 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.549299] kworker/2:1H 16789 14386028.273056 7532616 100 0.000000 31849.776317 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.560674] ldlm_cb02_001 6026 11403336.651315 11176517 120 0.000000 201556.983145 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.572014] mdt_attr02_000 6191 167920.009574 3 120 0.000000 0.061537 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.583350] mdt_attr02_001 6192 167920.029552 3 120 0.000000 0.028995 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.594707] osp-syn-11-2 6448 14386028.311629 433649 120 0.000000 12372.005148 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.606066] mdt_out02_002 7090 13870684.166140 8571681 120 0.000000 198524.326011 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.617406] mdt02_011 8829 13864700.108774 92058608 120 0.000000 4613440.913944 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.628741] mdt_io02_013 8962 14286340.499067 6271186 120 0.000000 244752.171931 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.640085] mdt02_023 9199 13864700.108925 87329216 120 0.000000 3924393.956846 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.651420] mdt_io02_022 9201 13425521.404276 6232037 120 0.000000 250169.238135 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.662779] kworker/2:0 47521 14386028.406573 194841 120 0.000000 11182.058426 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.674128] mdt02_035 15249 13864700.108674 11895867 120 0.000000 668091.096921 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.685464] mdt02_040 15268 14310206.883533 7704555 120 0.000000 645255.451586 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.696801] mdt02_046 15372 13864700.109004 8961840 120 0.000000 568514.542790 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.708139] mdt_io02_037 15455 13859057.208369 300 120 0.000000 11.025374 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.719472] mdt02_047 15509 13862832.430101 9153962 120 0.000000 997800.228621 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.730811] R ldlm_bl_25 15988 14455680.824172 2323911 120 0.000000 5944718.479424 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.742164] mdt_io02_054 27724 14075762.056781 2922 120 0.000000 2846.931609 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.753503] mdt_io02_073 27764 13425484.393906 44 120 0.000000 1.990491 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.764836] mdt_io02_078 27769 13425488.541197 23 120 0.000000 1.148714 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.776173] ldlm_cn02_027 28523 14310192.818387 39525 120 0.000000 7896.564059 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.787515] ll_mgs_0027 28995 13870760.858879 128 120 0.000000 28.031106 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.798855] kworker/2:2 29774 14095123.917956 2 120 0.000000 0.007975 0.000000 2 / [Wed Feb 6 09:40:35 2019][1696535.810211] [Wed Feb 6 09:40:35 2019][1696535.811893] cpu#3, 1996.199 MHz [Wed Feb 6 09:40:35 2019][1696535.815225] .nr_running : 3 [Wed Feb 6 09:40:35 2019][1696535.820025] .load : 88761 [Wed Feb 6 09:40:35 2019][1696535.825174] .nr_switches : 635952775 [Wed Feb 6 09:40:35 2019][1696535.830669] .nr_load_updates : 218819687 [Wed Feb 6 09:40:35 2019][1696535.836159] .nr_uninterruptible : -24531 [Wed Feb 6 09:40:35 2019][1696535.841388] .next_balance : 5991.159838 [Wed Feb 6 09:40:35 2019][1696535.847058] .curr->pid : 26 [Wed Feb 6 09:40:35 2019][1696535.851943] .clock : 1696535851.367170 [Wed Feb 6 09:40:35 2019][1696535.858130] .cpu_load[0] : 88761 [Wed Feb 6 09:40:35 2019][1696535.863280] .cpu_load[1] : 88761 [Wed Feb 6 09:40:35 2019][1696535.868430] .cpu_load[2] : 88761 [Wed Feb 6 09:40:35 2019][1696535.873578] .cpu_load[3] : 88761 [Wed Feb 6 09:40:36 2019][1696535.878724] .cpu_load[4] : 88761 [Wed Feb 6 09:40:36 2019][1696535.883874] .avg_idle : 2238532 [Wed Feb 6 09:40:36 2019][1696535.889191] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:36 2019][1696535.894510] [Wed Feb 6 09:40:36 2019][1696535.896182] cfs_rq[3]:/ [Wed Feb 6 09:40:36 2019][1696535.898807] .exec_clock : 0.000000 [Wed Feb 6 09:40:36 2019][1696535.904206] .MIN_vruntime : 38178252.966059 [Wed Feb 6 09:40:36 2019][1696535.910213] .min_vruntime : 38178264.966059 [Wed Feb 6 09:40:36 2019][1696535.916219] .max_vruntime : 38178252.966059 [Wed Feb 6 09:40:36 2019][1696535.922225] .spread : 0.000000 [Wed Feb 6 09:40:36 2019][1696535.927624] .spread0 : 10052692.387979 [Wed Feb 6 09:40:36 2019][1696535.933631] .nr_spread_over : 0 [Wed Feb 6 09:40:36 2019][1696535.938425] .nr_running : 1 [Wed Feb 6 09:40:36 2019][1696535.943216] .load : 88761 [Wed Feb 6 09:40:36 2019][1696535.948357] .runnable_load_avg : 0 [Wed Feb 6 09:40:36 2019][1696535.953149] .blocked_load_avg : 0 [Wed Feb 6 09:40:36 2019][1696535.957942] .tg_load_avg : 0 [Wed Feb 6 09:40:36 2019][1696535.962735] .tg_load_contrib : 0 [Wed Feb 6 09:40:36 2019][1696535.967528] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:36 2019][1696535.972320] .tg->runnable_avg : 0 [Wed Feb 6 09:40:36 2019][1696535.977111] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:36 2019][1696535.981906] .throttled : 0 [Wed Feb 6 09:40:36 2019][1696535.986715] .throttle_count : 0 [Wed Feb 6 09:40:36 2019][1696535.991508] .avg->runnable_avg_sum : 47433 [Wed Feb 6 09:40:36 2019][1696535.996647] .avg->runnable_avg_period : 47433 [Wed Feb 6 09:40:36 2019][1696536.001792] [Wed Feb 6 09:40:36 2019][1696536.003460] rt_rq[3]:/ [Wed Feb 6 09:40:36 2019][1696536.006000] .rt_nr_running : 1 [Wed Feb 6 09:40:36 2019][1696536.010792] .rt_throttled : 0 [Wed Feb 6 09:40:36 2019][1696536.015585] .rt_time : 0.000000 [Wed Feb 6 09:40:36 2019][1696536.020984] .rt_runtime : 688.284537 [Wed Feb 6 09:40:36 2019][1696536.026557] [Wed Feb 6 09:40:36 2019][1696536.028229] runnable tasks: [Wed Feb 6 09:40:36 2019][1696536.031202] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:36 2019][1696536.042148] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:36 2019][1696536.053012] watchdog/3 25 -5.970504 424079 0 0.000000 175536.071777 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.064346] R migration/3 26 0.000000 2721 0 0.000000 511932.722932 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.075680] ksoftirqd/3 27 38178215.402118 4289273 120 0.000000 40460.151050 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.087018] kworker/3:0H 29 4766.746322 9 100 0.000000 0.040406 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.098385] irq/29-AMD-Vi 271 0.000000 3 49 0.000000 0.034567 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.109725] kswapd3 279 35200722.512560 415 120 0.000000 810.978069 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.121061] kworker/3:1 293 38178252.980336 384159 120 0.000000 12668.020012 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.132404] kworker/u265:0 7443 1743.130161 4 100 0.000000 0.016882 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.143747] ccp-4-q1 10661 2499.558404 2 120 0.000000 0.006934 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.155079] ccp-4-q2 10666 2523.564349 2 120 0.000000 0.007154 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.166419] kworker/3:1H 16771 38178252.966059 5071471 100 0.000000 17360.509896 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.177757] mdadm 17554 38175218.258602 1699 120 0.000000 138.299871 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.189138] mdt03_002 6178 38082720.015975 35044701 120 0.000000 1862192.188427 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.200476] mdt_attr03_000 6193 305336.421548 3 120 0.000000 0.067539 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.211812] mdt_attr03_001 6194 305336.444061 3 120 0.000000 0.031870 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.223153] osp-syn-12-0 6248 38175163.168070 781150 120 0.000000 20550.131234 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.234487] osp-syn-0-0 6282 38175162.282012 790622 120 0.000000 20888.295717 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.245820] osp-syn-32-0 6288 38175920.012465 775672 120 0.000000 20942.253565 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.257191] mdt_io03_006 7248 38149735.277345 1978429 120 0.000000 74257.759986 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.268529] mdt03_011 8573 38102105.598211 35995625 120 0.000000 1748748.314037 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.279896] ldlm_cn03_008 23309 38154021.835390 5509100 120 0.000000 364019.737804 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.291238] ldlm_cn03_014 63239 38154021.835832 6336900 120 0.000000 264279.375594 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.302595] mdt03_019 17204 38086492.191451 8559193 120 0.000000 833897.131782 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.313941] mdt_io03_025 27459 37953650.617216 190 120 0.000000 1136.147975 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.325279] mdt_io03_026 27460 38172054.923899 16760 120 0.000000 584.167617 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.336636] mdt_io03_030 28960 37953715.891319 12 120 0.000000 0.568719 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.347972] ll_mgs_0023 28991 37965885.586394 111 120 0.000000 40.335948 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.359310] kworker/3:0 29739 38145202.720708 11 120 0.000000 0.070424 0.000000 3 / [Wed Feb 6 09:40:36 2019][1696536.370662] [Wed Feb 6 09:40:36 2019][1696536.372339] cpu#4, 1996.199 MHz [Wed Feb 6 09:40:36 2019][1696536.375668] .nr_running : 0 [Wed Feb 6 09:40:36 2019][1696536.380470] .load : 0 [Wed Feb 6 09:40:36 2019][1696536.385270] .nr_switches : 4164952517 [Wed Feb 6 09:40:36 2019][1696536.390852] .nr_load_updates : 533570007 [Wed Feb 6 09:40:36 2019][1696536.396346] .nr_uninterruptible : 476807 [Wed Feb 6 09:40:36 2019][1696536.401582] .next_balance : 5991.160287 [Wed Feb 6 09:40:36 2019][1696536.407246] .curr->pid : 0 [Wed Feb 6 09:40:36 2019][1696536.412046] .clock : 1696536411.381056 [Wed Feb 6 09:40:36 2019][1696536.418231] .cpu_load[0] : 0 [Wed Feb 6 09:40:36 2019][1696536.423029] .cpu_load[1] : 0 [Wed Feb 6 09:40:36 2019][1696536.427822] .cpu_load[2] : 0 [Wed Feb 6 09:40:36 2019][1696536.432623] .cpu_load[3] : 0 [Wed Feb 6 09:40:36 2019][1696536.437420] .cpu_load[4] : 0 [Wed Feb 6 09:40:36 2019][1696536.442222] .avg_idle : 4952091 [Wed Feb 6 09:40:36 2019][1696536.447542] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:36 2019][1696536.452861] [Wed Feb 6 09:40:36 2019][1696536.454538] cfs_rq[4]:/ [Wed Feb 6 09:40:36 2019][1696536.457163] .exec_clock : 0.000000 [Wed Feb 6 09:40:36 2019][1696536.462562] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:36 2019][1696536.467962] .min_vruntime : 27107011.663681 [Wed Feb 6 09:40:36 2019][1696536.473968] .max_vruntime : 0.000001 [Wed Feb 6 09:40:36 2019][1696536.479368] .spread : 0.000000 [Wed Feb 6 09:40:36 2019][1696536.484767] .spread0 : -1018561.454604 [Wed Feb 6 09:40:36 2019][1696536.490773] .nr_spread_over : 0 [Wed Feb 6 09:40:36 2019][1696536.495567] .nr_running : 0 [Wed Feb 6 09:40:36 2019][1696536.500358] .load : 0 [Wed Feb 6 09:40:36 2019][1696536.505151] .runnable_load_avg : 0 [Wed Feb 6 09:40:36 2019][1696536.509945] .blocked_load_avg : 0 [Wed Feb 6 09:40:36 2019][1696536.514739] .tg_load_avg : 0 [Wed Feb 6 09:40:36 2019][1696536.519530] .tg_load_contrib : 0 [Wed Feb 6 09:40:36 2019][1696536.524322] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:36 2019][1696536.529116] .tg->runnable_avg : 0 [Wed Feb 6 09:40:36 2019][1696536.533908] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:36 2019][1696536.538701] .throttled : 0 [Wed Feb 6 09:40:36 2019][1696536.543494] .throttle_count : 0 [Wed Feb 6 09:40:36 2019][1696536.548287] .avg->runnable_avg_sum : 35 [Wed Feb 6 09:40:36 2019][1696536.553167] .avg->runnable_avg_period : 47704 [Wed Feb 6 09:40:36 2019][1696536.558310] [Wed Feb 6 09:40:36 2019][1696536.559978] rt_rq[4]:/ [Wed Feb 6 09:40:36 2019][1696536.562516] .rt_nr_running : 0 [Wed Feb 6 09:40:36 2019][1696536.567310] .rt_throttled : 0 [Wed Feb 6 09:40:36 2019][1696536.572102] .rt_time : 0.000000 [Wed Feb 6 09:40:36 2019][1696536.577502] .rt_runtime : 849.012089 [Wed Feb 6 09:40:36 2019][1696536.583075] [Wed Feb 6 09:40:36 2019][1696536.584748] runnable tasks: [Wed Feb 6 09:40:36 2019][1696536.587720] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:36 2019][1696536.598668] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:36 2019][1696536.609529] watchdog/4 31 -8.964052 424075 0 0.000000 164561.271410 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.620863] migration/4 32 0.000000 2603 0 0.000000 524525.557157 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.632198] ksoftirqd/4 33 27106957.786206 13177472 120 0.000000 137102.037876 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.643535] kworker/4:0H 35 5402.609890 10 100 0.000000 0.053060 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.654910] ib-comp-unb-wq 7467 26406229.415696 4 100 0.000000 0.058742 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.666267] kworker/4:1H 28018 27106957.648752 13543919 100 0.000000 59419.735503 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.677613] kiblnd_sd_00_00 5875 27107011.879873 4155050213 120 0.000000 27449917.765545 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.689041] ptlrpcd_00_07 5903 27106997.684215 6069537 120 0.000000 87546.278551 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.700396] osp_up2-0 6241 27106884.808037 353 120 0.000000 10.718731 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.711735] osp-syn-11-0 6258 27106325.669352 771664 120 0.000000 20732.864045 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.723072] osp-syn-43-0 6272 27106300.169927 778395 120 0.000000 20858.861072 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.734410] osp-syn-23-0 6298 27106302.901210 783783 120 0.000000 21155.290401 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.745755] osp-syn-40-2 6496 27106936.468855 436466 120 0.000000 12644.208623 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.757099] mdt_rdpg00_002 6781 27058385.378786 76586522 120 0.000000 1650180.178775 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.768441] mdt_io00_011 7215 27060820.190215 5017348 120 0.000000 196232.944693 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.779780] ldlm_cn00_007 7739 27105124.153038 15297007 120 0.000000 559767.630977 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.791118] mdt_io00_013 8961 27060759.693298 5567252 120 0.000000 231090.055177 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.802459] ldlm_cn00_008 9139 27105130.651433 11096142 120 0.000000 557334.991807 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.813801] mdt_out00_004 10654 20883858.141453 45883563 120 0.000000 1081956.913427 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.825137] ldlm_cn00_019 19851 27100213.378291 13893481 120 0.000000 610301.233210 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.836469] mdt_rdpg00_006 19913 27071674.248490 57888265 120 0.000000 1316688.616717 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.847807] ldlm_cn00_024 20043 27106891.603588 10813219 120 0.000000 578469.194901 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.859146] ldlm_cn00_029 21239 27105132.203503 9729525 120 0.000000 488352.191891 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.870492] kworker/4:2 8526 27106992.789311 58003 120 0.000000 9761.801935 0.000000 0 / [Wed Feb 6 09:40:36 2019][1696536.881838] mdt00_042 15279 27087738.060712 9881906 120 0.000000 682402.712947 0.000000 0 / [Wed Feb 6 09:40:37 2019][1696536.893178] mdt00_045 15386 27106935.332586 11796990 120 0.000000 926787.312091 0.000000 0 / [Wed Feb 6 09:40:37 2019][1696536.904517] mdt_io00_030 15458 27060801.836692 231837 120 0.000000 18020.388131 0.000000 0 / [Wed Feb 6 09:40:37 2019][1696536.915850] mdt_io00_036 15467 27060832.140852 297818 120 0.000000 16105.841318 0.000000 0 / [Wed Feb 6 09:40:37 2019][1696536.927186] ldlm_bl_15 15972 27106922.979076 2324346 120 0.000000 326128.076299 0.000000 0 / [Wed Feb 6 09:40:37 2019][1696536.938521] kworker/u257:2 16011 27106978.130460 4786 120 0.000000 3013.774559 0.000000 0 / [Wed Feb 6 09:40:37 2019][1696536.949863] mdt_rdpg00_012 17453 27106997.856972 251329 120 0.000000 15751.251818 0.000000 0 / [Wed Feb 6 09:40:37 2019][1696536.961195] kworker/u257:0 22744 22076266.053415 3 120 0.000000 0.077457 0.000000 0 / [Wed Feb 6 09:40:37 2019][1696536.972535] mdt_io00_038 27672 27060759.712055 29399 120 0.000000 5563.684001 0.000000 0 / [Wed Feb 6 09:40:37 2019][1696536.983871] mdt_io00_047 27692 27060760.000391 197 120 0.000000 9.514347 0.000000 0 / [Wed Feb 6 09:40:37 2019][1696536.995205] mdt_io00_054 27712 27107000.239126 23966 120 0.000000 953.907710 0.000000 0 / [Wed Feb 6 09:40:37 2019][1696537.006555] mdt_io00_056 27714 27060789.912456 1041 120 0.000000 39.674705 0.000000 0 / [Wed Feb 6 09:40:37 2019][1696537.017917] kworker/4:0 30083 27102503.719522 13 120 0.000000 0.087826 0.000000 0 / [Wed Feb 6 09:40:37 2019][1696537.029257] kworker/u256:0 30284 27107000.296916 1860 120 0.000000 17.195886 0.000000 0 / [Wed Feb 6 09:40:37 2019][1696537.040593] kworker/u256:2 30376 27106852.804292 1068 120 0.000000 10.646666 0.000000 0 / [Wed Feb 6 09:40:37 2019][1696537.051949] [Wed Feb 6 09:40:37 2019][1696537.053627] cpu#5, 1996.199 MHz [Wed Feb 6 09:40:37 2019][1696537.056956] .nr_running : 0 [Wed Feb 6 09:40:37 2019][1696537.061754] .load : 0 [Wed Feb 6 09:40:37 2019][1696537.066559] .nr_switches : 3050464717 [Wed Feb 6 09:40:37 2019][1696537.072138] .nr_load_updates : 422880308 [Wed Feb 6 09:40:37 2019][1696537.077635] .nr_uninterruptible : -98887 [Wed Feb 6 09:40:37 2019][1696537.082868] .next_balance : 5991.160966 [Wed Feb 6 09:40:37 2019][1696537.088536] .curr->pid : 0 [Wed Feb 6 09:40:37 2019][1696537.093334] .clock : 1696537093.107657 [Wed Feb 6 09:40:37 2019][1696537.099515] .cpu_load[0] : 0 [Wed Feb 6 09:40:37 2019][1696537.104316] .cpu_load[1] : 0 [Wed Feb 6 09:40:37 2019][1696537.109117] .cpu_load[2] : 0 [Wed Feb 6 09:40:37 2019][1696537.113915] .cpu_load[3] : 0 [Wed Feb 6 09:40:37 2019][1696537.118714] .cpu_load[4] : 0 [Wed Feb 6 09:40:37 2019][1696537.123515] .avg_idle : 2882724 [Wed Feb 6 09:40:37 2019][1696537.128835] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:37 2019][1696537.134156] [Wed Feb 6 09:40:37 2019][1696537.135833] cfs_rq[5]:/ [Wed Feb 6 09:40:37 2019][1696537.138457] .exec_clock : 0.000000 [Wed Feb 6 09:40:37 2019][1696537.143858] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:37 2019][1696537.149256] .min_vruntime : 16880604.108403 [Wed Feb 6 09:40:37 2019][1696537.155264] .max_vruntime : 0.000001 [Wed Feb 6 09:40:37 2019][1696537.160664] .spread : 0.000000 [Wed Feb 6 09:40:37 2019][1696537.166062] .spread0 : -11244969.060825 [Wed Feb 6 09:40:37 2019][1696537.172155] .nr_spread_over : 0 [Wed Feb 6 09:40:37 2019][1696537.176948] .nr_running : 0 [Wed Feb 6 09:40:37 2019][1696537.181741] .load : 0 [Wed Feb 6 09:40:37 2019][1696537.186533] .runnable_load_avg : 0 [Wed Feb 6 09:40:37 2019][1696537.191327] .blocked_load_avg : 0 [Wed Feb 6 09:40:37 2019][1696537.196120] .tg_load_avg : 0 [Wed Feb 6 09:40:37 2019][1696537.200913] .tg_load_contrib : 0 [Wed Feb 6 09:40:37 2019][1696537.205706] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:37 2019][1696537.210497] .tg->runnable_avg : 0 [Wed Feb 6 09:40:37 2019][1696537.215290] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:37 2019][1696537.220084] .throttled : 0 [Wed Feb 6 09:40:37 2019][1696537.224875] .throttle_count : 0 [Wed Feb 6 09:40:37 2019][1696537.229668] .avg->runnable_avg_sum : 45 [Wed Feb 6 09:40:37 2019][1696537.234549] .avg->runnable_avg_period : 48032 [Wed Feb 6 09:40:37 2019][1696537.239691] [Wed Feb 6 09:40:37 2019][1696537.241359] rt_rq[5]:/ [Wed Feb 6 09:40:37 2019][1696537.243899] .rt_nr_running : 0 [Wed Feb 6 09:40:37 2019][1696537.248693] .rt_throttled : 0 [Wed Feb 6 09:40:37 2019][1696537.253485] .rt_time : 0.000000 [Wed Feb 6 09:40:37 2019][1696537.258883] .rt_runtime : 958.767362 [Wed Feb 6 09:40:37 2019][1696537.264457] [Wed Feb 6 09:40:37 2019][1696537.266129] runnable tasks: [Wed Feb 6 09:40:37 2019][1696537.269103] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:37 2019][1696537.280048] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:37 2019][1696537.290913] watchdog/5 36 -8.961928 424076 0 0.000000 148027.162926 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.302243] migration/5 37 0.000000 2357 0 0.000000 522698.693367 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.313582] ksoftirqd/5 38 16880589.380684 14597782 120 0.000000 143564.952886 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.324918] kworker/5:0H 40 4730.698174 10 100 0.000000 0.032801 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.336287] kworker/5:1 7260 16880591.759294 592421 120 0.000000 13639.900971 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.347633] ccp-6-q2 10819 2169.236509 2 120 0.000000 1.039235 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.358989] kworker/5:1H 30257 16880589.357935 11490402 100 0.000000 54628.256739 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.370335] monitor_thread 5891 16880592.026971 1029029 120 0.000000 255756.846375 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.381672] ptlrpcd_01_04 5912 16880591.750147 6964710 120 0.000000 104712.454443 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.393005] ptlrpcd_01_08 5916 16880589.404900 6967031 120 0.000000 103895.338348 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.404339] ptlrpcd_01_09 5917 16880589.408739 7002642 120 0.000000 98471.949344 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.415677] ptlrpcd_01_10 5918 16880591.794611 6949776 120 0.000000 109194.905612 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.427025] ll_mgs_0002 6048 16622044.692086 1492811 120 0.000000 30023.042287 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.438362] mdt01_001 6171 16778989.817451 90089595 120 0.000000 3487840.095271 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.449697] mdt_out01_000 6197 16880588.510698 28105529 120 0.000000 547707.232224 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.461030] mdt_fld_0000 6207 16590760.108771 693 120 0.000000 8.876355 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.472369] osp-syn-10-0 6256 16880053.032812 788170 120 0.000000 20961.883340 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.483705] osp-pre-32-0 6289 16880591.760026 247607 120 0.000000 4914.368793 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.495042] osp-pre-1-0 6323 16880591.763352 247642 120 0.000000 6408.849751 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.506376] osp-syn-36-0 6332 16880480.617983 778696 120 0.000000 21140.907564 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.517713] osp-pre-8-0 6335 16880586.627227 247809 120 0.000000 3768.791397 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.529048] lquota_wb_fir-M 6415 16880494.133122 17159 120 0.000000 813.829187 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.540392] osp-pre-21-2 6481 16880589.374603 234947 120 0.000000 5430.791079 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.551729] osp-syn-45-2 6498 16880516.144945 439341 120 0.000000 13822.557460 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.563069] osp-syn-34-2 6532 16880071.704603 434634 120 0.000000 12268.968283 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.574414] mdt_io01_007 6865 15811158.954039 1161345 120 0.000000 53382.674209 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.585752] ldlm_cb01_003 7214 13528450.903026 14850409 120 0.000000 245585.825014 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.597085] mdt_rdpg01_005 7471 16605092.563853 32210256 120 0.000000 1193760.529807 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.608425] mdt01_020 8973 16880578.336278 105769970 120 0.000000 4544029.407018 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.619758] mdt01_021 8996 16880564.329807 86511619 120 0.000000 3632997.653663 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.631095] mdt01_031 9015 16880512.109021 100213582 120 0.000000 4103703.726619 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.642430] mdt_io01_022 9037 16605070.642706 1893878 120 0.000000 84756.300890 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.653766] mdt_rdpg01_006 9146 16605092.595834 16015268 120 0.000000 687434.642502 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.665104] mdt_io01_025 9649 15811158.957695 4713490 120 0.000000 174049.845126 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.676443] mdt_io01_032 15082 15811485.463217 3594401 120 0.000000 137413.313833 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.687781] mdt_io01_044 20865 15811159.161765 4704455 120 0.000000 177376.925116 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.699120] mdt_io01_049 20870 15811166.953445 8592014 120 0.000000 315617.402099 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.710455] mdt_io01_054 20875 15811159.006499 5736962 120 0.000000 217602.562068 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.721791] ldlm_cn01_015 21075 16880594.521254 27116473 120 0.000000 826697.258199 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.733144] mdt01_042 15164 16880586.682081 10132112 120 0.000000 755825.960590 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.744483] mdt01_048 15224 16880607.178541 2870476 120 0.000000 266441.234579 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.755822] mdt01_067 15294 16690626.079924 2733036 120 0.000000 252018.421452 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.767159] mdt_io01_069 15437 16604802.523645 756 120 0.000000 27.393319 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.778489] mdt_io01_074 15443 16614628.874348 42935 120 0.000000 1438.814222 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.789826] mdt_io01_075 15444 16608420.995810 904 120 0.000000 35.998397 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.801164] mdt_io01_077 15447 16880591.753063 13285 120 0.000000 587.760409 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.812499] mdt_io01_078 15459 16604802.523645 660 120 0.000000 25.915991 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.823836] ldlm_bl_05 15962 16880431.087167 2325013 120 0.000000 322602.763406 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.835171] ldlm_bl_06 15963 16880418.127571 2326974 120 0.000000 1304451.130542 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.846509] ldlm_bl_23 15986 16880418.121448 2137117 120 0.000000 1034705.651992 0.000000 1 / [Wed Feb 6 09:40:37 2019][1696537.857857] mdt_io01_083 27682 15811166.844611 361 120 0.000000 15.883173 0.000000 1 / [Wed Feb 6 09:40:38 2019][1696537.869189] mdt_io01_087 27691 16604802.523675 859 120 0.000000 35.597142 0.000000 1 / [Wed Feb 6 09:40:38 2019][1696537.880526] mdt_io01_089 27695 16606731.087098 678 120 0.000000 1848.230857 0.000000 1 / [Wed Feb 6 09:40:38 2019][1696537.891870] mdt_io01_108 27757 16605070.615895 7333 120 0.000000 217.922650 0.000000 1 / [Wed Feb 6 09:40:38 2019][1696537.903206] mdt_io01_111 27762 16614640.830234 15251 120 0.000000 2918.299422 0.000000 1 / [Wed Feb 6 09:40:38 2019][1696537.914551] mdt_rdpg01_013 28954 16880565.471210 13095 120 0.000000 423.221723 0.000000 1 / [Wed Feb 6 09:40:38 2019][1696537.925897] kworker/5:0 30238 16859983.642507 9 120 0.000000 0.075794 0.000000 1 / [Wed Feb 6 09:40:38 2019][1696537.937251] [Wed Feb 6 09:40:38 2019][1696537.938931] cpu#6, 1996.199 MHz [Wed Feb 6 09:40:38 2019][1696537.942259] .nr_running : 0 [Wed Feb 6 09:40:38 2019][1696537.947062] .load : 0 [Wed Feb 6 09:40:38 2019][1696537.951863] .nr_switches : 1336216232 [Wed Feb 6 09:40:38 2019][1696537.957445] .nr_load_updates : 353701420 [Wed Feb 6 09:40:38 2019][1696537.962938] .nr_uninterruptible : -141403 [Wed Feb 6 09:40:38 2019][1696537.968261] .next_balance : 5991.161850 [Wed Feb 6 09:40:38 2019][1696537.973929] .curr->pid : 0 [Wed Feb 6 09:40:38 2019][1696537.978728] .clock : 1696537978.423530 [Wed Feb 6 09:40:38 2019][1696537.984918] .cpu_load[0] : 0 [Wed Feb 6 09:40:38 2019][1696537.989720] .cpu_load[1] : 0 [Wed Feb 6 09:40:38 2019][1696537.994519] .cpu_load[2] : 0 [Wed Feb 6 09:40:38 2019][1696537.999323] .cpu_load[3] : 0 [Wed Feb 6 09:40:38 2019][1696538.004123] .cpu_load[4] : 0 [Wed Feb 6 09:40:38 2019][1696538.008921] .avg_idle : 2606926 [Wed Feb 6 09:40:38 2019][1696538.014237] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:38 2019][1696538.019558] [Wed Feb 6 09:40:38 2019][1696538.021234] cfs_rq[6]:/ [Wed Feb 6 09:40:38 2019][1696538.023859] .exec_clock : 0.000000 [Wed Feb 6 09:40:38 2019][1696538.029258] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:38 2019][1696538.034657] .min_vruntime : 15475081.153033 [Wed Feb 6 09:40:38 2019][1696538.040663] .max_vruntime : 0.000001 [Wed Feb 6 09:40:38 2019][1696538.046064] .spread : 0.000000 [Wed Feb 6 09:40:38 2019][1696538.051464] .spread0 : -12650492.286630 [Wed Feb 6 09:40:38 2019][1696538.057555] .nr_spread_over : 0 [Wed Feb 6 09:40:38 2019][1696538.062349] .nr_running : 0 [Wed Feb 6 09:40:38 2019][1696538.067142] .load : 0 [Wed Feb 6 09:40:38 2019][1696538.071933] .runnable_load_avg : 0 [Wed Feb 6 09:40:38 2019][1696538.076726] .blocked_load_avg : 0 [Wed Feb 6 09:40:38 2019][1696538.081519] .tg_load_avg : 0 [Wed Feb 6 09:40:38 2019][1696538.086313] .tg_load_contrib : 0 [Wed Feb 6 09:40:38 2019][1696538.091106] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:38 2019][1696538.095898] .tg->runnable_avg : 0 [Wed Feb 6 09:40:38 2019][1696538.100691] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:38 2019][1696538.105482] .throttled : 0 [Wed Feb 6 09:40:38 2019][1696538.110276] .throttle_count : 0 [Wed Feb 6 09:40:38 2019][1696538.115069] .avg->runnable_avg_sum : 170 [Wed Feb 6 09:40:38 2019][1696538.120034] .avg->runnable_avg_period : 46244 [Wed Feb 6 09:40:38 2019][1696538.125177] [Wed Feb 6 09:40:38 2019][1696538.126848] rt_rq[6]:/ [Wed Feb 6 09:40:38 2019][1696538.129387] .rt_nr_running : 0 [Wed Feb 6 09:40:38 2019][1696538.134179] .rt_throttled : 0 [Wed Feb 6 09:40:38 2019][1696538.138973] .rt_time : 0.000000 [Wed Feb 6 09:40:38 2019][1696538.144372] .rt_runtime : 635.895540 [Wed Feb 6 09:40:38 2019][1696538.149945] [Wed Feb 6 09:40:38 2019][1696538.151615] runnable tasks: [Wed Feb 6 09:40:38 2019][1696538.154591] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:38 2019][1696538.165535] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:38 2019][1696538.176400] watchdog/6 41 -8.960936 424068 0 0.000000 82890.621297 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.187731] migration/6 42 0.000000 2418 0 0.000000 692582.715290 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.199067] ksoftirqd/6 43 15475069.281139 15582496 120 0.000000 145580.649039 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.210403] kworker/6:0H 45 4458.470855 10 100 0.000000 0.039276 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.221781] ccp-3-q3 10588 1965.812285 2 120 0.000000 0.011662 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.233113] ccp-5-q2 10700 1985.558105 2 120 0.000000 0.006212 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.244466] slapd 19486 15474402.146487 3617 120 0.000000 10066.013108 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.255803] kworker/6:1H 27841 15475033.063463 13727377 100 0.000000 66612.518901 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.267146] ptlrpc_hr02_000 5848 15475041.648394 83773823 120 0.000000 776750.404006 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.278482] ptlrpc_hr02_003 5852 15475044.892315 83761966 120 0.000000 771264.691882 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.289834] ldlm_cb02_000 6024 15466686.799918 10451180 120 0.000000 192708.406640 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.301183] mdt_out02_000 6199 15474396.719875 3635404 120 0.000000 85209.104185 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.312519] mdt_io02_002 6217 14934737.678084 7656767 120 0.000000 299640.059052 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.323856] osp-syn-42-0 6266 15474194.526411 778736 120 0.000000 20925.854477 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.335194] osp-syn-7-0 6304 15474112.664272 772071 120 0.000000 20802.686627 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.346538] osp-syn-47-2 6504 15474896.583723 440108 120 0.000000 12429.839852 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.357871] osp-syn-38-2 6506 15474900.507435 437370 120 0.000000 12475.641719 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.369216] ldlm_cn02_006 6776 15392948.662841 15760302 120 0.000000 529430.783811 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.380552] mdt_io02_003 6779 14934737.651974 6541427 120 0.000000 264725.612561 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.391896] mdt_io02_007 7098 14934737.728359 4386871 120 0.000000 182878.788928 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.403234] mdt_io02_012 7222 14934737.683524 7564624 120 0.000000 296993.703055 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.414577] mdt02_013 8981 15474895.942445 86203397 120 0.000000 3867754.580716 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.425917] mdt_io02_017 9007 14936720.320906 5974640 120 0.000000 237624.693568 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.437254] mdt02_019 9029 15466682.409857 88462653 120 0.000000 4439109.670237 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.448611] ldlm_cn02_013 23316 15431094.076648 10448052 120 0.000000 367018.945419 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.459951] ldlm_cn02_018 23322 15431093.225121 12400735 120 0.000000 478835.303521 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.471296] kworker/6:1 13218 15475033.860376 50531 120 0.000000 7832.698346 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.482631] python 14075 15475069.753565 108961 120 0.000000 95348.355842 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.493969] mdt02_029 15181 15475050.177821 9527461 120 0.000000 11337397.156888 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.505317] mdt_io02_034 15452 14934737.649720 289801 120 0.000000 11981.733649 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.516648] mdt_io02_035 15453 14934737.731234 32452 120 0.000000 1269.892094 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.527986] ldlm_bl_08 15965 15474909.257196 2324565 120 0.000000 377387.963906 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.539322] ldlm_bl_19 15982 15474915.187525 2255990 120 0.000000 664477.501181 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.550659] ldlm_bl_24 15987 15474894.548071 2326521 120 0.000000 111712.435823 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.562007] mdt_io02_040 27680 14934737.686039 234 120 0.000000 10.049014 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.573346] mdt_io02_053 27722 14934737.692030 475 120 0.000000 21.751136 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.584687] mdt_io02_064 27746 14934737.677132 5702 120 0.000000 199.665485 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.596023] mdt_io02_071 27759 14934737.667584 19458 120 0.000000 669.846519 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.607358] mdt_io02_076 27767 14934469.635283 178 120 0.000000 8.176807 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.618693] mdt_io02_079 27770 14683050.741816 52 120 0.000000 2.087557 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.630030] mdt_io02_087 27778 14934469.634841 146 120 0.000000 5.465895 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.641366] mdt_io02_090 27781 14934737.652715 13920 120 0.000000 814.590836 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.652715] kworker/6:2 29831 15344789.700396 11 120 0.000000 0.063810 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.664047] kworker/u259:2 30246 15475063.457913 303 120 0.000000 41.061444 0.000000 2 / [Wed Feb 6 09:40:38 2019][1696538.675405] [Wed Feb 6 09:40:38 2019][1696538.677080] cpu#7, 1996.199 MHz [Wed Feb 6 09:40:38 2019][1696538.680410] .nr_running : 0 [Wed Feb 6 09:40:38 2019][1696538.685213] .load : 0 [Wed Feb 6 09:40:38 2019][1696538.690013] .nr_switches : 593146984 [Wed Feb 6 09:40:38 2019][1696538.695505] .nr_load_updates : 226198135 [Wed Feb 6 09:40:38 2019][1696538.700999] .nr_uninterruptible : -30412 [Wed Feb 6 09:40:38 2019][1696538.706233] .next_balance : 5991.162589 [Wed Feb 6 09:40:38 2019][1696538.711894] .curr->pid : 0 [Wed Feb 6 09:40:38 2019][1696538.716699] .clock : 1696538716.438678 [Wed Feb 6 09:40:38 2019][1696538.722882] .cpu_load[0] : 0 [Wed Feb 6 09:40:38 2019][1696538.727678] .cpu_load[1] : 0 [Wed Feb 6 09:40:38 2019][1696538.732479] .cpu_load[2] : 81 [Wed Feb 6 09:40:38 2019][1696538.737367] .cpu_load[3] : 42 [Wed Feb 6 09:40:38 2019][1696538.742257] .cpu_load[4] : 24 [Wed Feb 6 09:40:38 2019][1696538.747146] .avg_idle : 8006814 [Wed Feb 6 09:40:38 2019][1696538.752467] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:38 2019][1696538.757787] [Wed Feb 6 09:40:38 2019][1696538.759462] cfs_rq[7]:/ [Wed Feb 6 09:40:38 2019][1696538.762088] .exec_clock : 0.000000 [Wed Feb 6 09:40:38 2019][1696538.767486] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:38 2019][1696538.772885] .min_vruntime : 35641627.159687 [Wed Feb 6 09:40:38 2019][1696538.778893] .max_vruntime : 0.000001 [Wed Feb 6 09:40:38 2019][1696538.784290] .spread : 0.000000 [Wed Feb 6 09:40:38 2019][1696538.789690] .spread0 : 7516053.716267 [Wed Feb 6 09:40:38 2019][1696538.795612] .nr_spread_over : 0 [Wed Feb 6 09:40:38 2019][1696538.800402] .nr_running : 0 [Wed Feb 6 09:40:38 2019][1696538.805196] .load : 0 [Wed Feb 6 09:40:38 2019][1696538.809989] .runnable_load_avg : 0 [Wed Feb 6 09:40:38 2019][1696538.814781] .blocked_load_avg : 0 [Wed Feb 6 09:40:38 2019][1696538.819573] .tg_load_avg : 0 [Wed Feb 6 09:40:38 2019][1696538.824367] .tg_load_contrib : 0 [Wed Feb 6 09:40:38 2019][1696538.829159] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:38 2019][1696538.833951] .tg->runnable_avg : 0 [Wed Feb 6 09:40:38 2019][1696538.838746] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:38 2019][1696538.843538] .throttled : 0 [Wed Feb 6 09:40:38 2019][1696538.848331] .throttle_count : 0 [Wed Feb 6 09:40:38 2019][1696538.853125] .avg->runnable_avg_sum : 329 [Wed Feb 6 09:40:38 2019][1696538.858089] .avg->runnable_avg_period : 46923 [Wed Feb 6 09:40:38 2019][1696538.863234] [Wed Feb 6 09:40:38 2019][1696538.864901] rt_rq[7]:/ [Wed Feb 6 09:40:38 2019][1696538.867443] .rt_nr_running : 0 [Wed Feb 6 09:40:38 2019][1696538.872235] .rt_throttled : 0 [Wed Feb 6 09:40:39 2019][1696538.877026] .rt_time : 0.000000 [Wed Feb 6 09:40:39 2019][1696538.882425] .rt_runtime : 698.762412 [Wed Feb 6 09:40:39 2019][1696538.887998] [Wed Feb 6 09:40:39 2019][1696538.889671] runnable tasks: [Wed Feb 6 09:40:39 2019][1696538.892644] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:39 2019][1696538.903589] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:39 2019][1696538.914457] watchdog/7 46 -8.966767 424118 0 0.000000 94082.479495 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696538.925794] migration/7 47 0.000000 2547 0 0.000000 399002.346290 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696538.937132] ksoftirqd/7 48 35636361.297650 4387229 120 0.000000 45821.406090 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696538.948468] kworker/7:0H 50 2836.547751 14 100 0.000000 0.039204 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696538.959839] kworker/7:1 7307 35641610.807202 396781 120 0.000000 14035.717081 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696538.971182] ccp-5-q3 10714 835.153874 2 120 0.000000 0.005661 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696538.982524] multipathd 13613 0.000000 208 0 0.000000 21.862669 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696538.993873] kworker/7:1H 20226 35641546.244007 5235899 100 0.000000 20428.617656 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.005220] ptlrpc_hr03_001 5856 35641586.430927 20348755 120 0.000000 179822.726136 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.016553] ptlrpc_hr03_002 5857 35641586.386522 20350416 120 0.000000 179035.302090 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.027905] ptlrpcd_03_09 5941 35641615.233306 13129779 120 0.000000 198497.857598 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.039242] ldlm_cn03_000 6038 35641602.348139 11911882 120 0.000000 365512.682720 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.050582] ll_mgs_0001 6047 35466431.247261 770979 120 0.000000 25237.705082 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.061918] ll_evictor 6049 35636460.094432 172 120 0.000000 1483.752119 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.073263] osp-pre-2-0 6242 35455448.943794 204462 120 0.000000 7311.122404 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.084600] osp-pre-39-0 6253 35641611.733655 247842 120 0.000000 6510.226241 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.095938] osp-syn-5-0 6284 35636057.561326 774725 120 0.000000 20961.995622 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.107277] osp-syn-30-0 6340 35636044.542299 774812 120 0.000000 20648.535436 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.118623] osp-pre-18-2 6467 35641606.166923 235075 120 0.000000 4903.352616 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.129965] osp-syn-7-2 6494 35641590.952165 433868 120 0.000000 12229.758494 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.141298] osp-pre-29-2 6511 35641616.896588 234997 120 0.000000 3346.369274 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.152634] osp-syn-27-2 6518 35641586.288415 445717 120 0.000000 12411.689835 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.163980] mdt03_008 6845 35641602.428842 35485457 120 0.000000 1565073.769923 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.175314] mdt_rdpg03_002 6861 35446230.400232 4589128 120 0.000000 191704.982244 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.186652] mdt_io03_004 6864 35501010.078151 1998700 120 0.000000 70317.172002 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.198017] mdt_io03_010 19906 35501010.076729 2371223 120 0.000000 82485.173485 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.209351] mdt_rdpg03_004 19922 35449115.629446 4126013 120 0.000000 174946.700020 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.220685] mdt_io03_011 19923 35501010.072591 2043971 120 0.000000 70021.228525 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.232039] ldlm_cn03_013 59873 35545375.888199 4063516 120 0.000000 351232.460795 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.243398] mdt_io03_016 17102 35512792.439385 55503 120 0.000000 5629.101477 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.254731] mdt_io03_019 17105 35451718.152681 269399 120 0.000000 12126.170026 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.266064] mdt03_018 17203 35641629.833027 7274117 120 0.000000 470011.407633 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.277404] mdt03_027 17222 35641586.208794 5672066 120 0.000000 392986.117206 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.288738] mdt03_029 17224 35641586.209736 7576261 120 0.000000 458541.965063 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.300097] ldlm_cn03_024 28512 35560213.169111 11860 120 0.000000 16764.280110 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.311431] ldlm_cn03_030 28542 35545373.655111 18736 120 0.000000 20070.828337 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.322765] kworker/7:2 28894 35432552.841948 3 120 0.000000 0.019728 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.334101] ldlm_cn03_031 28944 35560209.645791 8957 120 0.000000 3730.574163 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.345440] ll_mgs_0012 28980 35466427.742356 99 120 0.000000 1069.855561 0.000000 3 / [Wed Feb 6 09:40:39 2019][1696539.356801] [Wed Feb 6 09:40:39 2019][1696539.358478] cpu#8, 1996.199 MHz [Wed Feb 6 09:40:39 2019][1696539.361810] .nr_running : 0 [Wed Feb 6 09:40:39 2019][1696539.366610] .load : 0 [Wed Feb 6 09:40:39 2019][1696539.371412] .nr_switches : 4296025266 [Wed Feb 6 09:40:39 2019][1696539.376990] .nr_load_updates : 515358667 [Wed Feb 6 09:40:39 2019][1696539.382480] .nr_uninterruptible : 615109 [Wed Feb 6 09:40:39 2019][1696539.387714] .next_balance : 5991.163270 [Wed Feb 6 09:40:39 2019][1696539.393382] .curr->pid : 0 [Wed Feb 6 09:40:39 2019][1696539.398184] .clock : 1696539397.456212 [Wed Feb 6 09:40:39 2019][1696539.404371] .cpu_load[0] : 0 [Wed Feb 6 09:40:39 2019][1696539.409170] .cpu_load[1] : 0 [Wed Feb 6 09:40:39 2019][1696539.413974] .cpu_load[2] : 0 [Wed Feb 6 09:40:39 2019][1696539.418776] .cpu_load[3] : 0 [Wed Feb 6 09:40:39 2019][1696539.423579] .cpu_load[4] : 0 [Wed Feb 6 09:40:39 2019][1696539.428379] .avg_idle : 6687558 [Wed Feb 6 09:40:39 2019][1696539.433700] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:39 2019][1696539.439022] [Wed Feb 6 09:40:39 2019][1696539.440696] cfs_rq[8]:/ [Wed Feb 6 09:40:39 2019][1696539.443322] .exec_clock : 0.000000 [Wed Feb 6 09:40:39 2019][1696539.448722] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:39 2019][1696539.454119] .min_vruntime : 24623838.393893 [Wed Feb 6 09:40:39 2019][1696539.460126] .max_vruntime : 0.000001 [Wed Feb 6 09:40:39 2019][1696539.465526] .spread : 0.000000 [Wed Feb 6 09:40:39 2019][1696539.470924] .spread0 : -3501737.538540 [Wed Feb 6 09:40:39 2019][1696539.476930] .nr_spread_over : 0 [Wed Feb 6 09:40:39 2019][1696539.481723] .nr_running : 0 [Wed Feb 6 09:40:39 2019][1696539.486516] .load : 0 [Wed Feb 6 09:40:39 2019][1696539.491311] .runnable_load_avg : 0 [Wed Feb 6 09:40:39 2019][1696539.496103] .blocked_load_avg : 1 [Wed Feb 6 09:40:39 2019][1696539.500895] .tg_load_avg : 0 [Wed Feb 6 09:40:39 2019][1696539.505688] .tg_load_contrib : 0 [Wed Feb 6 09:40:39 2019][1696539.510481] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:39 2019][1696539.515273] .tg->runnable_avg : 0 [Wed Feb 6 09:40:39 2019][1696539.520068] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:39 2019][1696539.524860] .throttled : 0 [Wed Feb 6 09:40:39 2019][1696539.529650] .throttle_count : 0 [Wed Feb 6 09:40:39 2019][1696539.534445] .avg->runnable_avg_sum : 159 [Wed Feb 6 09:40:39 2019][1696539.539410] .avg->runnable_avg_period : 47435 [Wed Feb 6 09:40:39 2019][1696539.544553] [Wed Feb 6 09:40:39 2019][1696539.546224] rt_rq[8]:/ [Wed Feb 6 09:40:39 2019][1696539.548762] .rt_nr_running : 0 [Wed Feb 6 09:40:39 2019][1696539.553554] .rt_throttled : 0 [Wed Feb 6 09:40:39 2019][1696539.558348] .rt_time : 0.000000 [Wed Feb 6 09:40:39 2019][1696539.563747] .rt_runtime : 979.166667 [Wed Feb 6 09:40:39 2019][1696539.569320] [Wed Feb 6 09:40:39 2019][1696539.570993] runnable tasks: [Wed Feb 6 09:40:39 2019][1696539.573965] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:39 2019][1696539.584911] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:39 2019][1696539.595778] watchdog/8 51 -11.973549 424085 0 0.000000 121553.048804 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.607117] migration/8 52 0.000000 3356 0 0.000000 640043.675716 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.618452] ksoftirqd/8 53 24623636.204462 10838386 120 0.000000 114917.786791 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.629788] kworker/8:0H 55 4345.947450 9 100 0.000000 0.031883 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.641172] kworker/8:1H 16770 24623636.168072 11675185 100 0.000000 48491.501805 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.652510] smartd 17510 24587869.471151 10541 120 0.000000 362.097633 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.663867] ptlrpcd_00_00 5896 24623826.312738 6012831 120 0.000000 88484.824123 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.675205] ptlrpcd_00_09 5905 24623826.458165 6065066 120 0.000000 86444.228495 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.686540] ptlrpcd_00_10 5906 24623826.451854 6021534 120 0.000000 94766.346870 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.697883] jbd2/dm-0-8 6010 303747.835714 163 120 0.000000 1.766570 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.709229] mdt_rdpg00_001 6180 23810983.501134 67364812 120 0.000000 1575887.659719 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.720571] osp-syn-39-0 6252 24606779.836933 777804 120 0.000000 20690.296760 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.731905] osp-syn-25-0 6278 24601632.188668 788317 120 0.000000 21018.544149 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.743238] osp-syn-44-0 6286 24601635.536639 779196 120 0.000000 21278.334270 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.754586] osp-syn-5-2 6474 24623788.582607 431765 120 0.000000 12137.236178 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.765921] osp-syn-16-2 6482 24623789.403296 441286 120 0.000000 12226.439759 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.777259] osp-syn-1-2 6512 24623787.232190 435148 120 0.000000 13937.693128 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.788594] lquota_wb_fir-M 6593 24623826.398963 17159 120 0.000000 1893.519236 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.799937] mdt_io00_004 6786 23794542.524141 2791478 120 0.000000 123064.855843 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.811276] mdt_io00_006 6863 23794512.507368 5638009 120 0.000000 217859.548833 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.822610] mdt_io00_007 6909 23794512.512298 6166814 120 0.000000 258899.937380 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.833949] mdt_io00_008 7094 23794512.497610 4520168 120 0.000000 180813.531055 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.845293] mdt00_018 8995 24623802.049700 238585194 120 0.000000 8081711.136407 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.856627] mdt_io00_019 9002 23794554.463957 7330397 120 0.000000 284163.675411 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.867964] mdt00_020 9027 23804370.608393 230614110 120 0.000000 8230438.079260 0.000000 0 / [Wed Feb 6 09:40:39 2019][1696539.879313] ldlm_cn00_020 19871 24623831.081557 13660099 120 0.000000 553662.665049 0.000000 0 / [Wed Feb 6 09:40:40 2019][1696539.890648] ldlm_cn00_023 20040 24587827.881333 9338049 120 0.000000 499573.503832 0.000000 0 / [Wed Feb 6 09:40:40 2019][1696539.901982] mdt_io00_027 20199 23794512.507709 3513801 120 0.000000 146124.042974 0.000000 0 / [Wed Feb 6 09:40:40 2019][1696539.913328] kworker/8:2 34114 24587869.281182 521389 120 0.000000 17617.353036 0.000000 0 / [Wed Feb 6 09:40:40 2019][1696539.924666] mdt00_028 53421 24623829.948459 41961879 120 0.000000 1476417.531271 0.000000 0 / [Wed Feb 6 09:40:40 2019][1696539.936001] mdt00_029 15165 24623828.258619 7499086 120 0.000000 567271.470039 0.000000 0 / [Wed Feb 6 09:40:40 2019][1696539.947338] mdt00_038 15240 24403751.661773 14614149 120 0.000000 1134587.777606 0.000000 0 / [Wed Feb 6 09:40:40 2019][1696539.958673] mdt00_043 15280 23804370.608363 5555826 120 0.000000 633795.777537 0.000000 0 / [Wed Feb 6 09:40:40 2019][1696539.970012] mdt00_049 15408 24623826.396408 7050249 120 0.000000 216149.626995 0.000000 0 / [Wed Feb 6 09:40:40 2019][1696539.981348] mdt_io00_037 15468 23794512.515965 257156 120 0.000000 10327.897890 0.000000 0 / [Wed Feb 6 09:40:40 2019][1696539.992702] mdt_io00_057 27728 23794542.512117 392 120 0.000000 15.882965 0.000000 0 / [Wed Feb 6 09:40:40 2019][1696540.004050] ll_mgs_0026 28994 24623835.600262 184647 120 0.000000 4351.901833 0.000000 0 / [Wed Feb 6 09:40:40 2019][1696540.015393] kworker/8:1 30223 24623826.398462 340 120 0.000000 1.921548 0.000000 0 / [Wed Feb 6 09:40:40 2019][1696540.026753] [Wed Feb 6 09:40:40 2019][1696540.028429] cpu#9, 1996.199 MHz [Wed Feb 6 09:40:40 2019][1696540.031759] .nr_running : 0 [Wed Feb 6 09:40:40 2019][1696540.036560] .load : 0 [Wed Feb 6 09:40:40 2019][1696540.041363] .nr_switches : 2823292607 [Wed Feb 6 09:40:40 2019][1696540.046943] .nr_load_updates : 421858409 [Wed Feb 6 09:40:40 2019][1696540.052439] .nr_uninterruptible : -153779 [Wed Feb 6 09:40:40 2019][1696540.057761] .next_balance : 5991.163943 [Wed Feb 6 09:40:40 2019][1696540.063427] .curr->pid : 0 [Wed Feb 6 09:40:40 2019][1696540.068230] .clock : 1696540067.472368 [Wed Feb 6 09:40:40 2019][1696540.074417] .cpu_load[0] : 0 [Wed Feb 6 09:40:40 2019][1696540.079220] .cpu_load[1] : 0 [Wed Feb 6 09:40:40 2019][1696540.084021] .cpu_load[2] : 0 [Wed Feb 6 09:40:40 2019][1696540.088820] .cpu_load[3] : 128 [Wed Feb 6 09:40:40 2019][1696540.093794] .cpu_load[4] : 45 [Wed Feb 6 09:40:40 2019][1696540.098684] .avg_idle : 2338725 [Wed Feb 6 09:40:40 2019][1696540.104005] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:40 2019][1696540.109327] [Wed Feb 6 09:40:40 2019][1696540.111001] cfs_rq[9]:/ [Wed Feb 6 09:40:40 2019][1696540.113627] .exec_clock : 0.000000 [Wed Feb 6 09:40:40 2019][1696540.119027] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:40 2019][1696540.124425] .min_vruntime : 18619594.224702 [Wed Feb 6 09:40:40 2019][1696540.130433] .max_vruntime : 0.000001 [Wed Feb 6 09:40:40 2019][1696540.135832] .spread : 0.000000 [Wed Feb 6 09:40:40 2019][1696540.141231] .spread0 : -9505984.070485 [Wed Feb 6 09:40:40 2019][1696540.147237] .nr_spread_over : 0 [Wed Feb 6 09:40:40 2019][1696540.152029] .nr_running : 0 [Wed Feb 6 09:40:40 2019][1696540.156823] .load : 0 [Wed Feb 6 09:40:40 2019][1696540.161615] .runnable_load_avg : 0 [Wed Feb 6 09:40:40 2019][1696540.166407] .blocked_load_avg : 1 [Wed Feb 6 09:40:40 2019][1696540.171200] .tg_load_avg : 0 [Wed Feb 6 09:40:40 2019][1696540.175993] .tg_load_contrib : 0 [Wed Feb 6 09:40:40 2019][1696540.180786] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:40 2019][1696540.185578] .tg->runnable_avg : 0 [Wed Feb 6 09:40:40 2019][1696540.190371] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:40 2019][1696540.195164] .throttled : 0 [Wed Feb 6 09:40:40 2019][1696540.199958] .throttle_count : 0 [Wed Feb 6 09:40:40 2019][1696540.204750] .avg->runnable_avg_sum : 90 [Wed Feb 6 09:40:40 2019][1696540.209630] .avg->runnable_avg_period : 47031 [Wed Feb 6 09:40:40 2019][1696540.214772] [Wed Feb 6 09:40:40 2019][1696540.216443] rt_rq[9]:/ [Wed Feb 6 09:40:40 2019][1696540.218980] .rt_nr_running : 0 [Wed Feb 6 09:40:40 2019][1696540.223775] .rt_throttled : 0 [Wed Feb 6 09:40:40 2019][1696540.228566] .rt_time : 0.006322 [Wed Feb 6 09:40:40 2019][1696540.233966] .rt_runtime : 935.924208 [Wed Feb 6 09:40:40 2019][1696540.239538] [Wed Feb 6 09:40:40 2019][1696540.241212] runnable tasks: [Wed Feb 6 09:40:40 2019][1696540.244186] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:40 2019][1696540.255132] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:40 2019][1696540.265996] watchdog/9 56 -11.979231 424087 0 0.000000 109952.479770 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.277334] migration/9 57 0.000000 2314 0 0.000000 558832.307216 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.288673] ksoftirqd/9 58 18619580.490213 16953306 120 0.000000 161670.426638 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.300009] kworker/9:0H 60 4757.307286 10 100 0.000000 0.035779 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.311395] dsm_sa_datamgrd 18868 18619576.905494 7030632 120 0.000000 392324.662629 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.322737] agetty 19334 52306.316263 16 120 0.000000 17.847520 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.334071] slapd 29295 18619337.212828 136 120 0.000000 222.347499 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.345406] kworker/9:1H 29218 18619580.473354 13544957 100 0.000000 62464.524164 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.356743] kworker/9:0 126392 18619580.481346 596399 120 0.000000 19670.245646 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.368172] kiblnd_sd_01_01 5880 18619594.826021 2884986541 120 0.000000 21169114.161089 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.379609] ldlm_cb01_001 6023 18618502.894826 14793884 120 0.000000 288184.545068 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.390941] ldlm_cn01_000 6032 18609345.912267 25974216 120 0.000000 741678.026624 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.402282] jbd2/dm-3-8 6151 18619580.862691 43084478 120 0.000000 4159703.900888 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.413620] mdt_seqm_0000 6205 18619571.650174 2763 120 0.000000 33.674246 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.424961] osp-pre-10-0 6257 18619580.488139 247827 120 0.000000 3794.524701 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.436295] osp-pre-26-0 6263 18619582.644725 247838 120 0.000000 4832.375616 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.447637] osp-syn-8-0 6334 18618962.023807 787514 120 0.000000 21050.571515 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.458983] osp-syn-0-2 6472 18619542.317359 443002 120 0.000000 12556.549071 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.470324] osp-syn-15-2 6508 18619509.223945 439303 120 0.000000 12415.014390 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.481657] osp-syn-22-2 6514 18619516.002527 443502 120 0.000000 12103.838072 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.492995] mdt01_003 6591 18619573.241857 97052284 120 0.000000 3728939.687048 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.504330] mdt_out01_004 6612 18619580.541028 26863678 120 0.000000 518862.557921 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.515675] mdt01_010 7081 18619570.718697 100777103 120 0.000000 4005881.779519 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.527025] mdt01_032 9016 18504121.074803 103972313 120 0.000000 4345216.896239 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.538359] mdt_rdpg01_007 9149 18619580.475504 31594296 120 0.000000 1177392.116311 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.549691] ldlm_cn01_010 9155 18609336.300350 31567742 120 0.000000 834714.696824 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.561036] mdt_io01_038 15180 18497122.199594 7744209 120 0.000000 283375.241902 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.572377] ldlm_cn01_012 19973 18609336.221769 15119431 120 0.000000 408793.465470 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.583726] mdt01_037 48673 18504121.073871 30487467 120 0.000000 1537152.180968 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.595062] mdt01_038 48703 18504121.073500 25553203 120 0.000000 1161956.804001 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.606409] mdt01_054 15242 18504121.072679 2775669 120 0.000000 111163.794743 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.617746] mdt01_062 15275 18504869.881420 2086424 120 0.000000 218988.274650 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.629082] mdt01_065 15292 18504121.072970 3474611 120 0.000000 305166.771302 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.640419] mdt01_070 15384 18504121.072859 7979762 120 0.000000 653701.372344 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.651761] ldlm_bl_22 15985 18619541.986720 2327722 120 0.000000 18897.208805 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.663112] mdt_io01_081 27677 18309012.618432 819 120 0.000000 29.077060 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.674452] mdt_io01_103 27745 18619541.980648 43105 120 0.000000 1510.479352 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.685798] kworker/9:2 28872 18441943.405451 9 120 0.000000 0.050876 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.697135] ll_mgs_0007 28958 18506245.209994 7633 120 0.000000 303.766363 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.708471] ll_mgs_0019 28987 18506091.434272 104 120 0.000000 1639.401214 0.000000 1 / [Wed Feb 6 09:40:40 2019][1696540.719834] [Wed Feb 6 09:40:40 2019][1696540.721511] cpu#10, 1996.199 MHz [Wed Feb 6 09:40:40 2019][1696540.724929] .nr_running : 0 [Wed Feb 6 09:40:40 2019][1696540.729729] .load : 0 [Wed Feb 6 09:40:40 2019][1696540.734534] .nr_switches : 4327566957 [Wed Feb 6 09:40:40 2019][1696540.740110] .nr_load_updates : 396095302 [Wed Feb 6 09:40:40 2019][1696540.745604] .nr_uninterruptible : -57940 [Wed Feb 6 09:40:40 2019][1696540.750842] .next_balance : 5991.164634 [Wed Feb 6 09:40:40 2019][1696540.756510] .curr->pid : 0 [Wed Feb 6 09:40:40 2019][1696540.761312] .clock : 1696540760.493735 [Wed Feb 6 09:40:40 2019][1696540.767502] .cpu_load[0] : 0 [Wed Feb 6 09:40:40 2019][1696540.772302] .cpu_load[1] : 0 [Wed Feb 6 09:40:40 2019][1696540.777103] .cpu_load[2] : 0 [Wed Feb 6 09:40:40 2019][1696540.781904] .cpu_load[3] : 0 [Wed Feb 6 09:40:40 2019][1696540.786705] .cpu_load[4] : 0 [Wed Feb 6 09:40:40 2019][1696540.791504] .avg_idle : 5129340 [Wed Feb 6 09:40:40 2019][1696540.796828] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:40 2019][1696540.802147] [Wed Feb 6 09:40:40 2019][1696540.803823] cfs_rq[10]:/ [Wed Feb 6 09:40:40 2019][1696540.806536] .exec_clock : 0.000000 [Wed Feb 6 09:40:40 2019][1696540.811934] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:40 2019][1696540.817335] .min_vruntime : 10936013.104795 [Wed Feb 6 09:40:40 2019][1696540.823341] .max_vruntime : 0.000001 [Wed Feb 6 09:40:40 2019][1696540.828741] .spread : 0.000000 [Wed Feb 6 09:40:40 2019][1696540.834140] .spread0 : -17189567.613287 [Wed Feb 6 09:40:40 2019][1696540.840234] .nr_spread_over : 0 [Wed Feb 6 09:40:40 2019][1696540.845026] .nr_running : 0 [Wed Feb 6 09:40:40 2019][1696540.849817] .load : 0 [Wed Feb 6 09:40:40 2019][1696540.854610] .runnable_load_avg : 0 [Wed Feb 6 09:40:40 2019][1696540.859403] .blocked_load_avg : 0 [Wed Feb 6 09:40:41 2019][1696540.864195] .tg_load_avg : 0 [Wed Feb 6 09:40:41 2019][1696540.868989] .tg_load_contrib : 0 [Wed Feb 6 09:40:41 2019][1696540.873782] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:41 2019][1696540.878576] .tg->runnable_avg : 0 [Wed Feb 6 09:40:41 2019][1696540.883368] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:41 2019][1696540.888160] .throttled : 0 [Wed Feb 6 09:40:41 2019][1696540.892952] .throttle_count : 0 [Wed Feb 6 09:40:41 2019][1696540.897744] .avg->runnable_avg_sum : 215 [Wed Feb 6 09:40:41 2019][1696540.902711] .avg->runnable_avg_period : 46742 [Wed Feb 6 09:40:41 2019][1696540.907855] [Wed Feb 6 09:40:41 2019][1696540.909524] rt_rq[10]:/ [Wed Feb 6 09:40:41 2019][1696540.912149] .rt_nr_running : 0 [Wed Feb 6 09:40:41 2019][1696540.916943] .rt_throttled : 0 [Wed Feb 6 09:40:41 2019][1696540.921736] .rt_time : 0.000000 [Wed Feb 6 09:40:41 2019][1696540.927136] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:41 2019][1696540.932794] [Wed Feb 6 09:40:41 2019][1696540.934467] runnable tasks: [Wed Feb 6 09:40:41 2019][1696540.937439] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:41 2019][1696540.948386] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:41 2019][1696540.959246] kthreadd 2 10935994.939417 15841 120 0.000000 12168.836050 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696540.970590] watchdog/10 61 -11.979791 424079 0 0.000000 141168.081928 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696540.981927] migration/10 62 0.000000 2261 0 0.000000 544656.457879 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696540.993264] ksoftirqd/10 63 10935998.945607 5133917 120 0.000000 68266.829321 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.004599] kworker/10:0H 65 4496.719833 10 100 0.000000 0.032881 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.016002] chronyd 17580 10803147.483325 4943 120 0.000000 248.434038 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.027335] kdmwork-253:3 18449 10936001.156151 155913914 120 0.000000 1968660.617794 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.038670] kworker/10:1H 18727 10935998.929682 4406233 100 0.000000 22883.341397 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.050006] sshd 19077 10935879.787720 457 120 0.000000 82.245492 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.061342] writer#1 19093 10936001.182744 18667726 120 0.000000 197233.378932 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.072692] kiblnd_connd 5873 10936000.940213 1036002 120 0.000000 20561.527583 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.084025] kiblnd_sd_02_01 5884 13315540.248861 1595779007 120 0.000000 10393251.990981 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.095445] kiblnd_sd_02_02 5885 10936013.489598 1595558904 120 0.000000 10414412.593033 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.106869] kiblnd_sd_02_03 5886 13315540.307199 1595593852 120 0.000000 10409036.585871 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.118301] ptlrpcd_02_06 5926 10936000.983826 13160346 120 0.000000 188409.009452 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.129638] ptlrpcd_02_09 5929 10936001.077265 13206949 120 0.000000 184846.859519 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.141025] ptlrpcd_02_10 5930 10935999.635685 13228189 120 0.000000 190720.552873 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.152369] ldlm_cn02_002 6037 10803288.437319 22270098 120 0.000000 651911.486744 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.163720] osp-pre-42-0 6267 10936000.943920 247821 120 0.000000 4513.842918 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.175061] osp-pre-31-2 6455 10935999.655523 235001 120 0.000000 3514.880796 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.186396] osp-syn-42-2 6456 10935989.210778 437807 120 0.000000 12498.135491 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.197733] osp-syn-18-2 6466 10935910.978943 444091 120 0.000000 12305.467556 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.209078] mdt02_005 6688 10920582.918724 96108560 120 0.000000 4176759.538152 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.220420] mdt_rdpg02_002 6854 10935999.003995 18771634 120 0.000000 728532.490030 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.231775] mdt02_018 9020 10347765.681735 85137994 120 0.000000 3676298.703342 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.243132] mdt_rdpg02_006 23025 10406862.442366 20718324 120 0.000000 820579.553637 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.254477] kworker/10:0 7757 10936001.506887 38961 120 0.000000 3726.632697 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.265888] mdt02_038 15266 10161656.520577 586522 120 0.000000 474395.029446 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.277410] mdt02_041 15329 10935910.866281 4300731 120 0.000000 931618.173424 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.288821] mdt02_043 15331 10935999.691963 12430868 120 0.000000 862307.297777 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.300165] ldlm_bl_04 15961 10935890.751917 2325087 120 0.000000 365893.527561 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.311505] ldlm_bl_10 15967 10935911.869793 2326468 120 0.000000 354921.866245 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.322854] mdt_io02_041 27683 10935999.150968 20329 120 0.000000 1597.071351 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.334201] mdt_io02_060 27739 10159822.183215 18805 120 0.000000 1639.773393 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.345536] mdt_io02_062 27741 10159288.176897 196 120 0.000000 8.651642 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.356880] mdt_io02_083 27774 10935888.472728 20224 120 0.000000 2412.949408 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.368227] ll_mgs_0010 28978 10163902.165233 252 120 0.000000 775.625548 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.379571] kworker/10:1 29787 10791893.847341 9 120 0.000000 0.045536 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.390911] sshd 30345 10935700.130056 26641 120 0.000000 450.433491 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.402246] sshd 30459 10935898.689435 13 120 0.000000 10.123528 0.000000 2 / [Wed Feb 6 09:40:41 2019][1696541.413601] [Wed Feb 6 09:40:41 2019][1696541.415278] cpu#11, 1996.199 MHz [Wed Feb 6 09:40:41 2019][1696541.418703] .nr_running : 0 [Wed Feb 6 09:40:41 2019][1696541.423512] .load : 0 [Wed Feb 6 09:40:41 2019][1696541.428316] .nr_switches : 634537010 [Wed Feb 6 09:40:41 2019][1696541.433809] .nr_load_updates : 222209100 [Wed Feb 6 09:40:41 2019][1696541.439302] .nr_uninterruptible : -24451 [Wed Feb 6 09:40:41 2019][1696541.444541] .next_balance : 5991.165330 [Wed Feb 6 09:40:41 2019][1696541.450204] .curr->pid : 0 [Wed Feb 6 09:40:41 2019][1696541.455000] .clock : 1696541454.508109 [Wed Feb 6 09:40:41 2019][1696541.461189] .cpu_load[0] : 0 [Wed Feb 6 09:40:41 2019][1696541.465987] .cpu_load[1] : 0 [Wed Feb 6 09:40:41 2019][1696541.470787] .cpu_load[2] : 0 [Wed Feb 6 09:40:41 2019][1696541.475585] .cpu_load[3] : 0 [Wed Feb 6 09:40:41 2019][1696541.480383] .cpu_load[4] : 0 [Wed Feb 6 09:40:41 2019][1696541.485188] .avg_idle : 5894514 [Wed Feb 6 09:40:41 2019][1696541.490513] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:41 2019][1696541.495836] [Wed Feb 6 09:40:41 2019][1696541.497513] cfs_rq[11]:/ [Wed Feb 6 09:40:41 2019][1696541.500224] .exec_clock : 0.000000 [Wed Feb 6 09:40:41 2019][1696541.505624] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:41 2019][1696541.511024] .min_vruntime : 29476622.606500 [Wed Feb 6 09:40:41 2019][1696541.517031] .max_vruntime : 0.000001 [Wed Feb 6 09:40:41 2019][1696541.522430] .spread : 0.000000 [Wed Feb 6 09:40:41 2019][1696541.527829] .spread0 : 1351039.457529 [Wed Feb 6 09:40:41 2019][1696541.533748] .nr_spread_over : 0 [Wed Feb 6 09:40:41 2019][1696541.538539] .nr_running : 0 [Wed Feb 6 09:40:41 2019][1696541.543334] .load : 0 [Wed Feb 6 09:40:41 2019][1696541.548127] .runnable_load_avg : 0 [Wed Feb 6 09:40:41 2019][1696541.552936] .blocked_load_avg : 0 [Wed Feb 6 09:40:41 2019][1696541.557730] .tg_load_avg : 0 [Wed Feb 6 09:40:41 2019][1696541.562522] .tg_load_contrib : 0 [Wed Feb 6 09:40:41 2019][1696541.567314] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:41 2019][1696541.572108] .tg->runnable_avg : 0 [Wed Feb 6 09:40:41 2019][1696541.576901] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:41 2019][1696541.581694] .throttled : 0 [Wed Feb 6 09:40:41 2019][1696541.586485] .throttle_count : 0 [Wed Feb 6 09:40:41 2019][1696541.591278] .avg->runnable_avg_sum : 82 [Wed Feb 6 09:40:41 2019][1696541.596157] .avg->runnable_avg_period : 47796 [Wed Feb 6 09:40:41 2019][1696541.601301] [Wed Feb 6 09:40:41 2019][1696541.602971] rt_rq[11]:/ [Wed Feb 6 09:40:41 2019][1696541.605596] .rt_nr_running : 0 [Wed Feb 6 09:40:41 2019][1696541.610389] .rt_throttled : 0 [Wed Feb 6 09:40:41 2019][1696541.615181] .rt_time : 0.000000 [Wed Feb 6 09:40:41 2019][1696541.620581] .rt_runtime : 782.052281 [Wed Feb 6 09:40:41 2019][1696541.626154] [Wed Feb 6 09:40:41 2019][1696541.627827] runnable tasks: [Wed Feb 6 09:40:41 2019][1696541.630800] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:41 2019][1696541.641746] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:41 2019][1696541.652613] watchdog/11 66 -11.980223 424086 0 0.000000 76225.051715 0.000000 3 / [Wed Feb 6 09:40:41 2019][1696541.663949] migration/11 67 0.000000 2825 0 0.000000 557379.349647 0.000000 3 / [Wed Feb 6 09:40:41 2019][1696541.675286] ksoftirqd/11 68 29476593.045933 4248316 120 0.000000 43836.127053 0.000000 3 / [Wed Feb 6 09:40:41 2019][1696541.686632] kworker/11:0H 70 4796.449426 10 100 0.000000 0.040397 0.000000 3 / [Wed Feb 6 09:40:41 2019][1696541.698016] rpcbind 17518 29476565.838186 56493 120 0.000000 2135.004214 0.000000 3 / [Wed Feb 6 09:40:41 2019][1696541.709351] sssd_pam 17602 29476599.407493 339111 120 0.000000 6276.596201 0.000000 3 / [Wed Feb 6 09:40:41 2019][1696541.720689] in:imjournal 19089 29476622.606500 2007963 120 0.000000 45798.290802 0.000000 3 / [Wed Feb 6 09:40:41 2019][1696541.732021] reader#1 19099 29476589.998642 189961 120 0.000000 17123.862494 0.000000 3 / [Wed Feb 6 09:40:41 2019][1696541.743356] reader#3 19101 29476600.325887 173616 120 0.000000 34374.620790 0.000000 3 / [Wed Feb 6 09:40:41 2019][1696541.754693] crond 19272 29476585.601067 59962 120 0.000000 9599.986172 0.000000 3 / [Wed Feb 6 09:40:41 2019][1696541.766030] slapd 84636 29476389.449015 609 120 0.000000 1130.174592 0.000000 3 / [Wed Feb 6 09:40:41 2019][1696541.777367] kworker/11:1H 29734 29476593.033862 5089504 100 0.000000 17394.986944 0.000000 3 / [Wed Feb 6 09:40:41 2019][1696541.788707] ptlrpc_hr03_000 5855 29476607.982872 20346148 120 0.000000 179189.751772 0.000000 3 / [Wed Feb 6 09:40:41 2019][1696541.800050] ptlrpcd_03_05 5937 29476610.609135 13118784 120 0.000000 196923.569394 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.811391] ptlrpcd_03_11 5943 29476606.276780 13090481 120 0.000000 196010.546330 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.822739] mdt_rdpg03_000 6185 29476610.663989 7438105 120 0.000000 298805.118454 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.834080] osp-pre-18-0 6277 29476608.427688 247824 120 0.000000 6760.868401 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.845422] osp-syn-47-0 6314 29476481.110379 778279 120 0.000000 20686.214275 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.856758] lquota_wb_fir-M 6416 29476415.165567 17158 120 0.000000 2761.231827 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.868094] osp-pre-11-2 6449 29476599.925046 234859 120 0.000000 7187.586419 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.879429] osp-syn-21-2 6480 29476452.987996 440291 120 0.000000 12255.539030 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.890762] osp-pre-41-2 6485 29476601.862861 235020 120 0.000000 3291.626330 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.902099] osp-pre-20-2 6493 29476604.020683 235076 120 0.000000 4554.642668 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.913450] mdt_io03_003 6862 29476597.455445 1936121 120 0.000000 69895.425082 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.924811] mdt_io03_008 12779 28942576.193551 2199509 120 0.000000 75575.665351 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.936155] ldlm_cn03_007 22118 29476588.435401 6697431 120 0.000000 300201.823807 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.947515] ldlm_cn03_010 23311 29476616.110941 11563377 120 0.000000 495455.040910 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.958855] mdt_io03_014 35138 28942576.182589 1578176 120 0.000000 54275.951614 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.970189] ldlm_cn03_016 63247 29362875.481549 3776552 120 0.000000 295666.786553 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.981525] kworker/u260:0 14084 28877790.930921 5 120 0.000000 0.580772 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696541.992876] ldlm_bl_12 15969 29476464.255978 2292016 120 0.000000 741621.831846 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696542.004212] ldlm_bl_20 15983 29476478.632269 2326260 120 0.000000 18919.102359 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696542.015552] mdt_io03_015 17097 28942576.177449 188744 120 0.000000 6710.720148 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696542.026888] mdt_rdpg03_007 17450 29260073.484232 269352 120 0.000000 10166.190196 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696542.038223] kworker/11:2 17937 29476608.407219 49334 120 0.000000 5367.931999 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696542.049559] mdt03_034 24923 29476593.074818 1006293 120 0.000000 23267.130979 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696542.060916] kworker/u260:1 28593 29476601.660355 1497 120 0.000000 320.477496 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696542.072275] kworker/11:1 28842 28888353.058840 4 120 0.000000 0.020919 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696542.083611] ll_mgs_0006 28957 28945583.463358 353 120 0.000000 83.165267 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696542.094948] ll_mgs_0015 28983 28945769.806623 1172 120 0.000000 1315.439997 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696542.106284] ll_mgs_0030 29001 28945580.874365 597 120 0.000000 1154.294344 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696542.117620] pickup 29530 29476458.675640 54 120 0.000000 9.883525 0.000000 3 / [Wed Feb 6 09:40:42 2019][1696542.128978] [Wed Feb 6 09:40:42 2019][1696542.130662] cpu#12, 1996.199 MHz [Wed Feb 6 09:40:42 2019][1696542.134077] .nr_running : 0 [Wed Feb 6 09:40:42 2019][1696542.138879] .load : 0 [Wed Feb 6 09:40:42 2019][1696542.143680] .nr_switches : 4135603211 [Wed Feb 6 09:40:42 2019][1696542.149260] .nr_load_updates : 532037440 [Wed Feb 6 09:40:42 2019][1696542.154757] .nr_uninterruptible : 526785 [Wed Feb 6 09:40:42 2019][1696542.159989] .next_balance : 5991.166042 [Wed Feb 6 09:40:42 2019][1696542.165657] .curr->pid : 0 [Wed Feb 6 09:40:42 2019][1696542.170462] .clock : 1696542169.526331 [Wed Feb 6 09:40:42 2019][1696542.176649] .cpu_load[0] : 0 [Wed Feb 6 09:40:42 2019][1696542.181448] .cpu_load[1] : 4 [Wed Feb 6 09:40:42 2019][1696542.186255] .cpu_load[2] : 6 [Wed Feb 6 09:40:42 2019][1696542.191053] .cpu_load[3] : 10 [Wed Feb 6 09:40:42 2019][1696542.195943] .cpu_load[4] : 11 [Wed Feb 6 09:40:42 2019][1696542.200830] .avg_idle : 1609133 [Wed Feb 6 09:40:42 2019][1696542.206167] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:42 2019][1696542.211488] [Wed Feb 6 09:40:42 2019][1696542.213163] cfs_rq[12]:/ [Wed Feb 6 09:40:42 2019][1696542.215876] .exec_clock : 0.000000 [Wed Feb 6 09:40:42 2019][1696542.221275] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:42 2019][1696542.226675] .min_vruntime : 24101099.773753 [Wed Feb 6 09:40:42 2019][1696542.232681] .max_vruntime : 0.000001 [Wed Feb 6 09:40:42 2019][1696542.238081] .spread : 0.000000 [Wed Feb 6 09:40:42 2019][1696542.243480] .spread0 : -4024485.867023 [Wed Feb 6 09:40:42 2019][1696542.249486] .nr_spread_over : 0 [Wed Feb 6 09:40:42 2019][1696542.254279] .nr_running : 0 [Wed Feb 6 09:40:42 2019][1696542.259071] .load : 0 [Wed Feb 6 09:40:42 2019][1696542.263864] .runnable_load_avg : 0 [Wed Feb 6 09:40:42 2019][1696542.268657] .blocked_load_avg : 0 [Wed Feb 6 09:40:42 2019][1696542.273450] .tg_load_avg : 0 [Wed Feb 6 09:40:42 2019][1696542.278242] .tg_load_contrib : 0 [Wed Feb 6 09:40:42 2019][1696542.283035] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:42 2019][1696542.287828] .tg->runnable_avg : 0 [Wed Feb 6 09:40:42 2019][1696542.292621] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:42 2019][1696542.297413] .throttled : 0 [Wed Feb 6 09:40:42 2019][1696542.302205] .throttle_count : 0 [Wed Feb 6 09:40:42 2019][1696542.306997] .avg->runnable_avg_sum : 83 [Wed Feb 6 09:40:42 2019][1696542.311878] .avg->runnable_avg_period : 47984 [Wed Feb 6 09:40:42 2019][1696542.317021] [Wed Feb 6 09:40:42 2019][1696542.318691] rt_rq[12]:/ [Wed Feb 6 09:40:42 2019][1696542.321316] .rt_nr_running : 0 [Wed Feb 6 09:40:42 2019][1696542.326109] .rt_throttled : 0 [Wed Feb 6 09:40:42 2019][1696542.330902] .rt_time : 0.000000 [Wed Feb 6 09:40:42 2019][1696542.336327] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:42 2019][1696542.341986] [Wed Feb 6 09:40:42 2019][1696542.343659] runnable tasks: [Wed Feb 6 09:40:42 2019][1696542.346633] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:42 2019][1696542.357578] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:42 2019][1696542.368448] watchdog/12 71 -11.981395 424085 0 0.000000 94260.572054 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.379783] migration/12 72 0.000000 2554 0 0.000000 593338.015079 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.391119] ksoftirqd/12 73 24101087.271822 12614862 120 0.000000 133886.997233 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.402455] kworker/12:0H 75 4635.933920 10 100 0.000000 0.037060 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.413830] ccp-3-q4 10636 2154.876074 2 120 0.000000 0.007804 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.425180] kworker/12:1H 20590 24101087.256412 12925764 100 0.000000 55262.911612 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.436526] kiblnd_sd_00_00 5875 24101100.172065 4155051399 120 0.000000 27449926.521694 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.447955] ptlrpcd_00_03 5899 24101084.216328 6055788 120 0.000000 87762.641199 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.459294] ptlrpcd_00_08 5904 24101074.452700 6015677 120 0.000000 88293.560914 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.470665] ldlm_cn00_000 6029 24099377.792281 13585670 120 0.000000 601834.522891 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.482056] jbd2/dm-4-8 6150 24101087.417249 33743379 120 0.000000 2834445.905996 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.493391] mdt_rdpg00_000 6179 24101087.730692 64309788 120 0.000000 1476072.311125 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.504732] osp-syn-19-0 6274 24100547.408309 782342 120 0.000000 21130.910065 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.516072] osp-syn-24-0 6312 24100363.794605 776586 120 0.000000 21136.288788 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.527408] osp-pre-37-0 6331 24101044.993398 246748 120 0.000000 5650.998690 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.538747] osp-pre-26-2 6453 24101083.962422 234993 120 0.000000 6664.525098 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.550092] ldlm_cn00_004 6765 24100638.966813 18506467 120 0.000000 593309.376702 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.561426] mdt_out00_002 6784 24101044.945196 38075034 120 0.000000 871611.972273 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.572766] mdt00_010 7191 24101073.550648 255406267 120 0.000000 8832520.419251 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.584111] mdt_io00_012 8960 24086803.854783 4522040 120 0.000000 184173.913782 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.595447] mdt00_021 9028 24101095.932906 221250201 120 0.000000 7197506.180358 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.606852] mdt00_022 9033 24101072.340346 237862849 120 0.000000 7732442.407883 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.618205] mdt_io00_022 10285 24086803.854732 6690092 120 0.000000 285382.219102 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.629538] ldlm_cn00_014 14885 24099134.603022 10877585 120 0.000000 546612.778913 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.640881] ldlm_cn00_026 20331 24094374.714513 12245605 120 0.000000 565053.458237 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.652217] ldlm_cn00_027 20464 24099378.696269 9852240 120 0.000000 551736.474144 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.663554] ldlm_cn00_028 20558 24100930.210287 11536791 120 0.000000 552166.113288 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.674897] ldlm_cn00_031 21243 24099390.476665 8206272 120 0.000000 448349.892310 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.686245] kworker/12:0 38269 24101087.358115 419099 120 0.000000 9547.163071 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.697586] mdt00_032 15218 24101072.350455 10590345 120 0.000000 745765.093278 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.708929] mdt00_040 15277 24101084.348942 7094870 120 0.000000 510949.795096 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.720268] mdt_io00_033 15464 23696905.647702 893 120 0.000000 156.585850 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.731610] ldlm_bl_17 15974 24100990.391524 2324946 120 0.000000 324103.955961 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.742954] mdt_rdpg00_008 17028 24101072.679519 1712836 120 0.000000 57372.908801 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.754287] mdt_rdpg00_010 17030 24101080.655662 411497 120 0.000000 15066.093136 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.765630] kworker/12:1 18456 21466961.390450 3 120 0.000000 0.014687 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.776970] mdt_io00_049 27703 24066901.298375 199 120 0.000000 50.999218 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.788306] mdt_io00_052 27708 24084447.150145 307 120 0.000000 14.485792 0.000000 0 / [Wed Feb 6 09:40:42 2019][1696542.799689] [Wed Feb 6 09:40:42 2019][1696542.801365] cpu#13, 1996.199 MHz [Wed Feb 6 09:40:42 2019][1696542.804783] .nr_running : 0 [Wed Feb 6 09:40:42 2019][1696542.809584] .load : 0 [Wed Feb 6 09:40:42 2019][1696542.814383] .nr_switches : 3050452520 [Wed Feb 6 09:40:42 2019][1696542.819966] .nr_load_updates : 422523998 [Wed Feb 6 09:40:42 2019][1696542.825461] .nr_uninterruptible : -99661 [Wed Feb 6 09:40:42 2019][1696542.830695] .next_balance : 5991.166713 [Wed Feb 6 09:40:42 2019][1696542.836364] .curr->pid : 0 [Wed Feb 6 09:40:42 2019][1696542.841166] .clock : 1696542840.542327 [Wed Feb 6 09:40:42 2019][1696542.847353] .cpu_load[0] : 0 [Wed Feb 6 09:40:42 2019][1696542.852151] .cpu_load[1] : 0 [Wed Feb 6 09:40:42 2019][1696542.856944] .cpu_load[2] : 0 [Wed Feb 6 09:40:42 2019][1696542.861750] .cpu_load[3] : 0 [Wed Feb 6 09:40:42 2019][1696542.866554] .cpu_load[4] : 0 [Wed Feb 6 09:40:42 2019][1696542.871361] .avg_idle : 949167 [Wed Feb 6 09:40:43 2019][1696542.876663] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:43 2019][1696542.881985] [Wed Feb 6 09:40:43 2019][1696542.883660] cfs_rq[13]:/ [Wed Feb 6 09:40:43 2019][1696542.886370] .exec_clock : 0.000000 [Wed Feb 6 09:40:43 2019][1696542.891805] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:43 2019][1696542.897240] .min_vruntime : 15570991.193753 [Wed Feb 6 09:40:43 2019][1696542.903347] .max_vruntime : 0.000001 [Wed Feb 6 09:40:43 2019][1696542.908837] .spread : 0.000000 [Wed Feb 6 09:40:43 2019][1696542.914271] .spread0 : -12554596.794394 [Wed Feb 6 09:40:43 2019][1696542.920364] .nr_spread_over : 0 [Wed Feb 6 09:40:43 2019][1696542.925155] .nr_running : 0 [Wed Feb 6 09:40:43 2019][1696542.929948] .load : 0 [Wed Feb 6 09:40:43 2019][1696542.934741] .runnable_load_avg : 0 [Wed Feb 6 09:40:43 2019][1696542.939534] .blocked_load_avg : 2 [Wed Feb 6 09:40:43 2019][1696542.944328] .tg_load_avg : 0 [Wed Feb 6 09:40:43 2019][1696542.949120] .tg_load_contrib : 0 [Wed Feb 6 09:40:43 2019][1696542.953913] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:43 2019][1696542.958705] .tg->runnable_avg : 0 [Wed Feb 6 09:40:43 2019][1696542.963497] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:43 2019][1696542.968291] .throttled : 0 [Wed Feb 6 09:40:43 2019][1696542.973082] .throttle_count : 0 [Wed Feb 6 09:40:43 2019][1696542.977876] .avg->runnable_avg_sum : 121 [Wed Feb 6 09:40:43 2019][1696542.982842] .avg->runnable_avg_period : 47860 [Wed Feb 6 09:40:43 2019][1696542.987985] [Wed Feb 6 09:40:43 2019][1696542.989654] rt_rq[13]:/ [Wed Feb 6 09:40:43 2019][1696542.992280] .rt_nr_running : 0 [Wed Feb 6 09:40:43 2019][1696542.997074] .rt_throttled : 0 [Wed Feb 6 09:40:43 2019][1696543.001866] .rt_time : 0.000000 [Wed Feb 6 09:40:43 2019][1696543.007266] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:43 2019][1696543.012925] [Wed Feb 6 09:40:43 2019][1696543.014598] runnable tasks: [Wed Feb 6 09:40:43 2019][1696543.017569] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:43 2019][1696543.028516] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:43 2019][1696543.039394] watchdog/13 76 -11.979992 424088 0 0.000000 166366.092741 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.050896] migration/13 77 0.000000 2398 0 0.000000 481788.840662 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.062277] ksoftirqd/13 78 15570974.902456 14638027 120 0.000000 140597.216737 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.073621] kworker/13:0H 80 12444.483040 10 100 0.000000 0.037240 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.085001] gdbus 17529 15570910.585831 434636 120 0.000000 19641.027049 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.096501] dbus-daemon 17512 15570911.902042 1241844 120 0.000000 122281.599103 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.108002] kworker/13:1H 54683 15570974.873604 11507474 100 0.000000 56013.823862 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.119408] cfs_rh_03 5816 15456137.488441 1227 120 0.000000 1705.298562 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.130874] ptlrpc_hr01_002 5843 15570914.890154 121145607 120 0.000000 1151440.937687 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.142366] ptlrpc_hr01_005 5847 15570966.876634 121132056 120 0.000000 1154732.745625 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.153705] kiblnd_sd_01_03 5882 15570991.808544 2884492872 120 0.000000 21187877.118212 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.165130] ptlrpcd_01_10 5918 15570973.671784 6949781 120 0.000000 109195.020101 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.176486] mdt_fld_0001 6208 15279486.291593 1532 120 0.000000 15.682014 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.187841] osp-pre-25-2 6469 15570972.050420 234954 120 0.000000 6883.848453 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.199177] osp-syn-32-2 6478 15570824.312582 436031 120 0.000000 12083.885637 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.210515] osp-syn-6-2 6500 15570785.910340 444814 120 0.000000 12391.197640 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.221851] osp-pre-8-2 6525 15570972.071349 235010 120 0.000000 3371.745468 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.233189] mdt01_004 6618 15258787.804801 100577588 120 0.000000 3867460.384747 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.244523] mdt01_005 6683 15570985.207371 112880840 120 0.000000 4529076.053618 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.255862] mdt_io01_004 6783 14766180.689410 6284537 120 0.000000 230241.440212 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.267203] mdt_io01_011 7120 14765656.989426 4258500 120 0.000000 156960.030604 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.278555] mdt01_036 9185 15258787.804992 107860322 120 0.000000 4150757.935794 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.289893] mdt_io01_024 9190 14765653.525961 9301267 120 0.000000 345440.149428 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.301231] kworker/13:2 10873 15570976.810287 495882 120 0.000000 20879.527113 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.312565] mdt_io01_030 11093 14766180.345594 4775564 120 0.000000 174162.560876 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.323905] mdt_io01_036 15118 15254107.499698 3464263 120 0.000000 137159.432357 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.335244] mdt_io01_039 20860 15570976.463105 8642645 120 0.000000 326757.639170 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.346585] mdt_io01_043 20864 14765657.155752 966809 120 0.000000 46993.159821 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.357919] mdt_io01_048 20869 15250844.415367 305988 120 0.000000 13710.535552 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.369256] mdt_io01_055 20876 14765656.527900 690022 120 0.000000 30244.933336 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.380593] ldlm_cn01_014 21074 15523731.112189 17241859 120 0.000000 583871.312097 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.391928] ldlm_cn01_016 21076 15570979.151021 22248139 120 0.000000 672070.795802 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.403271] ldlm_cn01_031 28334 15570426.525872 7837733 120 0.000000 255398.001701 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.414619] mdt01_048 15224 15570981.252470 2870505 120 0.000000 266442.780685 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.425955] mdt01_057 15248 15570939.936337 1823372 120 0.000000 363523.631116 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.437294] mdt01_069 15383 15258787.804109 473527 120 0.000000 19479.041392 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.448627] mdt01_076 15393 15570818.649423 2916493 120 0.000000 409917.723822 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.459967] mdt_io01_080 15461 15254107.464911 1033 120 0.000000 41.042783 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.471301] ldlm_bl_21 15984 15570725.924005 2324387 120 0.000000 18830.468266 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.482648] mdt_io01_082 27678 14765657.703713 439 120 0.000000 17.906365 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.493981] mdt_io01_084 27684 15258186.290937 13673 120 0.000000 836.942704 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.505317] mdt_io01_085 27685 14765653.490413 421 120 0.000000 16.456147 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.516656] mdt_io01_092 27698 14765656.573443 526 120 0.000000 20.059469 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.527998] mdt_io01_107 27755 14765653.430780 561 120 0.000000 23.050294 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.539344] kworker/u258:2 28943 15570975.024598 1483 120 0.000000 326.480642 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.550686] kworker/13:0 29471 15372715.927481 6 120 0.000000 0.048503 0.000000 1 / [Wed Feb 6 09:40:43 2019][1696543.562049] [Wed Feb 6 09:40:43 2019][1696543.563731] cpu#14, 1996.199 MHz [Wed Feb 6 09:40:43 2019][1696543.567147] .nr_running : 5 [Wed Feb 6 09:40:43 2019][1696543.571946] .load : 3072 [Wed Feb 6 09:40:43 2019][1696543.577010] .nr_switches : 1294771762 [Wed Feb 6 09:40:43 2019][1696543.582590] .nr_load_updates : 348097548 [Wed Feb 6 09:40:43 2019][1696543.588086] .nr_uninterruptible : -124332 [Wed Feb 6 09:40:43 2019][1696543.593408] .next_balance : 5991.167571 [Wed Feb 6 09:40:43 2019][1696543.599076] .curr->pid : 30347 [Wed Feb 6 09:40:43 2019][1696543.604224] .clock : 1696401740.004441 [Wed Feb 6 09:40:43 2019][1696543.610412] .cpu_load[0] : 3072 [Wed Feb 6 09:40:43 2019][1696543.615471] .cpu_load[1] : 3072 [Wed Feb 6 09:40:43 2019][1696543.620532] .cpu_load[2] : 3072 [Wed Feb 6 09:40:43 2019][1696543.625596] .cpu_load[3] : 3072 [Wed Feb 6 09:40:43 2019][1696543.630657] .cpu_load[4] : 3072 [Wed Feb 6 09:40:43 2019][1696543.635719] .avg_idle : 6709672 [Wed Feb 6 09:40:43 2019][1696543.641040] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:43 2019][1696543.646358] [Wed Feb 6 09:40:43 2019][1696543.648028] cfs_rq[14]:/ [Wed Feb 6 09:40:43 2019][1696543.650738] .exec_clock : 0.000000 [Wed Feb 6 09:40:43 2019][1696543.656137] .MIN_vruntime : 14027240.339023 [Wed Feb 6 09:40:43 2019][1696543.662143] .min_vruntime : 14027252.339023 [Wed Feb 6 09:40:43 2019][1696543.668152] .max_vruntime : 14027240.339023 [Wed Feb 6 09:40:43 2019][1696543.674158] .spread : 0.000000 [Wed Feb 6 09:40:43 2019][1696543.679556] .spread0 : -14098338.253283 [Wed Feb 6 09:40:43 2019][1696543.685648] .nr_spread_over : 0 [Wed Feb 6 09:40:43 2019][1696543.690443] .nr_running : 3 [Wed Feb 6 09:40:43 2019][1696543.695236] .load : 3072 [Wed Feb 6 09:40:43 2019][1696543.700289] .runnable_load_avg : 149 [Wed Feb 6 09:40:43 2019][1696543.705252] .blocked_load_avg : 0 [Wed Feb 6 09:40:43 2019][1696543.710047] .tg_load_avg : 0 [Wed Feb 6 09:40:43 2019][1696543.714838] .tg_load_contrib : 0 [Wed Feb 6 09:40:43 2019][1696543.719631] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:43 2019][1696543.724424] .tg->runnable_avg : 0 [Wed Feb 6 09:40:43 2019][1696543.729217] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:43 2019][1696543.734011] .throttled : 0 [Wed Feb 6 09:40:43 2019][1696543.738803] .throttle_count : 0 [Wed Feb 6 09:40:43 2019][1696543.743595] .avg->runnable_avg_sum : 7031 [Wed Feb 6 09:40:43 2019][1696543.748648] .avg->runnable_avg_period : 46304 [Wed Feb 6 09:40:43 2019][1696543.753793] [Wed Feb 6 09:40:43 2019][1696543.755460] rt_rq[14]:/ [Wed Feb 6 09:40:43 2019][1696543.758086] .rt_nr_running : 1 [Wed Feb 6 09:40:43 2019][1696543.762878] .rt_throttled : 0 [Wed Feb 6 09:40:43 2019][1696543.767673] .rt_time : 0.000000 [Wed Feb 6 09:40:43 2019][1696543.773072] .rt_runtime : 979.166667 [Wed Feb 6 09:40:43 2019][1696543.778645] [Wed Feb 6 09:40:43 2019][1696543.780317] runnable tasks: [Wed Feb 6 09:40:43 2019][1696543.783289] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:43 2019][1696543.794235] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:43 2019][1696543.805106] watchdog/14 81 -11.978960 424034 0 0.000000 147942.410883 0.000000 2 / [Wed Feb 6 09:40:43 2019][1696543.816440] migration/14 82 0.000000 2560 0 0.000000 628714.403002 0.000000 2 / [Wed Feb 6 09:40:43 2019][1696543.827778] ksoftirqd/14 83 14027240.339023 15499494 120 0.000000 151140.096711 0.000000 2 / [Wed Feb 6 09:40:43 2019][1696543.839112] kworker/14:0H 85 5200.608710 10 100 0.000000 0.042721 0.000000 2 / [Wed Feb 6 09:40:43 2019][1696543.850485] ccp-1-q3 10469 2707.992529 2 120 0.000000 0.012534 0.000000 2 / [Wed Feb 6 09:40:43 2019][1696543.861842] kworker/14:1H 27914 14027223.528299 13942182 100 0.000000 69754.013746 0.000000 2 / [Wed Feb 6 09:40:44 2019][1696543.873177] cfs_rh_01 5814 13862683.123165 1232 120 0.000000 1580.876694 0.000000 2 / [Wed Feb 6 09:40:44 2019][1696543.884548] osp-syn-18-0 6276 14027028.110560 784025 120 0.000000 20562.748701 0.000000 2 / [Wed Feb 6 09:40:44 2019][1696543.895882] osp-syn-17-0 6280 14027028.024173 781814 120 0.000000 21211.182821 0.000000 2 / [Wed Feb 6 09:40:44 2019][1696543.907250] ldlm_cb02_002 7108 13383410.891579 11320964 120 0.000000 210092.503961 0.000000 2 / [Wed Feb 6 09:40:44 2019][1696543.918627] ldlm_cn02_023 28934 13869198.046826 12633682 120 0.000000 499806.866933 0.000000 2 / [Wed Feb 6 09:40:44 2019][1696543.929962] mdt_rdpg02_008 50796 13563573.400992 5669519 120 0.000000 242113.301205 0.000000 2 / [Wed Feb 6 09:40:44 2019][1696543.941300] kworker/14:1 122079 14027240.339023 77914 120 0.000000 5908.660006 0.000000 2 / [Wed Feb 6 09:40:44 2019][1696543.952727] mdt02_045 15371 13869222.499232 11157364 120 0.000000 510888.098319 0.000000 2 / [Wed Feb 6 09:40:44 2019][1696543.964092] mdt_io02_056 27732 13563833.061514 6082 120 0.000000 2022.273137 0.000000 2 / [Wed Feb 6 09:40:44 2019][1696543.975437] ldlm_cn02_031 28949 13869202.434906 83849 120 0.000000 10443.146979 0.000000 2 / [Wed Feb 6 09:40:44 2019][1696543.986777] kworker/14:2 29068 13427737.185361 25 120 0.000000 0.164981 0.000000 2 / [Wed Feb 6 09:40:44 2019][1696543.998134] kworker/u259:0 29378 13869189.781469 883 120 0.000000 135.541356 0.000000 2 / [Wed Feb 6 09:40:44 2019][1696544.009476] R bash 30347 14027250.150363 100 120 0.000000 22.151117 0.000000 2 / [Wed Feb 6 09:40:44 2019][1696544.020834] [Wed Feb 6 09:40:44 2019][1696544.022513] cpu#15, 1996.199 MHz [Wed Feb 6 09:40:44 2019][1696544.025928] .nr_running : 0 [Wed Feb 6 09:40:44 2019][1696544.030732] .load : 0 [Wed Feb 6 09:40:44 2019][1696544.035532] .nr_switches : 597109301 [Wed Feb 6 09:40:44 2019][1696544.041026] .nr_load_updates : 227317395 [Wed Feb 6 09:40:44 2019][1696544.046517] .nr_uninterruptible : -29543 [Wed Feb 6 09:40:44 2019][1696544.051754] .next_balance : 5991.167935 [Wed Feb 6 09:40:44 2019][1696544.057431] .curr->pid : 0 [Wed Feb 6 09:40:44 2019][1696544.062231] .clock : 1696544061.575661 [Wed Feb 6 09:40:44 2019][1696544.068419] .cpu_load[0] : 0 [Wed Feb 6 09:40:44 2019][1696544.073219] .cpu_load[1] : 0 [Wed Feb 6 09:40:44 2019][1696544.078017] .cpu_load[2] : 0 [Wed Feb 6 09:40:44 2019][1696544.082815] .cpu_load[3] : 0 [Wed Feb 6 09:40:44 2019][1696544.087617] .cpu_load[4] : 0 [Wed Feb 6 09:40:44 2019][1696544.092417] .avg_idle : 3620725 [Wed Feb 6 09:40:44 2019][1696544.097730] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:44 2019][1696544.103052] [Wed Feb 6 09:40:44 2019][1696544.104727] cfs_rq[15]:/ [Wed Feb 6 09:40:44 2019][1696544.107441] .exec_clock : 0.000000 [Wed Feb 6 09:40:44 2019][1696544.112839] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:44 2019][1696544.118239] .min_vruntime : 26751262.669495 [Wed Feb 6 09:40:44 2019][1696544.124246] .max_vruntime : 0.000001 [Wed Feb 6 09:40:44 2019][1696544.129643] .spread : 0.000000 [Wed Feb 6 09:40:44 2019][1696544.135045] .spread0 : -1374329.761146 [Wed Feb 6 09:40:44 2019][1696544.141051] .nr_spread_over : 0 [Wed Feb 6 09:40:44 2019][1696544.145844] .nr_running : 0 [Wed Feb 6 09:40:44 2019][1696544.150637] .load : 0 [Wed Feb 6 09:40:44 2019][1696544.155429] .runnable_load_avg : 0 [Wed Feb 6 09:40:44 2019][1696544.160220] .blocked_load_avg : 1 [Wed Feb 6 09:40:44 2019][1696544.165015] .tg_load_avg : 0 [Wed Feb 6 09:40:44 2019][1696544.169808] .tg_load_contrib : 0 [Wed Feb 6 09:40:44 2019][1696544.174601] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:44 2019][1696544.179394] .tg->runnable_avg : 0 [Wed Feb 6 09:40:44 2019][1696544.184187] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:44 2019][1696544.188980] .throttled : 0 [Wed Feb 6 09:40:44 2019][1696544.193771] .throttle_count : 0 [Wed Feb 6 09:40:44 2019][1696544.198563] .avg->runnable_avg_sum : 58 [Wed Feb 6 09:40:44 2019][1696544.203444] .avg->runnable_avg_period : 47514 [Wed Feb 6 09:40:44 2019][1696544.208588] [Wed Feb 6 09:40:44 2019][1696544.210255] rt_rq[15]:/ [Wed Feb 6 09:40:44 2019][1696544.212882] .rt_nr_running : 0 [Wed Feb 6 09:40:44 2019][1696544.217675] .rt_throttled : 0 [Wed Feb 6 09:40:44 2019][1696544.222466] .rt_time : 0.007624 [Wed Feb 6 09:40:44 2019][1696544.227866] .rt_runtime : 979.166769 [Wed Feb 6 09:40:44 2019][1696544.233438] [Wed Feb 6 09:40:44 2019][1696544.235111] runnable tasks: [Wed Feb 6 09:40:44 2019][1696544.238085] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:44 2019][1696544.249031] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:44 2019][1696544.259901] watchdog/15 86 -11.978409 424052 0 0.000000 139437.488663 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.271235] migration/15 87 0.000000 2512 0 0.000000 652146.999302 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.282572] ksoftirqd/15 88 26751246.447054 4387062 120 0.000000 46365.976166 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.293908] kworker/15:0H 90 4276.857152 9 100 0.000000 0.034492 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.305271] kworker/15:1 7264 26751250.611576 433368 120 0.000000 10377.208951 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.316619] kdmflush 16996 4273.837918 2 100 0.000000 0.013916 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.327951] bioset 17009 4273.842461 2 100 0.000000 0.013084 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.339288] kworker/15:1H 17116 26751246.435871 5261665 100 0.000000 18141.193687 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.350624] kdmflush 17146 4288.904103 2 100 0.000000 0.012645 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.361960] bioset 17148 4288.927484 2 100 0.000000 0.011411 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.373296] kdmflush 17261 4303.949688 2 100 0.000000 0.013014 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.384633] kdmwork-253:2 17265 26627306.143046 5100 120 0.000000 45.090711 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.395978] kdmflush 18445 4366.971780 2 100 0.000000 0.010179 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.407314] bioset 18447 4366.995483 2 100 0.000000 0.011803 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.418654] master 19477 26751127.526618 97240 120 0.000000 2555.722298 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.430003] kiblnd_sd_03_01 5888 26751263.127055 440058771 120 0.000000 3235059.890468 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.441341] lnet_discovery 5894 26751249.759189 1031334 120 0.000000 11120.160880 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.452683] ptlrpcd_03_02 5934 26751251.197548 12675290 120 0.000000 202219.208541 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.464023] ptlrpcd_03_06 5938 26751250.808699 12631379 120 0.000000 199341.467551 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.475358] kmmpd-dm-0 6009 26751246.459367 411745 120 0.000000 9152.461227 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.486698] ldlm_cn03_001 6039 26751239.886148 11384060 120 0.000000 460880.274340 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.498030] ldlm_bl_02 6042 26751126.342999 33864833 120 0.000000 819023.620799 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.509369] kmmpd-dm-4 6149 26751245.766555 411722 120 0.000000 8404.206564 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.520708] mdt_out03_000 6201 26751250.843024 442749 120 0.000000 10718.397941 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.532042] mdt_io03_002 6220 26686017.905485 2170578 120 0.000000 77207.231119 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.543377] osp_up3-0 6246 26750993.298715 365 120 0.000000 10.941152 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.554712] osp-pre-11-0 6259 26751250.141514 247832 120 0.000000 5504.445733 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.566049] osp-syn-14-0 6260 26750992.691966 783810 120 0.000000 20676.164774 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.577387] osp-pre-9-0 6269 26751250.166420 247816 120 0.000000 5875.351848 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.588722] osp-pre-16-0 6293 26751243.608980 247591 120 0.000000 3814.265137 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.600059] osp-syn-29-0 6320 26751167.952538 789954 120 0.000000 21045.002241 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.611394] osp-pre-3-0 6327 26751246.997184 247821 120 0.000000 3715.756901 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.622733] osp-pre-46-2 6445 26751243.660198 235056 120 0.000000 3335.370406 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.634070] osp-syn-23-2 6488 26751127.507732 438522 120 0.000000 12536.079254 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.645405] osp-syn-24-2 6502 26751127.469930 434336 120 0.000000 12108.899494 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.656741] osp-syn-3-2 6516 26751159.107372 434539 120 0.000000 12351.998184 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.668074] osp-pre-27-2 6519 26751246.786612 234904 120 0.000000 3338.284213 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.679416] mdt03_003 6682 26751242.838924 35935463 120 0.000000 1744782.907133 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.690751] ldlm_cn03_005 6771 26678488.301964 9922748 120 0.000000 281213.156263 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.702090] mdt03_009 6951 26542943.615271 37656249 120 0.000000 1674485.683800 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.713440] ldlm_cb03_002 8977 26718226.323204 4297652 120 0.000000 70801.052394 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.724794] ldlm_cn03_006 20593 26678484.950001 10040998 120 0.000000 554439.854295 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.736162] ldlm_bl_07 15964 26751128.958322 2327520 120 0.000000 366557.574164 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.747508] mdt_rdpg03_006 17292 26609279.326613 1690682 120 0.000000 73160.959348 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.758844] mdt_rdpg03_008 17452 26599225.007350 452231 120 0.000000 20094.954115 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.770180] mdt03_030 23853 26627475.269636 1998440 120 0.000000 46036.202179 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.781538] ldlm_cn03_027 28529 26685630.361519 11203 120 0.000000 13788.716875 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.792870] kworker/15:0 28874 26499339.421050 8 120 0.000000 0.067518 0.000000 3 / [Wed Feb 6 09:40:44 2019][1696544.804237] [Wed Feb 6 09:40:44 2019][1696544.805919] cpu#16, 1996.199 MHz [Wed Feb 6 09:40:44 2019][1696544.809337] .nr_running : 0 [Wed Feb 6 09:40:44 2019][1696544.814140] .load : 0 [Wed Feb 6 09:40:44 2019][1696544.818936] .nr_switches : 4213411948 [Wed Feb 6 09:40:44 2019][1696544.824523] .nr_load_updates : 508752576 [Wed Feb 6 09:40:44 2019][1696544.830017] .nr_uninterruptible : 546524 [Wed Feb 6 09:40:44 2019][1696544.835251] .next_balance : 5991.168717 [Wed Feb 6 09:40:44 2019][1696544.840915] .curr->pid : 0 [Wed Feb 6 09:40:44 2019][1696544.845709] .clock : 1696544845.596383 [Wed Feb 6 09:40:44 2019][1696544.851898] .cpu_load[0] : 0 [Wed Feb 6 09:40:44 2019][1696544.856698] .cpu_load[1] : 0 [Wed Feb 6 09:40:44 2019][1696544.861501] .cpu_load[2] : 0 [Wed Feb 6 09:40:44 2019][1696544.866300] .cpu_load[3] : 0 [Wed Feb 6 09:40:44 2019][1696544.871095] .cpu_load[4] : 0 [Wed Feb 6 09:40:45 2019][1696544.875896] .avg_idle : 3912773 [Wed Feb 6 09:40:45 2019][1696544.881217] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:45 2019][1696544.886537] [Wed Feb 6 09:40:45 2019][1696544.888206] cfs_rq[16]:/ [Wed Feb 6 09:40:45 2019][1696544.890919] .exec_clock : 0.000000 [Wed Feb 6 09:40:45 2019][1696544.896316] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:45 2019][1696544.901717] .min_vruntime : 24701527.099744 [Wed Feb 6 09:40:45 2019][1696544.907724] .max_vruntime : 0.000001 [Wed Feb 6 09:40:45 2019][1696544.913123] .spread : 0.000000 [Wed Feb 6 09:40:45 2019][1696544.918522] .spread0 : -3424067.956997 [Wed Feb 6 09:40:45 2019][1696544.924530] .nr_spread_over : 0 [Wed Feb 6 09:40:45 2019][1696544.929321] .nr_running : 0 [Wed Feb 6 09:40:45 2019][1696544.934112] .load : 0 [Wed Feb 6 09:40:45 2019][1696544.938906] .runnable_load_avg : 0 [Wed Feb 6 09:40:45 2019][1696544.943698] .blocked_load_avg : 0 [Wed Feb 6 09:40:45 2019][1696544.948491] .tg_load_avg : 0 [Wed Feb 6 09:40:45 2019][1696544.953284] .tg_load_contrib : 0 [Wed Feb 6 09:40:45 2019][1696544.958076] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:45 2019][1696544.962869] .tg->runnable_avg : 0 [Wed Feb 6 09:40:45 2019][1696544.967664] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:45 2019][1696544.972455] .throttled : 0 [Wed Feb 6 09:40:45 2019][1696544.977249] .throttle_count : 0 [Wed Feb 6 09:40:45 2019][1696544.982041] .avg->runnable_avg_sum : 9 [Wed Feb 6 09:40:45 2019][1696544.986834] .avg->runnable_avg_period : 47345 [Wed Feb 6 09:40:45 2019][1696544.991978] [Wed Feb 6 09:40:45 2019][1696544.993646] rt_rq[16]:/ [Wed Feb 6 09:40:45 2019][1696544.996272] .rt_nr_running : 0 [Wed Feb 6 09:40:45 2019][1696545.001066] .rt_throttled : 0 [Wed Feb 6 09:40:45 2019][1696545.005858] .rt_time : 0.000000 [Wed Feb 6 09:40:45 2019][1696545.011259] .rt_runtime : 837.630493 [Wed Feb 6 09:40:45 2019][1696545.016830] [Wed Feb 6 09:40:45 2019][1696545.018502] runnable tasks: [Wed Feb 6 09:40:45 2019][1696545.021475] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:45 2019][1696545.032421] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:45 2019][1696545.043293] watchdog/16 91 -11.984370 424050 0 0.000000 53260.986419 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.054627] migration/16 92 0.000000 3816 0 0.000000 807458.937682 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.065964] ksoftirqd/16 93 24701354.046845 10757739 120 0.000000 112517.831482 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.077298] kworker/16:0H 95 4446.620451 9 100 0.000000 0.040566 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.088658] khungtaskd 257 24701414.615605 14151 120 0.000000 5097.275273 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.099999] kworker/16:1H 7528 24701353.868014 11663979 100 0.000000 50714.256167 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.111358] kworker/16:1 59050 24701515.216077 1904569 120 0.000000 96191.816612 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.122699] ptlrpc_hr00_002 5836 24701486.140772 82444420 120 0.000000 764938.509540 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.134033] ptlrpc_hr00_003 5837 24701509.574360 82418649 120 0.000000 771495.769022 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.145369] ptlrpc_hr00_004 5838 24701457.792386 82471796 120 0.000000 766510.993607 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.156715] ptlrpcd_00_00 5896 24701486.181870 6012833 120 0.000000 88484.879659 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.168061] ldlm_cb00_001 6021 24688897.411491 6769655 120 0.000000 137531.967781 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.179399] ll_cfg_requeue 6050 91248.236796 15 120 0.000000 0.354683 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.190748] osp-syn-27-0 6328 24701338.910651 790066 120 0.000000 20951.035796 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.202086] osp_up1-2 6434 20335766.220090 13 120 0.000000 0.377878 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.213421] osp_up3-2 6436 20156735.157268 9 120 0.000000 0.249453 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.224759] osp-syn-13-2 6440 24701366.722816 440365 120 0.000000 12411.815809 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.236097] osp-syn-17-2 6470 24701428.001949 438707 120 0.000000 12668.221818 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.247442] mdt_io00_003 6773 24395657.795429 5591652 120 0.000000 222341.220678 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.258775] ldlm_cn00_006 6777 24701498.890912 12480780 120 0.000000 579283.620677 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.270116] mdt00_009 6883 24701527.261061 231597713 120 0.000000 8577421.924959 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.281452] mdt00_011 7500 24701485.753485 233446491 120 0.000000 8092315.363031 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.292789] mdt_io00_016 8976 24395657.799247 3627991 120 0.000000 170556.767620 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.304126] mdt00_024 9181 24701485.729599 213290589 120 0.000000 7203292.248725 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.315459] mdt_io00_023 10518 24395657.789819 5936722 120 0.000000 246232.996427 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.326794] ldlm_cn00_016 14889 24695687.426370 11912828 120 0.000000 560232.523197 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.338141] mdt_io00_029 22644 24701424.885567 4974998 120 0.000000 186511.736826 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.349481] mdt00_027 38457 24701485.817497 125506708 120 0.000000 4394569.263047 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.360831] mdt00_044 15281 24701449.843971 14251910 120 0.000000 851662.934761 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.372187] mdt_io00_042 27676 24395657.821959 661 120 0.000000 26.069302 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.383521] mdt_io00_046 27688 24701377.581150 28164 120 0.000000 7485.085791 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.394854] mdt_io00_048 27693 24701376.288362 25140 120 0.000000 899.661735 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.406193] mdt_io00_053 27709 24395657.832760 357 120 0.000000 1120.464556 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.417527] mdt_io00_055 27713 24678620.909514 23915 120 0.000000 2428.658019 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.428882] ll_mgs_0031 29002 24701515.070506 44659 120 0.000000 1123.286092 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.440219] kworker/16:0 29942 24684975.964679 82 120 0.000000 2.709403 0.000000 0 / [Wed Feb 6 09:40:45 2019][1696545.451577] [Wed Feb 6 09:40:45 2019][1696545.453257] cpu#17, 1996.199 MHz [Wed Feb 6 09:40:45 2019][1696545.456668] .nr_running : 0 [Wed Feb 6 09:40:45 2019][1696545.461468] .load : 0 [Wed Feb 6 09:40:45 2019][1696545.466271] .nr_switches : 2823485818 [Wed Feb 6 09:40:45 2019][1696545.471850] .nr_load_updates : 418468781 [Wed Feb 6 09:40:45 2019][1696545.477346] .nr_uninterruptible : -159133 [Wed Feb 6 09:40:45 2019][1696545.482668] .next_balance : 5991.169365 [Wed Feb 6 09:40:45 2019][1696545.488337] .curr->pid : 0 [Wed Feb 6 09:40:45 2019][1696545.493132] .clock : 1696545492.612176 [Wed Feb 6 09:40:45 2019][1696545.499314] .cpu_load[0] : 0 [Wed Feb 6 09:40:45 2019][1696545.504116] .cpu_load[1] : 0 [Wed Feb 6 09:40:45 2019][1696545.508916] .cpu_load[2] : 0 [Wed Feb 6 09:40:45 2019][1696545.513717] .cpu_load[3] : 0 [Wed Feb 6 09:40:45 2019][1696545.518511] .cpu_load[4] : 0 [Wed Feb 6 09:40:45 2019][1696545.523316] .avg_idle : 3123045 [Wed Feb 6 09:40:45 2019][1696545.528634] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:45 2019][1696545.533959] [Wed Feb 6 09:40:45 2019][1696545.535630] cfs_rq[17]:/ [Wed Feb 6 09:40:45 2019][1696545.538342] .exec_clock : 0.000000 [Wed Feb 6 09:40:45 2019][1696545.543744] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:45 2019][1696545.549141] .min_vruntime : 20310285.343182 [Wed Feb 6 09:40:45 2019][1696545.555148] .max_vruntime : 0.000001 [Wed Feb 6 09:40:45 2019][1696545.560549] .spread : 0.000000 [Wed Feb 6 09:40:45 2019][1696545.565946] .spread0 : -7815311.970957 [Wed Feb 6 09:40:45 2019][1696545.571954] .nr_spread_over : 0 [Wed Feb 6 09:40:45 2019][1696545.576746] .nr_running : 0 [Wed Feb 6 09:40:45 2019][1696545.581541] .load : 0 [Wed Feb 6 09:40:45 2019][1696545.586331] .runnable_load_avg : 0 [Wed Feb 6 09:40:45 2019][1696545.591125] .blocked_load_avg : 2 [Wed Feb 6 09:40:45 2019][1696545.595918] .tg_load_avg : 0 [Wed Feb 6 09:40:45 2019][1696545.600711] .tg_load_contrib : 0 [Wed Feb 6 09:40:45 2019][1696545.605503] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:45 2019][1696545.610295] .tg->runnable_avg : 0 [Wed Feb 6 09:40:45 2019][1696545.615087] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:45 2019][1696545.619882] .throttled : 0 [Wed Feb 6 09:40:45 2019][1696545.624675] .throttle_count : 0 [Wed Feb 6 09:40:45 2019][1696545.629468] .avg->runnable_avg_sum : 82 [Wed Feb 6 09:40:45 2019][1696545.634346] .avg->runnable_avg_period : 46586 [Wed Feb 6 09:40:45 2019][1696545.639489] [Wed Feb 6 09:40:45 2019][1696545.641158] rt_rq[17]:/ [Wed Feb 6 09:40:45 2019][1696545.643785] .rt_nr_running : 0 [Wed Feb 6 09:40:45 2019][1696545.648577] .rt_throttled : 0 [Wed Feb 6 09:40:45 2019][1696545.653369] .rt_time : 0.000000 [Wed Feb 6 09:40:45 2019][1696545.658771] .rt_runtime : 958.767362 [Wed Feb 6 09:40:45 2019][1696545.664342] [Wed Feb 6 09:40:45 2019][1696545.666016] runnable tasks: [Wed Feb 6 09:40:45 2019][1696545.668988] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:45 2019][1696545.679933] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:45 2019][1696545.690804] watchdog/17 96 -11.980583 424064 0 0.000000 78964.520633 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.702138] migration/17 97 0.000000 3176 0 0.000000 598564.034013 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.713475] ksoftirqd/17 98 20310185.973975 17020032 120 0.000000 163788.198155 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.724811] kworker/17:0H 100 7435.640773 10 100 0.000000 0.040378 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.736199] slapd 19485 20310000.450871 2468 120 0.000000 35.055866 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.747537] kworker/17:1H 29158 20310179.964712 13569766 100 0.000000 63058.304343 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.758877] ptlrpc_hr01_001 5842 20310249.491714 121153536 120 0.000000 1145358.834133 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.770207] ptlrpc_hr01_003 5845 20310250.191683 121100065 120 0.000000 1151500.328371 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.781549] kiblnd_sd_01_00 5879 20310285.781503 2884616788 120 0.000000 21169525.478128 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.792975] ptlrpcd_01_07 5915 20310272.576584 7013055 120 0.000000 107395.988273 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.804326] mdt01_000 6170 20243504.070842 108147603 120 0.000000 4318668.618579 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.815663] mdt_io01_000 6212 20226660.811287 6916789 120 0.000000 257598.085442 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.827001] mdt_io01_001 6213 20230570.405682 5609224 120 0.000000 207822.526837 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.838340] osp-pre-10-0 6257 20310270.312626 247828 120 0.000000 3794.541373 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.849675] osp-syn-2-0 6270 20309466.276928 790687 120 0.000000 20753.485653 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.861013] osp-pre-21-0 6291 20310272.687660 247506 120 0.000000 3782.544828 0.000000 1 / [Wed Feb 6 09:40:45 2019][1696545.872349] osp-syn-20-0 6302 20310208.489776 784587 120 0.000000 20876.899584 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696545.883686] osp-syn-15-0 6318 20310230.339740 779312 120 0.000000 20994.396381 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696545.895020] osp-syn-1-0 6322 20310257.933295 771848 120 0.000000 21053.406458 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696545.906356] osp-pre-1-0 6323 20310267.093507 247643 120 0.000000 6408.863277 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696545.917699] osp-pre-42-2 6457 20310272.491150 234947 120 0.000000 4120.073652 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696545.929039] osp-pre-19-2 6465 20310272.338589 234900 120 0.000000 7705.625912 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696545.940376] osp-syn-41-2 6484 20310226.774531 442612 120 0.000000 12387.900574 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696545.951713] osp-syn-29-2 6510 20310193.902137 443812 120 0.000000 12527.884433 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696545.963048] osp-syn-36-2 6522 20310196.348008 437872 120 0.000000 13402.748770 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696545.974383] mdt_out01_003 6611 20310268.709804 27962006 120 0.000000 550360.930760 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696545.985720] mdt01_006 6760 20232122.644676 113690639 120 0.000000 4748610.490601 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696545.997059] mdt01_007 6813 20232122.644395 101092500 120 0.000000 4069870.365850 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.008400] mdt_io01_012 7223 20230557.770163 5527984 120 0.000000 208270.477594 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.019736] mdt01_011 7415 20232122.644847 111341512 120 0.000000 4321683.471742 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.031075] mdt01_012 8946 20232122.644134 112619163 120 0.000000 4356409.510856 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.042408] mdt01_014 8948 20234359.867864 101831067 120 0.000000 3957819.635933 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.053745] mdt01_016 8950 20232122.644345 116851256 120 0.000000 4560468.758626 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.065090] mdt_io01_021 9001 20230570.488149 6029523 120 0.000000 226971.637305 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.076426] mdt01_024 9004 20232122.644475 102842019 120 0.000000 3969691.572029 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.087762] mdt01_025 9005 20310267.973880 111471950 120 0.000000 4596099.199160 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.099098] mdt01_026 9008 20232122.644305 87740471 120 0.000000 4169093.004517 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.110439] mdt_rdpg01_007 9149 20310273.170845 31594299 120 0.000000 1177392.216301 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.121774] mdt_seqm_0002 9874 20310242.803670 3436 120 0.000000 73.132283 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.133120] mdt_io01_040 20861 20226660.706057 6723998 120 0.000000 248879.229513 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.144462] ldlm_cn01_026 28327 20310267.966315 4775460 120 0.000000 302001.133153 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.155801] mdt01_040 48792 20310211.849415 31151815 120 0.000000 1158322.198837 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.167136] kworker/17:2 100481 20310274.214682 82231 120 0.000000 11203.942825 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.178558] mdt01_041 15163 20240382.919086 4307381 120 0.000000 2176607.743303 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.189896] mdt01_050 15230 20232122.644566 8254122 120 0.000000 516986.134243 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.201239] mdt_io01_063 15431 20310186.112408 371225 120 0.000000 13832.244714 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.212573] mdt_io01_068 15436 20225308.956739 73115 120 0.000000 2827.593891 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.223916] mdt_fld_0003 16814 20310015.469539 4055 120 0.000000 60.245233 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.235276] kworker/u258:0 28453 20232122.651338 21 120 0.000000 1029.684136 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.246611] kworker/17:0 28831 20168545.049827 13 120 0.000000 0.103357 0.000000 1 / [Wed Feb 6 09:40:46 2019][1696546.257979] [Wed Feb 6 09:40:46 2019][1696546.259660] cpu#18, 1996.199 MHz [Wed Feb 6 09:40:46 2019][1696546.263076] .nr_running : 0 [Wed Feb 6 09:40:46 2019][1696546.267878] .load : 0 [Wed Feb 6 09:40:46 2019][1696546.272680] .nr_switches : 2615351072 [Wed Feb 6 09:40:46 2019][1696546.278261] .nr_load_updates : 356556565 [Wed Feb 6 09:40:46 2019][1696546.283756] .nr_uninterruptible : -134471 [Wed Feb 6 09:40:46 2019][1696546.289076] .next_balance : 5991.170172 [Wed Feb 6 09:40:46 2019][1696546.294743] .curr->pid : 0 [Wed Feb 6 09:40:46 2019][1696546.299543] .clock : 1696546298.631045 [Wed Feb 6 09:40:46 2019][1696546.305723] .cpu_load[0] : 0 [Wed Feb 6 09:40:46 2019][1696546.310529] .cpu_load[1] : 0 [Wed Feb 6 09:40:46 2019][1696546.315328] .cpu_load[2] : 0 [Wed Feb 6 09:40:46 2019][1696546.320127] .cpu_load[3] : 0 [Wed Feb 6 09:40:46 2019][1696546.324925] .cpu_load[4] : 0 [Wed Feb 6 09:40:46 2019][1696546.329726] .avg_idle : 4412141 [Wed Feb 6 09:40:46 2019][1696546.335044] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:46 2019][1696546.340363] [Wed Feb 6 09:40:46 2019][1696546.342032] cfs_rq[18]:/ [Wed Feb 6 09:40:46 2019][1696546.344745] .exec_clock : 0.000000 [Wed Feb 6 09:40:46 2019][1696546.350144] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:46 2019][1696546.355543] .min_vruntime : 13315551.447299 [Wed Feb 6 09:40:46 2019][1696546.361549] .max_vruntime : 0.000001 [Wed Feb 6 09:40:46 2019][1696546.366951] .spread : 0.000000 [Wed Feb 6 09:40:46 2019][1696546.372350] .spread0 : -14810048.557032 [Wed Feb 6 09:40:46 2019][1696546.378441] .nr_spread_over : 0 [Wed Feb 6 09:40:46 2019][1696546.383235] .nr_running : 0 [Wed Feb 6 09:40:46 2019][1696546.388027] .load : 0 [Wed Feb 6 09:40:46 2019][1696546.392820] .runnable_load_avg : 0 [Wed Feb 6 09:40:46 2019][1696546.397614] .blocked_load_avg : 1 [Wed Feb 6 09:40:46 2019][1696546.402405] .tg_load_avg : 0 [Wed Feb 6 09:40:46 2019][1696546.407198] .tg_load_contrib : 0 [Wed Feb 6 09:40:46 2019][1696546.411990] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:46 2019][1696546.416785] .tg->runnable_avg : 0 [Wed Feb 6 09:40:46 2019][1696546.421577] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:46 2019][1696546.426370] .throttled : 0 [Wed Feb 6 09:40:46 2019][1696546.431163] .throttle_count : 0 [Wed Feb 6 09:40:46 2019][1696546.435954] .avg->runnable_avg_sum : 135 [Wed Feb 6 09:40:46 2019][1696546.440921] .avg->runnable_avg_period : 46811 [Wed Feb 6 09:40:46 2019][1696546.446065] [Wed Feb 6 09:40:46 2019][1696546.447735] rt_rq[18]:/ [Wed Feb 6 09:40:46 2019][1696546.450358] .rt_nr_running : 0 [Wed Feb 6 09:40:46 2019][1696546.455152] .rt_throttled : 0 [Wed Feb 6 09:40:46 2019][1696546.459945] .rt_time : 0.000000 [Wed Feb 6 09:40:46 2019][1696546.465344] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:46 2019][1696546.471004] [Wed Feb 6 09:40:46 2019][1696546.472675] runnable tasks: [Wed Feb 6 09:40:46 2019][1696546.475651] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:46 2019][1696546.486597] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:46 2019][1696546.497467] watchdog/18 101 -11.978660 424082 0 0.000000 122055.787990 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.508800] migration/18 102 0.000000 2227 0 0.000000 498389.049624 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.520137] ksoftirqd/18 103 13315539.383788 8084962 120 0.000000 82934.643453 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.531473] kworker/18:0H 105 3463.220153 10 100 0.000000 0.078811 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.542833] kauditd 404 13315497.207862 224062 120 0.000000 1972.615866 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.554176] kworker/u261:0 7439 1396.572682 4 100 0.000000 0.019097 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.565527] kdmwork-253:4 18505 13315539.645650 147013992 120 0.000000 1825312.097054 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.576863] collectd 19081 13315523.159619 182066 120 0.000000 9144.236809 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.588198] writer#0 19092 13315526.288032 18746701 120 0.000000 193089.910366 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.599534] reader#2 19100 13315526.484635 187932 120 0.000000 15317.720976 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.610869] kworker/18:1H 19228 13315523.971444 5529086 100 0.000000 23297.626852 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.622212] kworker/18:1 129420 13315539.058205 514225 120 0.000000 9030.765338 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.633635] kiblnd_connd 5873 13315539.352979 1036008 120 0.000000 20561.628206 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.644978] kiblnd_sd_02_00 5883 13315551.758605 1595560382 120 0.000000 10420292.868966 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.656397] kiblnd_sd_02_02 5885 10936016.318158 1595560308 120 0.000000 10414423.288569 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.667827] ptlrpcd_02_09 5929 13315537.325390 13206951 120 0.000000 184846.907680 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.679164] ptlrpcd_02_10 5930 13315537.866498 13228196 120 0.000000 190720.699882 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.690516] mdt_out02_001 6200 8018425.367006 7679423 120 0.000000 179447.079283 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.701857] mdt_io02_000 6215 13162540.849580 5482273 120 0.000000 221110.056882 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.713192] osp-syn-46-0 6254 13315296.351340 779864 120 0.000000 21328.988187 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.724529] osp-pre-42-0 6267 13315537.273823 247822 120 0.000000 4513.871262 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.735870] osp-syn-28-0 6338 13305765.364135 776552 120 0.000000 20995.905281 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.747216] osp-pre-33-2 6491 13315534.816952 235041 120 0.000000 4752.030601 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.758557] osp-syn-35-2 6526 13315318.964017 444773 120 0.000000 12434.907149 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.769906] mdt_io02_011 7110 13162624.180811 4377324 120 0.000000 176551.208535 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.781249] mdt_out02_003 8966 7593335.593013 7811908 120 0.000000 180132.254647 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.792581] mdt02_012 8978 13170294.295243 91041407 120 0.000000 3820480.618692 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.803921] mdt_io02_019 9036 13315311.094665 6582394 120 0.000000 265277.859239 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.815254] ldlm_cn02_009 9373 13315199.980619 16350597 120 0.000000 553294.922537 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.826595] ldlm_cn02_011 17691 13278554.311743 15876951 120 0.000000 536931.115897 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.837938] ldlm_cn02_019 23324 13315205.392067 9325734 120 0.000000 369514.265736 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.849288] mdt02_044 15370 13315399.704367 3354292 120 0.000000 834009.656748 0.000000 2 / [Wed Feb 6 09:40:46 2019][1696546.860630] mdt_io02_029 15445 13162624.177034 397640 120 0.000000 15647.947450 0.000000 2 / [Wed Feb 6 09:40:47 2019][1696546.871972] ldlm_bl_09 15966 13315405.784339 2325689 120 0.000000 322651.642632 0.000000 2 / [Wed Feb 6 09:40:47 2019][1696546.883307] ldlm_bl_13 15970 13315410.784209 2325341 120 0.000000 324174.692966 0.000000 2 / [Wed Feb 6 09:40:47 2019][1696546.894661] mdt_io02_050 27715 12922746.100810 154 120 0.000000 10.783092 0.000000 2 / [Wed Feb 6 09:40:47 2019][1696546.906002] mdt_io02_074 27765 13162682.612690 1413 120 0.000000 49.199869 0.000000 2 / [Wed Feb 6 09:40:47 2019][1696546.917342] mdt_io02_084 27775 12922772.018649 24 120 0.000000 1.030571 0.000000 2 / [Wed Feb 6 09:40:47 2019][1696546.928686] ll_mgs_0029 28997 13315536.457471 163 120 0.000000 820.523600 0.000000 2 / [Wed Feb 6 09:40:47 2019][1696546.940027] kworker/18:0 30339 13305598.825006 12 120 0.000000 0.068912 0.000000 2 / [Wed Feb 6 09:40:47 2019][1696546.951367] sshd 30460 13315376.213889 15 120 0.000000 6.747772 0.000000 2 / [Wed Feb 6 09:40:47 2019][1696546.962720] [Wed Feb 6 09:40:47 2019][1696546.964401] cpu#19, 1996.199 MHz [Wed Feb 6 09:40:47 2019][1696546.967816] .nr_running : 0 [Wed Feb 6 09:40:47 2019][1696546.972618] .load : 0 [Wed Feb 6 09:40:47 2019][1696546.977419] .nr_switches : 633029803 [Wed Feb 6 09:40:47 2019][1696546.982913] .nr_load_updates : 221466775 [Wed Feb 6 09:40:47 2019][1696546.988409] .nr_uninterruptible : -23902 [Wed Feb 6 09:40:47 2019][1696546.993645] .next_balance : 5991.170878 [Wed Feb 6 09:40:47 2019][1696546.999316] .curr->pid : 0 [Wed Feb 6 09:40:47 2019][1696547.004113] .clock : 1696547003.646928 [Wed Feb 6 09:40:47 2019][1696547.010303] .cpu_load[0] : 0 [Wed Feb 6 09:40:47 2019][1696547.015102] .cpu_load[1] : 0 [Wed Feb 6 09:40:47 2019][1696547.019904] .cpu_load[2] : 0 [Wed Feb 6 09:40:47 2019][1696547.024701] .cpu_load[3] : 0 [Wed Feb 6 09:40:47 2019][1696547.029498] .cpu_load[4] : 0 [Wed Feb 6 09:40:47 2019][1696547.034299] .avg_idle : 3692185 [Wed Feb 6 09:40:47 2019][1696547.039621] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:47 2019][1696547.044938] [Wed Feb 6 09:40:47 2019][1696547.046607] cfs_rq[19]:/ [Wed Feb 6 09:40:47 2019][1696547.049319] .exec_clock : 0.000000 [Wed Feb 6 09:40:47 2019][1696547.054719] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:47 2019][1696547.060119] .min_vruntime : 33566339.567686 [Wed Feb 6 09:40:47 2019][1696547.066123] .max_vruntime : 0.000001 [Wed Feb 6 09:40:47 2019][1696547.071524] .spread : 0.000000 [Wed Feb 6 09:40:47 2019][1696547.076924] .spread0 : 5440737.110255 [Wed Feb 6 09:40:47 2019][1696547.082844] .nr_spread_over : 0 [Wed Feb 6 09:40:47 2019][1696547.087636] .nr_running : 0 [Wed Feb 6 09:40:47 2019][1696547.092428] .load : 0 [Wed Feb 6 09:40:47 2019][1696547.097221] .runnable_load_avg : 0 [Wed Feb 6 09:40:47 2019][1696547.102015] .blocked_load_avg : 0 [Wed Feb 6 09:40:47 2019][1696547.106807] .tg_load_avg : 0 [Wed Feb 6 09:40:47 2019][1696547.111600] .tg_load_contrib : 0 [Wed Feb 6 09:40:47 2019][1696547.116392] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:47 2019][1696547.121185] .tg->runnable_avg : 0 [Wed Feb 6 09:40:47 2019][1696547.125978] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:47 2019][1696547.130772] .throttled : 0 [Wed Feb 6 09:40:47 2019][1696547.135564] .throttle_count : 0 [Wed Feb 6 09:40:47 2019][1696547.140355] .avg->runnable_avg_sum : 130 [Wed Feb 6 09:40:47 2019][1696547.145322] .avg->runnable_avg_period : 46938 [Wed Feb 6 09:40:47 2019][1696547.150465] [Wed Feb 6 09:40:47 2019][1696547.152134] rt_rq[19]:/ [Wed Feb 6 09:40:47 2019][1696547.154761] .rt_nr_running : 0 [Wed Feb 6 09:40:47 2019][1696547.159552] .rt_throttled : 0 [Wed Feb 6 09:40:47 2019][1696547.164348] .rt_time : 0.000000 [Wed Feb 6 09:40:47 2019][1696547.169747] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:47 2019][1696547.175404] [Wed Feb 6 09:40:47 2019][1696547.177077] runnable tasks: [Wed Feb 6 09:40:47 2019][1696547.180052] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:47 2019][1696547.190997] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:47 2019][1696547.201870] watchdog/19 106 -11.980914 424105 0 0.000000 146999.765012 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.213201] migration/19 107 0.000000 3267 0 0.000000 477221.850364 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.224537] ksoftirqd/19 108 33566315.279577 4228503 120 0.000000 40409.962751 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.235875] kworker/19:0H 110 4608.678505 10 100 0.000000 0.038373 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.247232] khugepaged 281 33566280.972214 220985 139 0.000000 3309749.644500 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.258566] kworker/19:1 7267 33566327.572235 403366 120 0.000000 12175.664688 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.269910] kmpathd 11776 2514.922430 2 100 0.000000 0.018226 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.281249] auditd 17473 33566281.553633 242000 116 0.000000 8254.438499 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.292584] JS Sour~ Thread 17533 4564.392014 5 120 0.000000 0.054653 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.303917] sssd 17535 33566295.583336 56582 120 0.000000 1305.960625 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.315258] dsm_sa_datamgrd 18867 4609.720729 2 120 0.000000 0.128655 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.326590] dsm_sa_datamgrd 18871 33566315.621652 338942 120 0.000000 10087.210556 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.337926] kworker/19:1H 18431 33566315.268185 5127334 100 0.000000 17491.667878 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.349267] in:imjournal 19089 33566339.567686 2008136 120 0.000000 45815.569708 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.360600] rs:main Q:Reg 19094 33566293.562350 158974 120 0.000000 5811.075515 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.371948] ptlrpc_hr03_005 5860 33566309.815578 20347822 120 0.000000 181014.851845 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.383282] kiblnd_sd_03_00 5887 33566333.465134 440110659 120 0.000000 3221615.448918 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.394625] ptlrpcd_03_00 5932 33566322.995329 12647092 120 0.000000 205970.184194 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.405962] ptlrpcd_03_03 5935 33566323.050986 13108971 120 0.000000 191544.694390 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.417297] ptlrpcd_03_05 5937 33566325.430154 13118800 120 0.000000 196923.779951 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.428636] ptlrpcd_03_08 5940 33566320.950321 12633519 120 0.000000 201140.485801 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.439972] ptlrpcd_03_11 5943 33566320.982972 13090494 120 0.000000 196010.688900 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.451313] kmmpd-dm-3 6148 33566315.286009 411721 120 0.000000 7347.898375 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.462660] mdt_io03_000 6218 33566309.616982 2099522 120 0.000000 72618.481082 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.474007] osp-pre-1-2 6435 33566315.182246 205442 120 0.000000 3411.843388 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.485343] osp-syn-10-2 6446 33565910.313871 443710 120 0.000000 12301.447464 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.496724] mdt_io03_009 19893 32508435.193142 1960017 120 0.000000 70296.869137 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.508069] ldlm_cn03_012 23314 33443882.963725 5244019 120 0.000000 183944.675559 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.519434] mdt_io03_018 17104 32508435.213932 256645 120 0.000000 10412.428866 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.530773] mdt03_020 17205 33566309.456711 8641028 120 0.000000 657362.141978 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.542116] ldlm_cn03_019 25318 33563070.445288 64570 120 0.000000 91192.563990 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.553457] mdt_io03_028 27462 32508435.190056 83 120 0.000000 2.866571 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.564809] sssd_be 28616 33566323.008264 2823 120 0.000000 6938.279673 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.576153] kworker/19:2 29155 32772071.454523 13 120 0.000000 0.084633 0.000000 3 / [Wed Feb 6 09:40:47 2019][1696547.587513] [Wed Feb 6 09:40:47 2019][1696547.589196] cpu#20, 1996.199 MHz [Wed Feb 6 09:40:47 2019][1696547.592612] .nr_running : 0 [Wed Feb 6 09:40:47 2019][1696547.597414] .load : 0 [Wed Feb 6 09:40:47 2019][1696547.602210] .nr_switches : 4148265918 [Wed Feb 6 09:40:47 2019][1696547.607787] .nr_load_updates : 530765672 [Wed Feb 6 09:40:47 2019][1696547.613281] .nr_uninterruptible : 568621 [Wed Feb 6 09:40:47 2019][1696547.618517] .next_balance : 5991.171500 [Wed Feb 6 09:40:47 2019][1696547.624187] .curr->pid : 0 [Wed Feb 6 09:40:47 2019][1696547.628983] .clock : 1696547628.661838 [Wed Feb 6 09:40:47 2019][1696547.635167] .cpu_load[0] : 0 [Wed Feb 6 09:40:47 2019][1696547.639969] .cpu_load[1] : 0 [Wed Feb 6 09:40:47 2019][1696547.644771] .cpu_load[2] : 0 [Wed Feb 6 09:40:47 2019][1696547.649568] .cpu_load[3] : 0 [Wed Feb 6 09:40:47 2019][1696547.654372] .cpu_load[4] : 0 [Wed Feb 6 09:40:47 2019][1696547.659174] .avg_idle : 10000000 [Wed Feb 6 09:40:47 2019][1696547.664580] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:47 2019][1696547.669902] [Wed Feb 6 09:40:47 2019][1696547.671575] cfs_rq[20]:/ [Wed Feb 6 09:40:47 2019][1696547.674290] .exec_clock : 0.000000 [Wed Feb 6 09:40:47 2019][1696547.679688] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:47 2019][1696547.685087] .min_vruntime : 27373080.743075 [Wed Feb 6 09:40:47 2019][1696547.691094] .max_vruntime : 0.000001 [Wed Feb 6 09:40:47 2019][1696547.696492] .spread : 0.000000 [Wed Feb 6 09:40:47 2019][1696547.701891] .spread0 : -752523.954945 [Wed Feb 6 09:40:47 2019][1696547.707812] .nr_spread_over : 0 [Wed Feb 6 09:40:47 2019][1696547.712605] .nr_running : 0 [Wed Feb 6 09:40:47 2019][1696547.717399] .load : 0 [Wed Feb 6 09:40:47 2019][1696547.722191] .runnable_load_avg : 0 [Wed Feb 6 09:40:47 2019][1696547.726982] .blocked_load_avg : 0 [Wed Feb 6 09:40:47 2019][1696547.731777] .tg_load_avg : 0 [Wed Feb 6 09:40:47 2019][1696547.736569] .tg_load_contrib : 0 [Wed Feb 6 09:40:47 2019][1696547.741360] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:47 2019][1696547.746155] .tg->runnable_avg : 0 [Wed Feb 6 09:40:47 2019][1696547.750948] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:47 2019][1696547.755739] .throttled : 0 [Wed Feb 6 09:40:47 2019][1696547.760534] .throttle_count : 0 [Wed Feb 6 09:40:47 2019][1696547.765325] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:40:47 2019][1696547.770119] .avg->runnable_avg_period : 48042 [Wed Feb 6 09:40:47 2019][1696547.775261] [Wed Feb 6 09:40:47 2019][1696547.776932] rt_rq[20]:/ [Wed Feb 6 09:40:47 2019][1696547.779557] .rt_nr_running : 0 [Wed Feb 6 09:40:47 2019][1696547.784350] .rt_throttled : 0 [Wed Feb 6 09:40:47 2019][1696547.789141] .rt_time : 0.007064 [Wed Feb 6 09:40:47 2019][1696547.794542] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:47 2019][1696547.800200] [Wed Feb 6 09:40:47 2019][1696547.801874] runnable tasks: [Wed Feb 6 09:40:47 2019][1696547.804845] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:47 2019][1696547.815794] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:47 2019][1696547.826665] watchdog/20 111 -11.981885 424079 0 0.000000 182121.251601 0.000000 0 / [Wed Feb 6 09:40:47 2019][1696547.837996] migration/20 112 0.000000 2945 0 0.000000 523118.147901 0.000000 0 / [Wed Feb 6 09:40:47 2019][1696547.849332] ksoftirqd/20 113 27373015.297882 12551186 120 0.000000 128435.400594 0.000000 0 / [Wed Feb 6 09:40:47 2019][1696547.860669] kworker/20:0H 115 4723.076991 10 100 0.000000 0.037809 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696547.872038] ccp-4-q3 10667 1980.927229 2 120 0.000000 0.013947 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696547.883377] ccp-5-q4 10755 1988.543334 2 120 0.000000 0.010410 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696547.894714] ccp-6-q1 10816 2010.145026 2 120 0.000000 0.008808 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696547.906056] gmain 17524 3976.865412 2 120 0.000000 0.075584 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696547.917399] dsm_sa_datamgrd 18951 4864.290780 34 120 0.000000 2.751250 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696547.928736] kworker/20:1H 27801 27373015.231987 12847176 100 0.000000 55000.933044 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696547.940071] ptlrpc_hr00_000 5834 27373058.868710 82505668 120 0.000000 767257.884704 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696547.951403] ptlrpc_hr00_001 5835 27373058.814614 82445754 120 0.000000 767444.486364 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696547.962741] ptlrpc_hr00_005 5839 27373058.861826 82469743 120 0.000000 765415.028023 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696547.974115] osp-syn-22-0 6324 27373058.521397 781270 120 0.000000 20746.033787 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696547.985467] ldlm_cn00_005 6769 27370681.727950 15013465 120 0.000000 637124.668825 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696547.996820] mdt00_017 8994 27373080.743075 229468684 120 0.000000 7777083.532249 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696548.008162] ldlm_cn00_010 11333 27370681.724953 9375749 120 0.000000 491020.687763 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696548.019500] mdt_rdpg00_005 12704 27308856.664488 84208150 120 0.000000 1710765.870228 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696548.030837] ldlm_cn00_013 14884 27372251.398582 11272355 120 0.000000 553139.234201 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696548.042184] mdt_io00_028 22161 27367707.123742 5613577 120 0.000000 210152.767871 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696548.053523] kworker/20:2 33577 27373068.765448 558960 120 0.000000 16669.198703 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696548.064872] mdt00_034 15220 27373058.460291 12836999 120 0.000000 657847.610851 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696548.076210] mdt00_041 15278 27373068.800495 3281534 120 0.000000 411437.183546 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696548.087551] mdt_io00_032 15463 27110993.216903 1250 120 0.000000 69.111317 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696548.098891] mdt_rdpg00_009 17029 27341458.477587 1720776 120 0.000000 55189.704101 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696548.110233] kworker/20:0 25001 26040605.782790 6 120 0.000000 0.041108 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696548.121574] mdt_io00_043 27679 27319873.399238 222 120 0.000000 9.591063 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696548.132908] mdt_io00_051 27706 27319915.388939 288 120 0.000000 53.879592 0.000000 0 / [Wed Feb 6 09:40:48 2019][1696548.144288] [Wed Feb 6 09:40:48 2019][1696548.145965] cpu#21, 1996.199 MHz [Wed Feb 6 09:40:48 2019][1696548.149381] .nr_running : 1 [Wed Feb 6 09:40:48 2019][1696548.154184] .load : 1024 [Wed Feb 6 09:40:48 2019][1696548.159240] .nr_switches : 3044612033 [Wed Feb 6 09:40:48 2019][1696548.164813] .nr_load_updates : 423114884 [Wed Feb 6 09:40:48 2019][1696548.170304] .nr_uninterruptible : -99845 [Wed Feb 6 09:40:48 2019][1696548.175534] .next_balance : 5991.172103 [Wed Feb 6 09:40:48 2019][1696548.181195] .curr->pid : 7089 [Wed Feb 6 09:40:48 2019][1696548.186255] .clock : 1696548185.676445 [Wed Feb 6 09:40:48 2019][1696548.192436] .cpu_load[0] : 1024 [Wed Feb 6 09:40:48 2019][1696548.197500] .cpu_load[1] : 1024 [Wed Feb 6 09:40:48 2019][1696548.202558] .cpu_load[2] : 1024 [Wed Feb 6 09:40:48 2019][1696548.207624] .cpu_load[3] : 1024 [Wed Feb 6 09:40:48 2019][1696548.212688] .cpu_load[4] : 1024 [Wed Feb 6 09:40:48 2019][1696548.217745] .avg_idle : 10000000 [Wed Feb 6 09:40:48 2019][1696548.223152] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:48 2019][1696548.228475] [Wed Feb 6 09:40:48 2019][1696548.230148] cfs_rq[21]:/ [Wed Feb 6 09:40:48 2019][1696548.232861] .exec_clock : 0.000000 [Wed Feb 6 09:40:48 2019][1696548.238261] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:48 2019][1696548.243660] .min_vruntime : 16505433.169354 [Wed Feb 6 09:40:48 2019][1696548.249667] .max_vruntime : 0.000001 [Wed Feb 6 09:40:48 2019][1696548.255066] .spread : 0.000000 [Wed Feb 6 09:40:48 2019][1696548.260466] .spread0 : -11620173.616494 [Wed Feb 6 09:40:48 2019][1696548.266557] .nr_spread_over : 0 [Wed Feb 6 09:40:48 2019][1696548.271351] .nr_running : 1 [Wed Feb 6 09:40:48 2019][1696548.276143] .load : 1024 [Wed Feb 6 09:40:48 2019][1696548.281196] .runnable_load_avg : 1023 [Wed Feb 6 09:40:48 2019][1696548.286249] .blocked_load_avg : 0 [Wed Feb 6 09:40:48 2019][1696548.291041] .tg_load_avg : 0 [Wed Feb 6 09:40:48 2019][1696548.295833] .tg_load_contrib : 0 [Wed Feb 6 09:40:48 2019][1696548.300628] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:48 2019][1696548.305420] .tg->runnable_avg : 0 [Wed Feb 6 09:40:48 2019][1696548.310211] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:48 2019][1696548.315004] .throttled : 0 [Wed Feb 6 09:40:48 2019][1696548.319797] .throttle_count : 0 [Wed Feb 6 09:40:48 2019][1696548.324591] .avg->runnable_avg_sum : 46094 [Wed Feb 6 09:40:48 2019][1696548.329731] .avg->runnable_avg_period : 46896 [Wed Feb 6 09:40:48 2019][1696548.334873] [Wed Feb 6 09:40:48 2019][1696548.336543] rt_rq[21]:/ [Wed Feb 6 09:40:48 2019][1696548.339168] .rt_nr_running : 0 [Wed Feb 6 09:40:48 2019][1696548.343960] .rt_throttled : 0 [Wed Feb 6 09:40:48 2019][1696548.348753] .rt_time : 53.401630 [Wed Feb 6 09:40:48 2019][1696548.354240] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:48 2019][1696548.359899] [Wed Feb 6 09:40:48 2019][1696548.361571] runnable tasks: [Wed Feb 6 09:40:48 2019][1696548.364545] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:48 2019][1696548.375491] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:48 2019][1696548.386367] watchdog/21 116 -11.979271 424104 0 0.000000 162381.471561 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.397705] migration/21 117 0.000000 2868 0 0.000000 456958.285088 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.409039] ksoftirqd/21 118 16437657.733198 14661370 120 0.000000 139791.883729 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.420378] kworker/21:0H 120 3944.085316 9 100 0.000000 0.037451 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.431735] scsi_tmf_0 2291 352.893669 2 100 0.000000 0.017603 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.443069] mlx5_health0000 7334 1523.976778 2 100 0.000000 0.013024 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.454408] systemd-udevd 7905 6856949.714439 3321 120 0.000000 1572.207742 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.465741] ccp-1-q4 10487 1873.687248 2 120 0.000000 0.008417 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.477077] ccp-3-q2 10585 1898.798460 2 120 0.000000 0.007645 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.488418] kworker/21:1H 16788 16437657.609774 11561947 100 0.000000 54744.167334 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.499790] ll_mgs_0003 6070 15950386.966339 873349 120 0.000000 20475.274331 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.511150] osp-syn-39-2 6442 16437638.186853 440160 120 0.000000 13275.741876 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.522493] mdt_out01_002 6610 16424439.608962 26524563 120 0.000000 526613.746099 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.533836] mdt01_009 6985 15941584.431352 114828906 120 0.000000 4393045.695209 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.545174] R ldlm_bl_03 7089 16505742.176462 33520666 120 0.000000 7901191.382285 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.556508] mdt_io01_010 7095 15330053.985152 9644025 120 0.000000 360429.115382 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.567849] mdt_io01_016 8954 15330059.355302 2671538 120 0.000000 114404.465392 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.579190] ldlm_cn01_008 9129 16353720.916857 31075107 120 0.000000 840020.584995 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.590526] mdt_rdpg01_009 9151 15925264.017741 20754195 120 0.000000 873027.285632 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.601865] mdt_io01_027 9838 15933051.604045 7608978 120 0.000000 276493.184287 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.613201] mdt_io01_033 15083 15933039.742629 7393243 120 0.000000 285698.194750 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.624534] mdt_io01_034 15084 15330398.517555 4848062 120 0.000000 178747.545387 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.635873] kworker/21:0 18390 16437852.290558 507633 120 0.000000 11865.501551 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.647213] mdt_io01_047 20868 15330058.280559 4708283 120 0.000000 187458.091299 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.658552] mdt_io01_050 20871 15330057.766072 374 120 0.000000 76.983289 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.669897] ldlm_cn01_029 28330 16353720.916256 13575972 120 0.000000 511748.610139 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.681236] mdt01_039 48704 15941584.429919 35620280 120 0.000000 1327514.135553 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.692577] mdt01_046 15222 16206400.666080 6516359 120 0.000000 1226017.275461 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.703913] mdt01_047 15223 16182017.259569 2037548 120 0.000000 57773.633602 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.715252] mdt01_053 15241 15939564.628428 886402 120 0.000000 36683.812075 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.726589] mdt01_056 15244 15940620.612789 689550 120 0.000000 28005.949725 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.737924] mdt01_058 15256 15940620.614753 704061 120 0.000000 28682.503228 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.749259] mdt01_060 15259 15941687.021235 2323063 120 0.000000 225707.669039 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.760598] mdt01_071 15385 16356821.449335 1832089 120 0.000000 348584.104124 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.771934] mdt01_074 15390 15938511.833744 2159079 120 0.000000 79652.898438 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.783270] mdt_io01_058 15425 15330054.038965 333 120 0.000000 15.132569 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.794606] mdt_io01_061 15428 15330057.768468 669 120 0.000000 22.745124 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.805957] kworker/21:1 25297 14970855.522519 5 120 0.000000 0.022814 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.817307] mdt_io01_101 27742 15330058.394497 441 120 0.000000 18.212360 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.828639] mdt_io01_104 27750 15330406.285573 908 120 0.000000 30.851568 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.839988] ll_mgs_0011 28979 15950385.589762 176 120 0.000000 47.978282 0.000000 1 / [Wed Feb 6 09:40:48 2019][1696548.851349] [Wed Feb 6 09:40:48 2019][1696548.853027] cpu#22, 1996.199 MHz [Wed Feb 6 09:40:48 2019][1696548.856442] .nr_running : 0 [Wed Feb 6 09:40:48 2019][1696548.861246] .load : 0 [Wed Feb 6 09:40:48 2019][1696548.866046] .nr_switches : 1325698449 [Wed Feb 6 09:40:48 2019][1696548.871625] .nr_load_updates : 356440896 [Wed Feb 6 09:40:49 2019][1696548.877119] .nr_uninterruptible : -127482 [Wed Feb 6 09:40:49 2019][1696548.882441] .next_balance : 5991.172764 [Wed Feb 6 09:40:49 2019][1696548.888110] .curr->pid : 0 [Wed Feb 6 09:40:49 2019][1696548.892911] .clock : 1696548892.694442 [Wed Feb 6 09:40:49 2019][1696548.899101] .cpu_load[0] : 0 [Wed Feb 6 09:40:49 2019][1696548.903899] .cpu_load[1] : 0 [Wed Feb 6 09:40:49 2019][1696548.908692] .cpu_load[2] : 0 [Wed Feb 6 09:40:49 2019][1696548.913500] .cpu_load[3] : 0 [Wed Feb 6 09:40:49 2019][1696548.918299] .cpu_load[4] : 0 [Wed Feb 6 09:40:49 2019][1696548.923096] .avg_idle : 6943577 [Wed Feb 6 09:40:49 2019][1696548.928410] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:49 2019][1696548.933731] [Wed Feb 6 09:40:49 2019][1696548.935399] cfs_rq[22]:/ [Wed Feb 6 09:40:49 2019][1696548.938111] .exec_clock : 0.000000 [Wed Feb 6 09:40:49 2019][1696548.943511] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:49 2019][1696548.948909] .min_vruntime : 16071378.994517 [Wed Feb 6 09:40:49 2019][1696548.954916] .max_vruntime : 0.000001 [Wed Feb 6 09:40:49 2019][1696548.960316] .spread : 0.000000 [Wed Feb 6 09:40:49 2019][1696548.965715] .spread0 : -12054231.400238 [Wed Feb 6 09:40:49 2019][1696548.971808] .nr_spread_over : 0 [Wed Feb 6 09:40:49 2019][1696548.976602] .nr_running : 0 [Wed Feb 6 09:40:49 2019][1696548.981393] .load : 0 [Wed Feb 6 09:40:49 2019][1696548.986187] .runnable_load_avg : 0 [Wed Feb 6 09:40:49 2019][1696548.990980] .blocked_load_avg : 0 [Wed Feb 6 09:40:49 2019][1696548.995772] .tg_load_avg : 0 [Wed Feb 6 09:40:49 2019][1696549.000565] .tg_load_contrib : 0 [Wed Feb 6 09:40:49 2019][1696549.005358] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:49 2019][1696549.010151] .tg->runnable_avg : 0 [Wed Feb 6 09:40:49 2019][1696549.014943] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:49 2019][1696549.019735] .throttled : 0 [Wed Feb 6 09:40:49 2019][1696549.024529] .throttle_count : 0 [Wed Feb 6 09:40:49 2019][1696549.029321] .avg->runnable_avg_sum : 7 [Wed Feb 6 09:40:49 2019][1696549.034113] .avg->runnable_avg_period : 47882 [Wed Feb 6 09:40:49 2019][1696549.039259] [Wed Feb 6 09:40:49 2019][1696549.040935] rt_rq[22]:/ [Wed Feb 6 09:40:49 2019][1696549.043561] .rt_nr_running : 0 [Wed Feb 6 09:40:49 2019][1696549.048354] .rt_throttled : 0 [Wed Feb 6 09:40:49 2019][1696549.053146] .rt_time : 0.000000 [Wed Feb 6 09:40:49 2019][1696549.058563] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:49 2019][1696549.064223] [Wed Feb 6 09:40:49 2019][1696549.065896] runnable tasks: [Wed Feb 6 09:40:49 2019][1696549.068868] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:49 2019][1696549.079814] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:49 2019][1696549.090690] watchdog/22 121 -11.979391 424071 0 0.000000 167535.699471 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.102028] migration/22 122 0.000000 2581 0 0.000000 669935.164856 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.113364] ksoftirqd/22 123 16071284.916137 15856352 120 0.000000 149061.851349 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.124699] kworker/22:0H 125 4440.706622 9 100 0.000000 0.036260 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.136074] kworker/22:1H 16787 16071276.330359 14035691 100 0.000000 68635.634936 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.147415] dsm_sa_eventmgr 17568 4453.626209 25 120 0.000000 2.703020 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.158756] rsyslogd 19079 16070915.211103 2840 120 0.000000 56.972655 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.170090] reader#0 19098 16071369.729795 6965339 120 0.000000 599954.255821 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.181432] cfs_rh_02 5815 16057202.134196 1218 120 0.000000 1709.190929 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.192786] ptlrpcd_02_04 5924 16071367.100995 13154461 120 0.000000 187762.896245 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.204125] ptlrpcd_02_08 5928 16071364.012548 13180512 120 0.000000 187247.089786 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.215472] mdt02_000 6173 15469694.418125 95065314 120 0.000000 4602102.362059 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.226805] mdt02_001 6174 15468884.430594 88760036 120 0.000000 3982045.326352 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.238152] osp-syn-4-0 6296 16071004.504993 788928 120 0.000000 20937.813934 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.249486] osp-syn-45-0 6308 16071351.160035 778869 120 0.000000 20777.114106 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.260822] osp-pre-45-0 6309 16071363.996926 247856 120 0.000000 4641.844325 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.272166] osp-syn-2-2 6460 16071350.733897 445166 120 0.000000 12377.933966 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.283506] osp-syn-33-2 6490 16071301.953427 445519 120 0.000000 12360.347353 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.294848] ldlm_cn02_004 6764 16056801.413158 21114426 120 0.000000 625257.432310 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.306210] mdt02_021 9171 15530318.546919 88823757 120 0.000000 4128452.259423 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.317547] ldlm_cn02_008 9191 16056804.627399 22195973 120 0.000000 745868.292644 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.328894] mdt_io02_024 20150 15557675.110321 4195688 120 0.000000 175543.436024 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.340236] mdt_out02_005 23072 16071188.970727 4329358 120 0.000000 100539.510547 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.351575] ldlm_cn02_014 23317 16056803.636886 13459507 120 0.000000 473241.333554 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.362912] ldlm_cn02_020 23325 16071366.983427 17594358 120 0.000000 579949.064820 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.374246] ldlm_cn02_021 23326 16071154.567944 8476385 120 0.000000 411817.337763 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.385586] mdt02_024 35530 16015358.956354 42381073 120 0.000000 1705336.683451 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.396923] kworker/22:1 11151 16071367.023538 67961 120 0.000000 11492.105760 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.408257] mdt02_029 15181 16071374.189870 9527913 120 0.000000 11337408.034681 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.419589] mdt02_030 15182 15468884.430474 8854889 120 0.000000 422120.603328 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.430927] mdt02_033 15185 15468784.792957 2508048 120 0.000000 663212.621097 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.442267] mdt02_036 15257 16071362.249720 2112150 120 0.000000 516874.111365 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.453601] mdt02_042 15330 15468784.792476 10383625 120 0.000000 774887.598332 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.464963] mdt_io02_045 27700 16031065.976951 43022 120 0.000000 2528.323946 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.476309] mdt_io02_082 27773 15546209.566508 31890 120 0.000000 1229.088548 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.487645] mdt_io02_089 27780 15462588.516322 87 120 0.000000 3.582887 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.498983] ldlm_cn02_028 28532 16056804.482644 17692 120 0.000000 11084.077983 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.510318] ldlm_cn02_030 28946 16071363.937377 8759 120 0.000000 6546.153694 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.521662] kworker/22:0 29879 15983951.654229 14 120 0.000000 0.105049 0.000000 2 / [Wed Feb 6 09:40:49 2019][1696549.533019] [Wed Feb 6 09:40:49 2019][1696549.534694] cpu#23, 1996.199 MHz [Wed Feb 6 09:40:49 2019][1696549.538112] .nr_running : 0 [Wed Feb 6 09:40:49 2019][1696549.542912] .load : 0 [Wed Feb 6 09:40:49 2019][1696549.547721] .nr_switches : 594920847 [Wed Feb 6 09:40:49 2019][1696549.553217] .nr_load_updates : 227997738 [Wed Feb 6 09:40:49 2019][1696549.558717] .nr_uninterruptible : -30009 [Wed Feb 6 09:40:49 2019][1696549.563960] .next_balance : 5991.173446 [Wed Feb 6 09:40:49 2019][1696549.569624] .curr->pid : 0 [Wed Feb 6 09:40:49 2019][1696549.574426] .clock : 1696549573.715723 [Wed Feb 6 09:40:49 2019][1696549.580612] .cpu_load[0] : 0 [Wed Feb 6 09:40:49 2019][1696549.585414] .cpu_load[1] : 0 [Wed Feb 6 09:40:49 2019][1696549.590216] .cpu_load[2] : 0 [Wed Feb 6 09:40:49 2019][1696549.595019] .cpu_load[3] : 0 [Wed Feb 6 09:40:49 2019][1696549.599821] .cpu_load[4] : 0 [Wed Feb 6 09:40:49 2019][1696549.604620] .avg_idle : 5263746 [Wed Feb 6 09:40:49 2019][1696549.609945] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:49 2019][1696549.615261] [Wed Feb 6 09:40:49 2019][1696549.616937] cfs_rq[23]:/ [Wed Feb 6 09:40:49 2019][1696549.619650] .exec_clock : 0.000000 [Wed Feb 6 09:40:49 2019][1696549.625048] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:49 2019][1696549.630447] .min_vruntime : 30363084.038831 [Wed Feb 6 09:40:49 2019][1696549.636455] .max_vruntime : 0.000001 [Wed Feb 6 09:40:49 2019][1696549.641854] .spread : 0.000000 [Wed Feb 6 09:40:49 2019][1696549.647254] .spread0 : 2237471.230710 [Wed Feb 6 09:40:49 2019][1696549.653171] .nr_spread_over : 0 [Wed Feb 6 09:40:49 2019][1696549.657964] .nr_running : 0 [Wed Feb 6 09:40:49 2019][1696549.662759] .load : 0 [Wed Feb 6 09:40:49 2019][1696549.667551] .runnable_load_avg : 0 [Wed Feb 6 09:40:49 2019][1696549.672346] .blocked_load_avg : 4 [Wed Feb 6 09:40:49 2019][1696549.677136] .tg_load_avg : 0 [Wed Feb 6 09:40:49 2019][1696549.681928] .tg_load_contrib : 0 [Wed Feb 6 09:40:49 2019][1696549.686721] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:49 2019][1696549.691515] .tg->runnable_avg : 0 [Wed Feb 6 09:40:49 2019][1696549.696308] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:49 2019][1696549.701101] .throttled : 0 [Wed Feb 6 09:40:49 2019][1696549.705892] .throttle_count : 0 [Wed Feb 6 09:40:49 2019][1696549.710687] .avg->runnable_avg_sum : 173 [Wed Feb 6 09:40:49 2019][1696549.715652] .avg->runnable_avg_period : 48076 [Wed Feb 6 09:40:49 2019][1696549.720795] [Wed Feb 6 09:40:49 2019][1696549.722463] rt_rq[23]:/ [Wed Feb 6 09:40:49 2019][1696549.725092] .rt_nr_running : 0 [Wed Feb 6 09:40:49 2019][1696549.729881] .rt_throttled : 0 [Wed Feb 6 09:40:49 2019][1696549.730729] LustreError: 53421:0:(upcall_cache.c:233:upcall_cache_get_entry()) acquire for key 0: error -110 [Wed Feb 6 09:40:49 2019][1696549.730732] LustreError: 53421:0:(upcall_cache.c:233:upcall_cache_get_entry()) Skipped 2 previous similar messages [Wed Feb 6 09:40:49 2019][1696549.755183] .rt_time : 0.000000 [Wed Feb 6 09:40:49 2019][1696549.760582] .rt_runtime : 920.310656 [Wed Feb 6 09:40:49 2019][1696549.766153] [Wed Feb 6 09:40:49 2019][1696549.767826] runnable tasks: [Wed Feb 6 09:40:49 2019][1696549.770799] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:49 2019][1696549.781746] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:49 2019][1696549.792622] watchdog/23 126 -11.980192 424096 0 0.000000 111296.253578 0.000000 3 / [Wed Feb 6 09:40:49 2019][1696549.803960] migration/23 127 0.000000 2872 0 0.000000 437328.825834 0.000000 3 / [Wed Feb 6 09:40:49 2019][1696549.815296] ksoftirqd/23 128 30363071.980260 4358108 120 0.000000 47052.820272 0.000000 3 / [Wed Feb 6 09:40:49 2019][1696549.826631] kworker/23:0H 130 3990.492011 10 100 0.000000 0.038655 0.000000 3 / [Wed Feb 6 09:40:49 2019][1696549.837990] poll_mpt3sas0_h 5684 29377752.202942 5 100 0.000000 0.063771 0.000000 3 / [Wed Feb 6 09:40:49 2019][1696549.849325] mlx5_cmd_0000:8 7346 29377752.202709 5 100 0.000000 0.031619 0.000000 3 / [Wed Feb 6 09:40:49 2019][1696549.860659] poll_mpt3sas0_s 7530 29377752.202941 4 100 0.000000 0.046117 0.000000 3 / [Wed Feb 6 09:40:49 2019][1696549.871999] multipathd 13038 4344.865380 39 120 0.000000 1.974009 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696549.883330] multipathd 13048 0.000000 77 0 0.000000 4.364739 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696549.894670] kdmwork-253:1 17150 30168323.902052 4917 120 0.000000 44.525957 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696549.906002] bioset 17264 3923.371332 2 100 0.000000 0.011592 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696549.917347] kworker/23:1H 18429 30363071.962478 5436890 100 0.000000 18577.714933 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696549.928691] qmgr 19491 30363005.572868 10337 120 0.000000 576.689432 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696549.940051] ptlrpcd_03_01 5933 30363072.170081 13125231 120 0.000000 199676.717878 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696549.951384] ptlrpcd_03_07 5939 30363072.067135 13143034 120 0.000000 195160.300497 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696549.962722] ldlm_cb03_001 6028 30308333.855952 3624241 120 0.000000 60459.140965 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696549.974069] osp-pre-12-0 6249 30363072.066053 239500 120 0.000000 7047.966128 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696549.985402] osp-pre-9-0 6269 30363072.055463 247817 120 0.000000 5875.368480 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696549.996736] osp-pre-18-0 6277 30363072.052086 247825 120 0.000000 6760.881656 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.008073] osp-syn-16-0 6292 30309363.081726 785845 120 0.000000 21103.303073 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.019411] osp-pre-4-0 6297 30363072.054040 247812 120 0.000000 4378.897045 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.030748] osp-syn-35-0 6336 30363036.752855 788498 120 0.000000 20878.511590 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.042083] osp-pre-34-0 6343 30363072.067816 247840 120 0.000000 3577.788296 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.053419] osp-pre-3-2 6437 30363072.053819 205445 120 0.000000 3329.812209 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.064756] osp-pre-14-2 6451 30363072.054781 235167 120 0.000000 6493.618687 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.076091] osp-syn-9-2 6458 30363036.666209 434794 120 0.000000 12082.628974 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.087426] osp-syn-19-2 6464 30363036.686406 438577 120 0.000000 12595.712954 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.098764] osp-pre-18-2 6467 30363072.048259 235077 120 0.000000 4903.378135 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.110100] osp-pre-5-2 6475 30363072.055964 234749 120 0.000000 4442.389866 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.121436] osp-syn-20-2 6492 30363033.531717 444899 120 0.000000 12550.181220 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.132776] osp-syn-28-2 6528 30363036.673242 435691 120 0.000000 12471.935093 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.144114] ldlm_cn03_004 6770 30363082.964410 8421979 120 0.000000 317762.671833 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.155457] mdt03_008 6845 30363079.654642 35485804 120 0.000000 1565081.394992 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.166813] mdt03_012 10287 29440569.477702 32079247 120 0.000000 1294827.703447 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.178174] kworker/23:0 33134 30363072.058319 435029 120 0.000000 8551.981661 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.189508] ldlm_cn03_015 63246 30167955.971931 4274645 120 0.000000 178976.899496 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.200842] ldlm_cn03_017 63248 30283675.718183 3704185 120 0.000000 212095.381409 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.212200] mdt03_016 17052 30363072.078366 8378608 120 0.000000 798158.596586 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.223534] mdt03_024 17219 30363081.552897 10872987 120 0.000000 730950.846486 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.234868] mdt03_028 17223 30363084.038831 8030950 120 0.000000 465872.402121 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.246206] mdt03_031 24224 30363072.177584 1913161 120 0.000000 45696.902239 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.257560] kworker/23:2 28109 29169019.256838 4 120 0.000000 0.022012 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.268896] ldlm_cn03_026 28521 30293083.183695 19851 120 0.000000 21158.121757 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.280232] ldlm_cn03_028 28536 30167955.972070 57554 120 0.000000 12135.675226 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.291571] mdt_io03_031 28969 29669728.005493 3930 120 0.000000 3150.108660 0.000000 3 / [Wed Feb 6 09:40:50 2019][1696550.302931] [Wed Feb 6 09:40:50 2019][1696550.304609] cpu#24, 1996.199 MHz [Wed Feb 6 09:40:50 2019][1696550.308025] .nr_running : 0 [Wed Feb 6 09:40:50 2019][1696550.312828] .load : 0 [Wed Feb 6 09:40:50 2019][1696550.317628] .nr_switches : 4375784717 [Wed Feb 6 09:40:50 2019][1696550.323209] .nr_load_updates : 450922258 [Wed Feb 6 09:40:50 2019][1696550.328705] .nr_uninterruptible : -563013 [Wed Feb 6 09:40:50 2019][1696550.334027] .next_balance : 5991.174216 [Wed Feb 6 09:40:50 2019][1696550.339694] .curr->pid : 0 [Wed Feb 6 09:40:50 2019][1696550.344495] .clock : 1696550343.734905 [Wed Feb 6 09:40:50 2019][1696550.350679] .cpu_load[0] : 0 [Wed Feb 6 09:40:50 2019][1696550.355477] .cpu_load[1] : 0 [Wed Feb 6 09:40:50 2019][1696550.360273] .cpu_load[2] : 0 [Wed Feb 6 09:40:50 2019][1696550.365071] .cpu_load[3] : 0 [Wed Feb 6 09:40:50 2019][1696550.369869] .cpu_load[4] : 0 [Wed Feb 6 09:40:50 2019][1696550.374669] .avg_idle : 5636856 [Wed Feb 6 09:40:50 2019][1696550.379983] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:50 2019][1696550.385303] [Wed Feb 6 09:40:50 2019][1696550.386973] cfs_rq[24]:/ [Wed Feb 6 09:40:50 2019][1696550.389685] .exec_clock : 0.000000 [Wed Feb 6 09:40:50 2019][1696550.395085] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:50 2019][1696550.400483] .min_vruntime : 26836191.338917 [Wed Feb 6 09:40:50 2019][1696550.406489] .max_vruntime : 0.000001 [Wed Feb 6 09:40:50 2019][1696550.411890] .spread : 0.000000 [Wed Feb 6 09:40:50 2019][1696550.417289] .spread0 : -1289424.134161 [Wed Feb 6 09:40:50 2019][1696550.423294] .nr_spread_over : 0 [Wed Feb 6 09:40:50 2019][1696550.428089] .nr_running : 0 [Wed Feb 6 09:40:50 2019][1696550.432880] .load : 0 [Wed Feb 6 09:40:50 2019][1696550.437672] .runnable_load_avg : 0 [Wed Feb 6 09:40:50 2019][1696550.442467] .blocked_load_avg : 0 [Wed Feb 6 09:40:50 2019][1696550.447257] .tg_load_avg : 0 [Wed Feb 6 09:40:50 2019][1696550.452050] .tg_load_contrib : 0 [Wed Feb 6 09:40:50 2019][1696550.456845] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:50 2019][1696550.461636] .tg->runnable_avg : 0 [Wed Feb 6 09:40:50 2019][1696550.466430] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:50 2019][1696550.471224] .throttled : 0 [Wed Feb 6 09:40:50 2019][1696550.476015] .throttle_count : 0 [Wed Feb 6 09:40:50 2019][1696550.480809] .avg->runnable_avg_sum : 29 [Wed Feb 6 09:40:50 2019][1696550.485689] .avg->runnable_avg_period : 47486 [Wed Feb 6 09:40:50 2019][1696550.490832] [Wed Feb 6 09:40:50 2019][1696550.492499] rt_rq[24]:/ [Wed Feb 6 09:40:50 2019][1696550.495125] .rt_nr_running : 0 [Wed Feb 6 09:40:50 2019][1696550.499918] .rt_throttled : 0 [Wed Feb 6 09:40:50 2019][1696550.504711] .rt_time : 0.000000 [Wed Feb 6 09:40:50 2019][1696550.510111] .rt_runtime : 979.166667 [Wed Feb 6 09:40:50 2019][1696550.515683] [Wed Feb 6 09:40:50 2019][1696550.517355] runnable tasks: [Wed Feb 6 09:40:50 2019][1696550.520328] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:50 2019][1696550.531274] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:50 2019][1696550.542151] watchdog/24 131 -11.982015 424105 0 0.000000 152984.619318 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.553488] migration/24 132 0.000000 2648 0 0.000000 459321.932365 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.564824] ksoftirqd/24 133 26835843.309465 8598741 120 0.000000 91028.668421 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.576162] kworker/24:0H 135 4266.286378 10 100 0.000000 0.050098 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.587518] kworker/24:1 318 26836178.910876 717289 120 0.000000 13691.742513 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.598875] kworker/24:1H 18903 26835843.307457 9973550 100 0.000000 47536.932715 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.610229] kiblnd_sd_00_01 5876 26836191.508426 4154609307 120 0.000000 27485503.038315 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.621661] ptlrpcd_00_04 5900 26836172.168737 6018437 120 0.000000 90020.656656 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.633010] ldlm_cn00_001 6030 26772800.421396 13696120 120 0.000000 530834.568313 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.644364] osp-syn-37-0 6330 25543524.419707 778736 120 0.000000 20737.504362 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.655709] osp-pre-3-2 6517 26836175.896087 234810 120 0.000000 6600.741483 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.667044] mdt00_003 6613 26030459.252369 210190248 120 0.000000 7215467.703023 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.678385] mdt00_007 6795 26836160.767729 231160197 120 0.000000 8032669.107621 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.689729] mdt_io00_009 7097 26025397.387317 4875561 120 0.000000 244370.785537 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.701067] mdt00_013 8967 26030459.251708 238310115 120 0.000000 8188038.458513 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.712405] mdt_io00_017 8986 26025397.387086 4919146 120 0.000000 186993.360128 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.723742] mdt_io00_018 8993 26025397.409127 5212741 120 0.000000 208082.588805 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.735083] mdt_io00_021 9188 26025409.355115 8028135 120 0.000000 324039.827387 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.746423] ldlm_cn00_025 20158 26772504.631154 10711287 120 0.000000 528491.709476 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.757788] mdt00_046 15391 26030459.251558 4916105 120 0.000000 624100.051056 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.769139] kworker/24:0 27009 25509307.812371 20 120 0.000000 0.126128 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.780482] mdt_io00_062 27734 25533557.308927 225 120 0.000000 7.993067 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.791818] mdt_io00_063 27736 26025397.402014 17585 120 0.000000 3027.362272 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.803179] l_getidentity 30458 26835896.248163 2 120 0.000000 0.831691 0.000000 0 / [Wed Feb 6 09:40:50 2019][1696550.814536] [Wed Feb 6 09:40:50 2019][1696550.816215] cpu#25, 1996.199 MHz [Wed Feb 6 09:40:50 2019][1696550.819633] .nr_running : 0 [Wed Feb 6 09:40:50 2019][1696550.824435] .load : 0 [Wed Feb 6 09:40:50 2019][1696550.829230] .nr_switches : 2956878156 [Wed Feb 6 09:40:50 2019][1696550.834803] .nr_load_updates : 362250984 [Wed Feb 6 09:40:50 2019][1696550.840295] .nr_uninterruptible : 150697 [Wed Feb 6 09:40:50 2019][1696550.845528] .next_balance : 5991.174727 [Wed Feb 6 09:40:50 2019][1696550.851193] .curr->pid : 0 [Wed Feb 6 09:40:50 2019][1696550.855996] .clock : 1696550855.747224 [Wed Feb 6 09:40:50 2019][1696550.862186] .cpu_load[0] : 0 [Wed Feb 6 09:40:51 2019][1696550.866984] .cpu_load[1] : 0 [Wed Feb 6 09:40:51 2019][1696550.871785] .cpu_load[2] : 0 [Wed Feb 6 09:40:51 2019][1696550.876578] .cpu_load[3] : 0 [Wed Feb 6 09:40:51 2019][1696550.881370] .cpu_load[4] : 0 [Wed Feb 6 09:40:51 2019][1696550.886168] .avg_idle : 8397800 [Wed Feb 6 09:40:51 2019][1696550.891489] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:51 2019][1696550.896809] [Wed Feb 6 09:40:51 2019][1696550.898484] cfs_rq[25]:/ [Wed Feb 6 09:40:51 2019][1696550.901196] .exec_clock : 0.000000 [Wed Feb 6 09:40:51 2019][1696550.906596] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:51 2019][1696550.911993] .min_vruntime : 19452778.197309 [Wed Feb 6 09:40:51 2019][1696550.918002] .max_vruntime : 0.000001 [Wed Feb 6 09:40:51 2019][1696550.923400] .spread : 0.000000 [Wed Feb 6 09:40:51 2019][1696550.928801] .spread0 : -8672839.268749 [Wed Feb 6 09:40:51 2019][1696550.934806] .nr_spread_over : 0 [Wed Feb 6 09:40:51 2019][1696550.939598] .nr_running : 0 [Wed Feb 6 09:40:51 2019][1696550.944391] .load : 0 [Wed Feb 6 09:40:51 2019][1696550.949183] .runnable_load_avg : 0 [Wed Feb 6 09:40:51 2019][1696550.953978] .blocked_load_avg : 0 [Wed Feb 6 09:40:51 2019][1696550.958769] .tg_load_avg : 0 [Wed Feb 6 09:40:51 2019][1696550.963563] .tg_load_contrib : 0 [Wed Feb 6 09:40:51 2019][1696550.968356] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:51 2019][1696550.973149] .tg->runnable_avg : 0 [Wed Feb 6 09:40:51 2019][1696550.977941] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:51 2019][1696550.982733] .throttled : 0 [Wed Feb 6 09:40:51 2019][1696550.987527] .throttle_count : 0 [Wed Feb 6 09:40:51 2019][1696550.992319] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:40:51 2019][1696550.997112] .avg->runnable_avg_period : 46377 [Wed Feb 6 09:40:51 2019][1696551.002254] [Wed Feb 6 09:40:51 2019][1696551.003925] rt_rq[25]:/ [Wed Feb 6 09:40:51 2019][1696551.006551] .rt_nr_running : 0 [Wed Feb 6 09:40:51 2019][1696551.011343] .rt_throttled : 0 [Wed Feb 6 09:40:51 2019][1696551.016134] .rt_time : 0.000000 [Wed Feb 6 09:40:51 2019][1696551.021535] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:51 2019][1696551.027194] [Wed Feb 6 09:40:51 2019][1696551.028866] runnable tasks: [Wed Feb 6 09:40:51 2019][1696551.031839] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:51 2019][1696551.042786] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:51 2019][1696551.053664] watchdog/25 136 -11.985983 424068 0 0.000000 244794.101849 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.065000] migration/25 137 0.000000 2704 0 0.000000 343949.440983 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.076334] ksoftirqd/25 138 19452678.988741 9154043 120 0.000000 83911.510343 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.087672] kworker/25:0H 140 4676.827895 9 100 0.000000 0.034908 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.099040] kworker/25:1H 16785 19452660.209132 5973014 100 0.000000 23459.405161 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.110383] kworker/25:2 17391 19452764.042877 599772 120 0.000000 23446.712276 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.121718] dsm_sa_eventmgr 17587 19452755.076784 559699 120 0.000000 9365.833316 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.133066] ptlrpc_hr01_000 5841 19452756.301359 121146799 120 0.000000 1154287.166290 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.144410] ptlrpcd_01_08 5916 19452766.282631 6967048 120 0.000000 103895.698071 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.155763] osp-pre-46-0 6255 19452766.220854 247816 120 0.000000 4308.326360 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.167098] osp-pre-21-0 6291 19452766.208861 247507 120 0.000000 3782.556380 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.178433] osp-syn-38-0 6316 19452692.770971 778385 120 0.000000 20912.565372 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.189769] osp-pre-8-0 6335 19452765.624408 247812 120 0.000000 3768.837253 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.201112] osp-pre-25-2 6469 19452765.650547 234956 120 0.000000 6883.870474 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.212455] osp-syn-8-2 6524 19452588.173280 443743 120 0.000000 12309.339690 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.223789] mdt01_005 6683 19452767.998989 112880985 120 0.000000 4529078.396243 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.235125] ldlm_cn01_006 6772 19449958.980841 30337981 120 0.000000 797540.168758 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.246461] mdt_io01_008 6866 19452678.987529 1887279 120 0.000000 74439.575327 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.257804] mdt_io01_014 8952 19452679.156770 6509252 120 0.000000 244718.656749 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.269155] mdt_out01_006 13924 19452707.612247 22590224 120 0.000000 471571.813621 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.280505] mdt_io01_057 20878 16900505.609385 7516177 120 0.000000 281042.747877 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.291856] mdt01_045 15188 17322692.972507 2222497 120 0.000000 102469.619791 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.303193] mdt01_049 15229 19452728.725351 9262140 120 0.000000 524354.032959 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.314532] mdt01_064 15291 17322692.976905 8628066 120 0.000000 372840.004512 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.325863] mdt01_066 15293 17323409.451722 8178381 120 0.000000 506339.782016 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.337207] mdt_io01_079 15460 19452678.989633 18858 120 0.000000 716.174212 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.348557] kworker/25:1 26159 16683198.605761 6 120 0.000000 0.041017 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.359901] mdt_io01_105 27751 16900495.746656 654 120 0.000000 26.509181 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.371236] mdt_io01_106 27754 19452678.980034 1369 120 0.000000 174.840958 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.382572] mdt_io01_109 27760 19452678.979233 21887 120 0.000000 785.096054 0.000000 1 / [Wed Feb 6 09:40:51 2019][1696551.393947] [Wed Feb 6 09:40:51 2019][1696551.395621] cpu#26, 1996.199 MHz [Wed Feb 6 09:40:51 2019][1696551.399039] .nr_running : 0 [Wed Feb 6 09:40:51 2019][1696551.403840] .load : 0 [Wed Feb 6 09:40:51 2019][1696551.408642] .nr_switches : 4379086262 [Wed Feb 6 09:40:51 2019][1696551.414222] .nr_load_updates : 380355392 [Wed Feb 6 09:40:51 2019][1696551.419713] .nr_uninterruptible : 28789 [Wed Feb 6 09:40:51 2019][1696551.424855] .next_balance : 5991.175309 [Wed Feb 6 09:40:51 2019][1696551.430523] .curr->pid : 0 [Wed Feb 6 09:40:51 2019][1696551.435329] .clock : 1696551434.757724 [Wed Feb 6 09:40:51 2019][1696551.441516] .cpu_load[0] : 0 [Wed Feb 6 09:40:51 2019][1696551.446313] .cpu_load[1] : 0 [Wed Feb 6 09:40:51 2019][1696551.451107] .cpu_load[2] : 0 [Wed Feb 6 09:40:51 2019][1696551.455912] .cpu_load[3] : 0 [Wed Feb 6 09:40:51 2019][1696551.460707] .cpu_load[4] : 0 [Wed Feb 6 09:40:51 2019][1696551.465501] .avg_idle : 10000000 [Wed Feb 6 09:40:51 2019][1696551.470901] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:51 2019][1696551.476213] [Wed Feb 6 09:40:51 2019][1696551.477881] cfs_rq[26]:/ [Wed Feb 6 09:40:51 2019][1696551.480595] .exec_clock : 0.000000 [Wed Feb 6 09:40:51 2019][1696551.485992] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:51 2019][1696551.491394] .min_vruntime : 17606154.739408 [Wed Feb 6 09:40:51 2019][1696551.497398] .max_vruntime : 0.000001 [Wed Feb 6 09:40:51 2019][1696551.502798] .spread : 0.000000 [Wed Feb 6 09:40:51 2019][1696551.508199] .spread0 : -10519464.929265 [Wed Feb 6 09:40:51 2019][1696551.514290] .nr_spread_over : 0 [Wed Feb 6 09:40:51 2019][1696551.519083] .nr_running : 0 [Wed Feb 6 09:40:51 2019][1696551.523876] .load : 0 [Wed Feb 6 09:40:51 2019][1696551.528670] .runnable_load_avg : 0 [Wed Feb 6 09:40:51 2019][1696551.533462] .blocked_load_avg : 0 [Wed Feb 6 09:40:51 2019][1696551.538254] .tg_load_avg : 0 [Wed Feb 6 09:40:51 2019][1696551.543047] .tg_load_contrib : 0 [Wed Feb 6 09:40:51 2019][1696551.547839] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:51 2019][1696551.552633] .tg->runnable_avg : 0 [Wed Feb 6 09:40:51 2019][1696551.557426] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:51 2019][1696551.562220] .throttled : 0 [Wed Feb 6 09:40:51 2019][1696551.567012] .throttle_count : 0 [Wed Feb 6 09:40:51 2019][1696551.571805] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:40:51 2019][1696551.576596] .avg->runnable_avg_period : 47810 [Wed Feb 6 09:40:51 2019][1696551.581742] [Wed Feb 6 09:40:51 2019][1696551.583410] rt_rq[26]:/ [Wed Feb 6 09:40:51 2019][1696551.586035] .rt_nr_running : 0 [Wed Feb 6 09:40:51 2019][1696551.590827] .rt_throttled : 0 [Wed Feb 6 09:40:51 2019][1696551.595620] .rt_time : 0.000000 [Wed Feb 6 09:40:51 2019][1696551.601021] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:51 2019][1696551.606680] [Wed Feb 6 09:40:51 2019][1696551.608351] runnable tasks: [Wed Feb 6 09:40:51 2019][1696551.611326] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:51 2019][1696551.622272] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:51 2019][1696551.633148] watchdog/26 141 -11.986114 424061 0 0.000000 59260.231056 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.644485] migration/26 142 0.000000 1905 0 0.000000 640942.158997 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.655822] ksoftirqd/26 143 17606069.007411 3045047 120 0.000000 52598.849631 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.667156] kworker/26:0H 145 5046.160878 10 100 0.000000 0.036156 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.678513] kworker/26:1 7271 17606117.603030 593832 120 0.000000 10431.212743 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.689862] auditd 17474 17606049.299222 13573 116 0.000000 2591.734944 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.701213] kworker/26:1H 30703 17606068.733805 1511658 100 0.000000 6133.040838 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.712561] ptlrpcd_02_02 5922 17606143.098045 13200400 120 0.000000 191459.379313 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.723908] ldlm_cn02_001 6036 17606142.746822 21844742 120 0.000000 596164.659985 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.735248] ldlm_cn02_003 6072 17569382.492233 23173023 120 0.000000 798696.911250 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.746582] mdt02_002 6175 17606142.625339 85936059 120 0.000000 4289851.671067 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.757936] osp-pre-2-2 6461 17606142.775757 234946 120 0.000000 3351.150887 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.769280] mdt02_003 6617 17606142.765377 96955335 120 0.000000 4221888.746006 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.780620] mdt_io02_004 6782 17292285.498071 7214959 120 0.000000 286576.685047 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.791959] mdt_io02_010 7109 17292285.626053 4612575 120 0.000000 176793.991702 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.803305] mdt_io02_016 8980 17292285.541082 6804428 120 0.000000 259121.050825 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.814641] mdt02_014 8982 17606142.741983 73862788 120 0.000000 3178110.258269 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.825980] mdt_io02_020 9062 17292467.645622 7212126 120 0.000000 277815.810369 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.837353] mdt02_037 15265 17606068.735746 3042732 120 0.000000 108446.686855 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.848711] mdt_io02_042 27686 17292285.658084 432 120 0.000000 18.573711 0.000000 2 / [Wed Feb 6 09:40:51 2019][1696551.860051] mdt_io02_047 27702 17292285.545610 2221 120 0.000000 89.974657 0.000000 2 / [Wed Feb 6 09:40:52 2019][1696551.871385] mdt_io02_048 27710 16671906.924388 192 120 0.000000 7.518841 0.000000 2 / [Wed Feb 6 09:40:52 2019][1696551.882723] mdt_io02_063 27743 17606119.804231 17148 120 0.000000 2103.915421 0.000000 2 / [Wed Feb 6 09:40:52 2019][1696551.894063] mdt_io02_085 27776 17292285.522788 676 120 0.000000 2785.962734 0.000000 2 / [Wed Feb 6 09:40:52 2019][1696551.905410] ll_mgs_0013 28981 17297357.118485 94 120 0.000000 109.273892 0.000000 2 / [Wed Feb 6 09:40:52 2019][1696551.916745] ll_mgs_0014 28982 17297356.049904 136 120 0.000000 2256.863447 0.000000 2 / [Wed Feb 6 09:40:52 2019][1696551.928088] kworker/26:2 29612 17501222.421471 2 120 0.000000 0.003476 0.000000 2 / [Wed Feb 6 09:40:52 2019][1696551.939443] [Wed Feb 6 09:40:52 2019][1696551.941124] cpu#27, 1996.199 MHz [Wed Feb 6 09:40:52 2019][1696551.944541] .nr_running : 0 [Wed Feb 6 09:40:52 2019][1696551.949342] .load : 0 [Wed Feb 6 09:40:52 2019][1696551.954145] .nr_switches : 479555168 [Wed Feb 6 09:40:52 2019][1696551.959640] .nr_load_updates : 161900775 [Wed Feb 6 09:40:52 2019][1696551.965133] .nr_uninterruptible : 24509 [Wed Feb 6 09:40:52 2019][1696551.970278] .next_balance : 5991.175858 [Wed Feb 6 09:40:52 2019][1696551.975947] .curr->pid : 0 [Wed Feb 6 09:40:52 2019][1696551.980751] .clock : 1696551979.770941 [Wed Feb 6 09:40:52 2019][1696551.986935] .cpu_load[0] : 0 [Wed Feb 6 09:40:52 2019][1696551.991745] .cpu_load[1] : 0 [Wed Feb 6 09:40:52 2019][1696551.996543] .cpu_load[2] : 0 [Wed Feb 6 09:40:52 2019][1696552.001344] .cpu_load[3] : 0 [Wed Feb 6 09:40:52 2019][1696552.006145] .cpu_load[4] : 0 [Wed Feb 6 09:40:52 2019][1696552.010947] .avg_idle : 10000000 [Wed Feb 6 09:40:52 2019][1696552.016352] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:52 2019][1696552.021672] [Wed Feb 6 09:40:52 2019][1696552.023341] cfs_rq[27]:/ [Wed Feb 6 09:40:52 2019][1696552.026053] .exec_clock : 0.000000 [Wed Feb 6 09:40:52 2019][1696552.031452] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:52 2019][1696552.036853] .min_vruntime : 21088450.583876 [Wed Feb 6 09:40:52 2019][1696552.042857] .max_vruntime : 0.000001 [Wed Feb 6 09:40:52 2019][1696552.048257] .spread : 0.000000 [Wed Feb 6 09:40:52 2019][1696552.053657] .spread0 : -7037171.169981 [Wed Feb 6 09:40:52 2019][1696552.059663] .nr_spread_over : 0 [Wed Feb 6 09:40:52 2019][1696552.064456] .nr_running : 0 [Wed Feb 6 09:40:52 2019][1696552.069250] .load : 0 [Wed Feb 6 09:40:52 2019][1696552.074041] .runnable_load_avg : 0 [Wed Feb 6 09:40:52 2019][1696552.078835] .blocked_load_avg : 0 [Wed Feb 6 09:40:52 2019][1696552.083629] .tg_load_avg : 0 [Wed Feb 6 09:40:52 2019][1696552.088419] .tg_load_contrib : 0 [Wed Feb 6 09:40:52 2019][1696552.093213] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:52 2019][1696552.098005] .tg->runnable_avg : 0 [Wed Feb 6 09:40:52 2019][1696552.102800] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:52 2019][1696552.107592] .throttled : 0 [Wed Feb 6 09:40:52 2019][1696552.112384] .throttle_count : 0 [Wed Feb 6 09:40:52 2019][1696552.117178] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:40:52 2019][1696552.121969] .avg->runnable_avg_period : 47098 [Wed Feb 6 09:40:52 2019][1696552.127111] [Wed Feb 6 09:40:52 2019][1696552.128781] rt_rq[27]:/ [Wed Feb 6 09:40:52 2019][1696552.131409] .rt_nr_running : 0 [Wed Feb 6 09:40:52 2019][1696552.136200] .rt_throttled : 0 [Wed Feb 6 09:40:52 2019][1696552.140992] .rt_time : 0.014228 [Wed Feb 6 09:40:52 2019][1696552.146394] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:52 2019][1696552.152051] [Wed Feb 6 09:40:52 2019][1696552.153725] runnable tasks: [Wed Feb 6 09:40:52 2019][1696552.156697] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:52 2019][1696552.167643] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:52 2019][1696552.178521] watchdog/27 146 -11.986945 424094 0 0.000000 172442.097770 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.189856] migration/27 147 0.000000 2455 0 0.000000 479243.951975 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.201192] ksoftirqd/27 148 21088344.584300 1217735 120 0.000000 14331.207131 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.212529] kworker/27:0H 150 5007.300279 10 100 0.000000 0.039946 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.223888] kworker/27:1 7272 21088438.597421 201871 120 0.000000 5611.043140 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.235230] multipathd 13050 0.000000 339248 0 0.000000 60192.240757 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.246564] multipathd 13612 0.000000 3391287 0 0.000000 204778.384273 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.257915] kworker/27:1H 27809 21088438.583931 1001180 100 0.000000 3109.381505 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.269274] ldlm_cn03_002 6040 21068213.346675 7874944 120 0.000000 299918.629651 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.280610] ldlm_elt 6043 20993971.197599 4996 120 0.000000 11343.981150 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.291948] mdt03_001 6177 21088424.109377 35911254 120 0.000000 1750353.162609 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.303295] dist_txn-0 6417 21088344.686465 247 120 0.000000 14.849576 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.314678] mdt_out03_002 10218 21032683.153323 470656 120 0.000000 11264.676750 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.326029] ldlm_cn03_011 23312 21037125.655565 6476007 120 0.000000 275094.698314 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.337402] mdt_io03_017 17103 20545893.126818 280783 120 0.000000 10160.810316 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.348740] mdt03_021 17206 21088374.487298 6908035 120 0.000000 583873.156443 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.360080] mdt03_033 24900 21088450.583876 1998360 120 0.000000 195480.963905 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.371412] ldlm_cn03_020 25319 21037128.788824 49203 120 0.000000 136655.303842 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.382769] mdt_io03_029 28948 20545893.109695 6 120 0.000000 0.262028 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.394105] ll_mgs_0024 28992 20558389.616177 102 120 0.000000 123.159793 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.405439] kworker/27:0 29045 20564157.150358 11 120 0.000000 0.089782 0.000000 3 / [Wed Feb 6 09:40:52 2019][1696552.416799] [Wed Feb 6 09:40:52 2019][1696552.418479] cpu#28, 1996.199 MHz [Wed Feb 6 09:40:52 2019][1696552.421896] .nr_running : 0 [Wed Feb 6 09:40:52 2019][1696552.426698] .load : 0 [Wed Feb 6 09:40:52 2019][1696552.431496] .nr_switches : 4131414804 [Wed Feb 6 09:40:52 2019][1696552.437076] .nr_load_updates : 473975684 [Wed Feb 6 09:40:52 2019][1696552.442567] .nr_uninterruptible : -550742 [Wed Feb 6 09:40:52 2019][1696552.447888] .next_balance : 5991.176330 [Wed Feb 6 09:40:52 2019][1696552.453553] .curr->pid : 0 [Wed Feb 6 09:40:52 2019][1696552.458354] .clock : 1696552457.783625 [Wed Feb 6 09:40:52 2019][1696552.464533] .cpu_load[0] : 0 [Wed Feb 6 09:40:52 2019][1696552.469327] .cpu_load[1] : 0 [Wed Feb 6 09:40:52 2019][1696552.474132] .cpu_load[2] : 0 [Wed Feb 6 09:40:52 2019][1696552.478933] .cpu_load[3] : 0 [Wed Feb 6 09:40:52 2019][1696552.483730] .cpu_load[4] : 0 [Wed Feb 6 09:40:52 2019][1696552.488523] .avg_idle : 9607631 [Wed Feb 6 09:40:52 2019][1696552.493836] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:52 2019][1696552.499150] [Wed Feb 6 09:40:52 2019][1696552.500827] cfs_rq[28]:/ [Wed Feb 6 09:40:52 2019][1696552.503539] .exec_clock : 0.000000 [Wed Feb 6 09:40:52 2019][1696552.508939] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:52 2019][1696552.514338] .min_vruntime : 25092572.405613 [Wed Feb 6 09:40:52 2019][1696552.520343] .max_vruntime : 0.000001 [Wed Feb 6 09:40:52 2019][1696552.525741] .spread : 0.000000 [Wed Feb 6 09:40:52 2019][1696552.531142] .spread0 : -3033051.292328 [Wed Feb 6 09:40:52 2019][1696552.537148] .nr_spread_over : 0 [Wed Feb 6 09:40:52 2019][1696552.541941] .nr_running : 0 [Wed Feb 6 09:40:52 2019][1696552.546733] .load : 0 [Wed Feb 6 09:40:52 2019][1696552.551527] .runnable_load_avg : 0 [Wed Feb 6 09:40:52 2019][1696552.556320] .blocked_load_avg : 0 [Wed Feb 6 09:40:52 2019][1696552.561111] .tg_load_avg : 0 [Wed Feb 6 09:40:52 2019][1696552.565906] .tg_load_contrib : 0 [Wed Feb 6 09:40:52 2019][1696552.570698] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:52 2019][1696552.575492] .tg->runnable_avg : 0 [Wed Feb 6 09:40:52 2019][1696552.580285] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:52 2019][1696552.585077] .throttled : 0 [Wed Feb 6 09:40:52 2019][1696552.589868] .throttle_count : 0 [Wed Feb 6 09:40:52 2019][1696552.594662] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:40:52 2019][1696552.599456] .avg->runnable_avg_period : 47949 [Wed Feb 6 09:40:52 2019][1696552.604597] [Wed Feb 6 09:40:52 2019][1696552.606267] rt_rq[28]:/ [Wed Feb 6 09:40:52 2019][1696552.608893] .rt_nr_running : 0 [Wed Feb 6 09:40:52 2019][1696552.613686] .rt_throttled : 0 [Wed Feb 6 09:40:52 2019][1696552.618479] .rt_time : 0.000000 [Wed Feb 6 09:40:52 2019][1696552.623878] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:52 2019][1696552.629538] [Wed Feb 6 09:40:52 2019][1696552.631210] runnable tasks: [Wed Feb 6 09:40:52 2019][1696552.634183] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:52 2019][1696552.645128] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:52 2019][1696552.656008] watchdog/28 151 -11.988317 424032 0 0.000000 81821.540699 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.667341] migration/28 152 0.000000 1865 0 0.000000 736124.150946 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.678678] ksoftirqd/28 153 25092548.485156 9614969 120 0.000000 97391.273681 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.690014] kworker/28:0 154 25092550.719996 830194 120 0.000000 15947.513469 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.701350] kworker/28:0H 155 5348.198982 10 100 0.000000 0.044824 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.712711] ccp-2-q0 10521 2560.022400 2 120 0.000000 0.012755 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.724049] ccp-2-q3 10550 2589.106690 2 120 0.000000 0.008587 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.735388] ccp-7-q2 10865 2662.352541 2 120 0.000000 0.015420 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.746740] kworker/28:1H 29233 25092548.410987 9886822 100 0.000000 45674.744491 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.758088] ptlrpcd_00_05 5901 25092559.101410 6059803 120 0.000000 84542.249094 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.769436] mdt00_000 6167 25059465.186680 224576178 120 0.000000 7493605.963955 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.780775] mdt_rdpg00_000 6179 25092560.438888 64309890 120 0.000000 1476072.627774 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.792112] mdt_out00_001 6196 25092548.517460 42725837 120 0.000000 992920.529530 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.803468] osp-pre-23-2 6489 25092560.368843 234941 120 0.000000 4011.671357 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.814816] ldlm_cb00_002 7093 21803373.534013 6700092 120 0.000000 128232.274464 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.826158] mdt00_012 8929 25092569.950659 227291722 120 0.000000 7599357.792316 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.837495] mdt00_015 8969 25060399.999027 250700710 120 0.000000 8622624.433612 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.848834] mdt00_023 9174 25092558.966902 226486864 120 0.000000 7962791.347060 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.860171] mdt00_025 9182 25090767.246081 228498295 120 0.000000 8303579.499702 0.000000 0 / [Wed Feb 6 09:40:52 2019][1696552.871515] mdt_io00_024 14901 25053188.756039 5507001 120 0.000000 221812.301202 0.000000 0 / [Wed Feb 6 09:40:53 2019][1696552.882858] ldlm_cn00_022 20039 25092572.417857 14993797 120 0.000000 606379.140740 0.000000 0 / [Wed Feb 6 09:40:53 2019][1696552.894211] mdt00_030 15169 25058588.196019 10454911 120 0.000000 1263066.906093 0.000000 0 / [Wed Feb 6 09:40:53 2019][1696552.905543] mdt00_031 15202 25059191.337630 2934766 120 0.000000 510233.404334 0.000000 0 / [Wed Feb 6 09:40:53 2019][1696552.916880] mdt00_035 15221 25092506.758490 9231612 120 0.000000 376245.797701 0.000000 0 / [Wed Feb 6 09:40:53 2019][1696552.928217] mdt00_037 15232 25058708.982905 13241576 120 0.000000 688766.244591 0.000000 0 / [Wed Feb 6 09:40:53 2019][1696552.939558] mdt00_048 15407 25058860.925251 14159104 120 0.000000 717466.598447 0.000000 0 / [Wed Feb 6 09:40:53 2019][1696552.950906] kworker/28:2 16073 24603626.815896 13 120 0.000000 0.027304 0.000000 0 / [Wed Feb 6 09:40:53 2019][1696552.962255] mdt_io00_045 27687 25090252.541120 2540 120 0.000000 797.893012 0.000000 0 / [Wed Feb 6 09:40:53 2019][1696552.973588] mdt_io00_050 27705 25053146.454746 1072 120 0.000000 40.727609 0.000000 0 / [Wed Feb 6 09:40:53 2019][1696552.984926] mdt_io00_059 27730 25053028.334525 325 120 0.000000 1096.772315 0.000000 0 / [Wed Feb 6 09:40:53 2019][1696552.996301] [Wed Feb 6 09:40:53 2019][1696552.997981] cpu#29, 1996.199 MHz [Wed Feb 6 09:40:53 2019][1696553.001399] .nr_running : 0 [Wed Feb 6 09:40:53 2019][1696553.006200] .load : 0 [Wed Feb 6 09:40:53 2019][1696553.011002] .nr_switches : 3291948084 [Wed Feb 6 09:40:53 2019][1696553.016583] .nr_load_updates : 375233057 [Wed Feb 6 09:40:53 2019][1696553.022079] .nr_uninterruptible : 100951 [Wed Feb 6 09:40:53 2019][1696553.027314] .next_balance : 5991.176908 [Wed Feb 6 09:40:53 2019][1696553.032980] .curr->pid : 0 [Wed Feb 6 09:40:53 2019][1696553.037782] .clock : 1696553036.798663 [Wed Feb 6 09:40:53 2019][1696553.043970] .cpu_load[0] : 0 [Wed Feb 6 09:40:53 2019][1696553.048772] .cpu_load[1] : 0 [Wed Feb 6 09:40:53 2019][1696553.053574] .cpu_load[2] : 0 [Wed Feb 6 09:40:53 2019][1696553.058374] .cpu_load[3] : 0 [Wed Feb 6 09:40:53 2019][1696553.063174] .cpu_load[4] : 0 [Wed Feb 6 09:40:53 2019][1696553.067978] .avg_idle : 1395972 [Wed Feb 6 09:40:53 2019][1696553.073301] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:53 2019][1696553.078620] [Wed Feb 6 09:40:53 2019][1696553.080294] cfs_rq[29]:/ [Wed Feb 6 09:40:53 2019][1696553.083006] .exec_clock : 0.000000 [Wed Feb 6 09:40:53 2019][1696553.088406] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:53 2019][1696553.093805] .min_vruntime : 17118766.020088 [Wed Feb 6 09:40:53 2019][1696553.099811] .max_vruntime : 0.000001 [Wed Feb 6 09:40:53 2019][1696553.105211] .spread : 0.000000 [Wed Feb 6 09:40:53 2019][1696553.110609] .spread0 : -11006859.822791 [Wed Feb 6 09:40:53 2019][1696553.116704] .nr_spread_over : 0 [Wed Feb 6 09:40:53 2019][1696553.121495] .nr_running : 0 [Wed Feb 6 09:40:53 2019][1696553.126289] .load : 0 [Wed Feb 6 09:40:53 2019][1696553.131082] .runnable_load_avg : 0 [Wed Feb 6 09:40:53 2019][1696553.135874] .blocked_load_avg : 3 [Wed Feb 6 09:40:53 2019][1696553.140667] .tg_load_avg : 0 [Wed Feb 6 09:40:53 2019][1696553.145459] .tg_load_contrib : 0 [Wed Feb 6 09:40:53 2019][1696553.150251] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:53 2019][1696553.155046] .tg->runnable_avg : 0 [Wed Feb 6 09:40:53 2019][1696553.159838] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:53 2019][1696553.164629] .throttled : 0 [Wed Feb 6 09:40:53 2019][1696553.169423] .throttle_count : 0 [Wed Feb 6 09:40:53 2019][1696553.174216] .avg->runnable_avg_sum : 117 [Wed Feb 6 09:40:53 2019][1696553.179183] .avg->runnable_avg_period : 48052 [Wed Feb 6 09:40:53 2019][1696553.184325] [Wed Feb 6 09:40:53 2019][1696553.185995] rt_rq[29]:/ [Wed Feb 6 09:40:53 2019][1696553.188620] .rt_nr_running : 0 [Wed Feb 6 09:40:53 2019][1696553.193414] .rt_throttled : 0 [Wed Feb 6 09:40:53 2019][1696553.198207] .rt_time : 0.000000 [Wed Feb 6 09:40:53 2019][1696553.203606] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:53 2019][1696553.209265] [Wed Feb 6 09:40:53 2019][1696553.210939] runnable tasks: [Wed Feb 6 09:40:53 2019][1696553.213911] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:53 2019][1696553.224856] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:53 2019][1696553.235736] watchdog/29 156 -11.985823 424103 0 0.000000 170712.328310 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.247069] migration/29 157 0.000000 6554 0 0.000000 327727.197996 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.258406] ksoftirqd/29 158 17118749.243318 7107800 120 0.000000 69288.116284 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.269741] kworker/29:0H 160 8513.827421 10 100 0.000000 0.035508 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.281113] polkitd 17508 17118698.730514 425458 120 0.000000 17021.310756 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.292463] kworker/29:1H 54700 17118749.224686 4688354 100 0.000000 18727.002291 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.303803] kiblnd_sd_01_03 5882 17118766.488173 2884495164 120 0.000000 21187896.510219 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.315243] jbd2/dm-4-8 6150 17118749.368545 33743382 120 0.000000 2834446.049929 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.326592] osp-pre-25-0 6279 17118747.320124 247808 120 0.000000 6310.950920 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.337934] osp-syn-46-2 6444 17112797.054649 435616 120 0.000000 12284.592400 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.349284] ldlm_cn01_004 6766 17010721.724396 27264472 120 0.000000 656403.002807 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.360627] mdt_io01_013 8951 16319736.810956 2249622 120 0.000000 85561.434738 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.371961] mdt_io01_019 8957 16319726.248837 6607894 120 0.000000 253965.403380 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.383295] mdt_io01_020 8958 16320210.992184 9830527 120 0.000000 375218.433257 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.394634] mdt01_019 8972 16918586.611095 102779778 120 0.000000 3959993.711766 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.405971] mdt01_023 8998 17118696.033575 107362655 120 0.000000 4947558.862259 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.417305] mdt01_027 9009 16921092.578672 100017078 120 0.000000 4228887.778720 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.428642] mdt01_034 9018 16830585.042733 104967351 120 0.000000 4014271.189995 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.439983] mdt_io01_026 9759 16798782.158474 5637597 120 0.000000 219991.386753 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.451319] mdt_io01_037 15179 17118637.805661 6078622 120 0.000000 229815.228256 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.462649] kworker/29:0 17671 17118750.955545 496612 120 0.000000 26295.050295 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.473991] mdt_io01_041 20862 16319726.341653 69991 120 0.000000 7207.107493 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.485323] mdt_io01_042 20863 16319736.437938 9662835 120 0.000000 355166.291495 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.496661] mdt_io01_046 20867 16319735.178420 480056 120 0.000000 28856.946588 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.507997] mdt_io01_053 20874 16798739.515292 3291574 120 0.000000 134163.763955 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.519337] ldlm_cn01_018 23321 17058667.613684 11944887 120 0.000000 321692.667279 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.530670] ldlm_cn01_022 28087 17058676.332238 16915776 120 0.000000 482664.069320 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.542007] ldlm_cn01_030 28331 17118724.883589 15537227 120 0.000000 438460.281174 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.553356] mdt01_063 15276 16831641.125367 9637442 120 0.000000 419809.882783 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.564700] mdt_io01_073 15441 16320207.558449 416 120 0.000000 17.989160 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.576030] mdt_io01_077 15447 17118750.180651 13306 120 0.000000 588.853024 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.587386] mdt_io01_088 27694 16319735.305594 422 120 0.000000 18.518124 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.598721] mdt_io01_091 27697 16319735.676179 411 120 0.000000 18.019851 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.610059] mdt_io01_094 27707 16798460.668221 34679 120 0.000000 1146.807408 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.621395] mdt_io01_099 27725 17118640.070964 9374 120 0.000000 354.372764 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.632747] systemd-logind 29049 17118699.207350 6195 120 0.000000 226.613159 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.644082] kworker/29:2 29130 16896035.797467 5 120 0.000000 0.051648 0.000000 1 / [Wed Feb 6 09:40:53 2019][1696553.655442] [Wed Feb 6 09:40:53 2019][1696553.657124] cpu#30, 1996.199 MHz [Wed Feb 6 09:40:53 2019][1696553.660540] .nr_running : 1 [Wed Feb 6 09:40:53 2019][1696553.665341] .load : 1024 [Wed Feb 6 09:40:53 2019][1696553.670405] .nr_switches : 917241386 [Wed Feb 6 09:40:53 2019][1696553.675900] .nr_load_updates : 252091501 [Wed Feb 6 09:40:53 2019][1696553.681393] .nr_uninterruptible : 123693 [Wed Feb 6 09:40:53 2019][1696553.686628] .next_balance : 5991.177587 [Wed Feb 6 09:40:53 2019][1696553.692298] .curr->pid : 15973 [Wed Feb 6 09:40:53 2019][1696553.697444] .clock : 1696553696.814623 [Wed Feb 6 09:40:53 2019][1696553.703632] .cpu_load[0] : 1024 [Wed Feb 6 09:40:53 2019][1696553.708690] .cpu_load[1] : 1024 [Wed Feb 6 09:40:53 2019][1696553.713748] .cpu_load[2] : 1024 [Wed Feb 6 09:40:53 2019][1696553.718810] .cpu_load[3] : 1024 [Wed Feb 6 09:40:53 2019][1696553.723870] .cpu_load[4] : 1024 [Wed Feb 6 09:40:53 2019][1696553.728932] .avg_idle : 10000000 [Wed Feb 6 09:40:53 2019][1696553.734340] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:53 2019][1696553.739661] [Wed Feb 6 09:40:53 2019][1696553.741334] cfs_rq[30]:/ [Wed Feb 6 09:40:53 2019][1696553.744047] .exec_clock : 0.000000 [Wed Feb 6 09:40:53 2019][1696553.749446] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:53 2019][1696553.754846] .min_vruntime : 10280265.548639 [Wed Feb 6 09:40:53 2019][1696553.760852] .max_vruntime : 0.000001 [Wed Feb 6 09:40:53 2019][1696553.766253] .spread : 0.000000 [Wed Feb 6 09:40:53 2019][1696553.771652] .spread0 : -17845362.687672 [Wed Feb 6 09:40:53 2019][1696553.777743] .nr_spread_over : 0 [Wed Feb 6 09:40:53 2019][1696553.782535] .nr_running : 1 [Wed Feb 6 09:40:53 2019][1696553.787329] .load : 1024 [Wed Feb 6 09:40:53 2019][1696553.792381] .runnable_load_avg : 1023 [Wed Feb 6 09:40:53 2019][1696553.797435] .blocked_load_avg : 0 [Wed Feb 6 09:40:53 2019][1696553.802227] .tg_load_avg : 0 [Wed Feb 6 09:40:53 2019][1696553.807021] .tg_load_contrib : 0 [Wed Feb 6 09:40:53 2019][1696553.811813] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:53 2019][1696553.816607] .tg->runnable_avg : 0 [Wed Feb 6 09:40:53 2019][1696553.821398] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:53 2019][1696553.826192] .throttled : 0 [Wed Feb 6 09:40:53 2019][1696553.830985] .throttle_count : 0 [Wed Feb 6 09:40:53 2019][1696553.835777] .avg->runnable_avg_sum : 46753 [Wed Feb 6 09:40:53 2019][1696553.840917] .avg->runnable_avg_period : 46553 [Wed Feb 6 09:40:53 2019][1696553.846060] [Wed Feb 6 09:40:53 2019][1696553.847727] rt_rq[30]:/ [Wed Feb 6 09:40:53 2019][1696553.850354] .rt_nr_running : 0 [Wed Feb 6 09:40:53 2019][1696553.855149] .rt_throttled : 0 [Wed Feb 6 09:40:53 2019][1696553.859941] .rt_time : 0.000000 [Wed Feb 6 09:40:53 2019][1696553.865339] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:53 2019][1696553.870998] [Wed Feb 6 09:40:53 2019][1696553.872670] runnable tasks: [Wed Feb 6 09:40:53 2019][1696553.875645] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:54 2019][1696553.886590] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:54 2019][1696553.897469] watchdog/30 161 -11.984901 424098 0 0.000000 157298.143803 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696553.908803] migration/30 162 0.000000 2274 0 0.000000 480609.314889 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696553.920140] ksoftirqd/30 163 10209509.730269 7743617 120 0.000000 82509.074977 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696553.931476] kworker/30:0H 165 5246.347268 10 100 0.000000 0.034255 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696553.942832] kworker/30:1 7273 10209001.579699 503823 120 0.000000 21269.216722 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696553.954175] ccp-4-q0 10650 2752.180679 2 120 0.000000 0.008245 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696553.965533] kworker/30:1H 52904 10208948.666197 5576367 100 0.000000 22481.081706 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696553.976904] osp-syn-21-0 6290 10208748.075022 786347 120 0.000000 20803.002252 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696553.988267] mdt_rdpg02_003 7111 9558009.145184 7869680 120 0.000000 345532.592499 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696553.999657] mdt_io02_028 15442 10069393.592782 398428 120 0.000000 17006.704709 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696554.010996] mdt_io02_038 15456 9562178.223949 53592 120 0.000000 2128.311776 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696554.022333] R ldlm_bl_16 15973 10280540.555831 2326967 120 0.000000 5388806.878666 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696554.033670] kworker/u264:1 16127 10081911.081052 684 100 0.000000 513.308721 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696554.045026] mdt_io02_066 27749 9738234.675795 359 120 0.000000 3133.797632 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696554.056361] mdt_io02_070 27758 9562178.296257 176 120 0.000000 8.029817 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696554.067703] mdt_io02_072 27763 9562178.208841 171 120 0.000000 7.729901 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696554.079055] kworker/30:0 29820 10042830.836222 5 120 0.000000 0.027484 0.000000 2 / [Wed Feb 6 09:40:54 2019][1696554.090415] [Wed Feb 6 09:40:54 2019][1696554.092089] cpu#31, 1996.199 MHz [Wed Feb 6 09:40:54 2019][1696554.095507] .nr_running : 1 [Wed Feb 6 09:40:54 2019][1696554.100310] .load : 1024 [Wed Feb 6 09:40:54 2019][1696554.105371] .nr_switches : 424262399 [Wed Feb 6 09:40:54 2019][1696554.110865] .nr_load_updates : 162378918 [Wed Feb 6 09:40:54 2019][1696554.116358] .nr_uninterruptible : 30389 [Wed Feb 6 09:40:54 2019][1696554.121508] .next_balance : 5991.178030 [Wed Feb 6 09:40:54 2019][1696554.127175] .curr->pid : 15968 [Wed Feb 6 09:40:54 2019][1696554.132325] .clock : 1696554131.825862 [Wed Feb 6 09:40:54 2019][1696554.138508] .cpu_load[0] : 1024 [Wed Feb 6 09:40:54 2019][1696554.143560] .cpu_load[1] : 1024 [Wed Feb 6 09:40:54 2019][1696554.148612] .cpu_load[2] : 1024 [Wed Feb 6 09:40:54 2019][1696554.153667] .cpu_load[3] : 1024 [Wed Feb 6 09:40:54 2019][1696554.158729] .cpu_load[4] : 1024 [Wed Feb 6 09:40:54 2019][1696554.163794] .avg_idle : 10000000 [Wed Feb 6 09:40:54 2019][1696554.169202] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:54 2019][1696554.174522] [Wed Feb 6 09:40:54 2019][1696554.176194] cfs_rq[31]:/ [Wed Feb 6 09:40:54 2019][1696554.178908] .exec_clock : 0.000000 [Wed Feb 6 09:40:54 2019][1696554.184308] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:54 2019][1696554.189708] .min_vruntime : 9869802.518817 [Wed Feb 6 09:40:54 2019][1696554.195625] .max_vruntime : 0.000001 [Wed Feb 6 09:40:54 2019][1696554.201026] .spread : 0.000000 [Wed Feb 6 09:40:54 2019][1696554.206425] .spread0 : -18255827.523608 [Wed Feb 6 09:40:54 2019][1696554.212517] .nr_spread_over : 0 [Wed Feb 6 09:40:54 2019][1696554.217310] .nr_running : 1 [Wed Feb 6 09:40:54 2019][1696554.222103] .load : 1024 [Wed Feb 6 09:40:54 2019][1696554.227156] .runnable_load_avg : 1023 [Wed Feb 6 09:40:54 2019][1696554.232210] .blocked_load_avg : 0 [Wed Feb 6 09:40:54 2019][1696554.237002] .tg_load_avg : 0 [Wed Feb 6 09:40:54 2019][1696554.241794] .tg_load_contrib : 0 [Wed Feb 6 09:40:54 2019][1696554.246587] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:54 2019][1696554.251380] .tg->runnable_avg : 0 [Wed Feb 6 09:40:54 2019][1696554.256173] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:54 2019][1696554.260966] .throttled : 0 [Wed Feb 6 09:40:54 2019][1696554.265759] .throttle_count : 0 [Wed Feb 6 09:40:54 2019][1696554.270551] .avg->runnable_avg_sum : 46557 [Wed Feb 6 09:40:54 2019][1696554.275691] .avg->runnable_avg_period : 46387 [Wed Feb 6 09:40:54 2019][1696554.280834] [Wed Feb 6 09:40:54 2019][1696554.282502] rt_rq[31]:/ [Wed Feb 6 09:40:54 2019][1696554.285128] .rt_nr_running : 0 [Wed Feb 6 09:40:54 2019][1696554.289922] .rt_throttled : 0 [Wed Feb 6 09:40:54 2019][1696554.294715] .rt_time : 0.000000 [Wed Feb 6 09:40:54 2019][1696554.300115] .rt_runtime : 958.767362 [Wed Feb 6 09:40:54 2019][1696554.305686] [Wed Feb 6 09:40:54 2019][1696554.307359] runnable tasks: [Wed Feb 6 09:40:54 2019][1696554.310332] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:54 2019][1696554.321279] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:54 2019][1696554.332157] watchdog/31 166 -11.986374 424093 0 0.000000 115070.229177 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.343490] migration/31 167 0.000000 2854 0 0.000000 453464.841738 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.354827] ksoftirqd/31 168 9715428.791332 1459814 120 0.000000 17417.753412 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.366163] kworker/31:0H 170 4088.619703 9 100 0.000000 0.036860 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.377521] kworker/31:1 7275 9731352.388112 220516 120 0.000000 8625.200111 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.388860] ccp-2-q1 10530 2009.793957 2 120 0.000000 0.020299 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.400206] multipathd 17011 0.000000 1 0 0.000000 0.611322 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.411543] multipathd 17151 0.000000 1 0 0.000000 0.588418 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.422881] multipathd 17266 0.000000 1 0 0.000000 0.874833 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.434217] multipathd 18450 0.000000 1 0 0.000000 0.649615 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.445552] multipathd 18506 0.000000 1 0 0.000000 0.832632 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.456889] kworker/31:1H 16786 9695591.060877 1172935 100 0.000000 4009.359874 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.468233] kdmflush 18501 4314.215311 2 100 0.000000 0.014337 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.479569] bioset 18504 4314.220451 2 100 0.000000 0.013545 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.490936] ldlm_cb03_000 6027 5987109.976619 842391 120 0.000000 13087.324074 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.502345] ldlm_cb03_003 19917 5987109.973865 3588329 120 0.000000 56768.381535 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.513723] R ldlm_bl_11 15968 9870133.527348 2326020 120 0.000000 5191386.577718 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.525061] kworker/31:0 18035 6098944.913000 2 120 0.000000 0.011052 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.536425] ll_mgs_0004 28952 7684655.367939 124 120 0.000000 1.320951 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.547763] ll_mgs_0009 28966 7684658.994215 305 120 0.000000 15.380623 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.559099] ll_mgs_0016 28984 7684660.062002 82 120 0.000000 1038.686848 0.000000 3 / [Wed Feb 6 09:40:54 2019][1696554.570461] [Wed Feb 6 09:40:54 2019][1696554.572139] cpu#32, 1996.199 MHz [Wed Feb 6 09:40:54 2019][1696554.575558] .nr_running : 0 [Wed Feb 6 09:40:54 2019][1696554.580354] .load : 0 [Wed Feb 6 09:40:54 2019][1696554.585151] .nr_switches : 4315719752 [Wed Feb 6 09:40:54 2019][1696554.590727] .nr_load_updates : 442413111 [Wed Feb 6 09:40:54 2019][1696554.596215] .nr_uninterruptible : -481400 [Wed Feb 6 09:40:54 2019][1696554.601541] .next_balance : 5991.178483 [Wed Feb 6 09:40:54 2019][1696554.607207] .curr->pid : 0 [Wed Feb 6 09:40:54 2019][1696554.612007] .clock : 1696554611.837434 [Wed Feb 6 09:40:54 2019][1696554.618194] .cpu_load[0] : 0 [Wed Feb 6 09:40:54 2019][1696554.622995] .cpu_load[1] : 0 [Wed Feb 6 09:40:54 2019][1696554.627793] .cpu_load[2] : 0 [Wed Feb 6 09:40:54 2019][1696554.632591] .cpu_load[3] : 0 [Wed Feb 6 09:40:54 2019][1696554.637396] .cpu_load[4] : 0 [Wed Feb 6 09:40:54 2019][1696554.642197] .avg_idle : 2461895 [Wed Feb 6 09:40:54 2019][1696554.647518] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:54 2019][1696554.652844] [Wed Feb 6 09:40:54 2019][1696554.654522] cfs_rq[32]:/ [Wed Feb 6 09:40:54 2019][1696554.657234] .exec_clock : 0.000000 [Wed Feb 6 09:40:54 2019][1696554.662633] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:54 2019][1696554.668032] .min_vruntime : 26790255.934151 [Wed Feb 6 09:40:54 2019][1696554.674038] .max_vruntime : 0.000001 [Wed Feb 6 09:40:54 2019][1696554.679438] .spread : 0.000000 [Wed Feb 6 09:40:54 2019][1696554.684838] .spread0 : -1335376.559306 [Wed Feb 6 09:40:54 2019][1696554.690845] .nr_spread_over : 0 [Wed Feb 6 09:40:54 2019][1696554.695637] .nr_running : 0 [Wed Feb 6 09:40:54 2019][1696554.700428] .load : 0 [Wed Feb 6 09:40:54 2019][1696554.705222] .runnable_load_avg : 0 [Wed Feb 6 09:40:54 2019][1696554.710015] .blocked_load_avg : 1 [Wed Feb 6 09:40:54 2019][1696554.714807] .tg_load_avg : 0 [Wed Feb 6 09:40:54 2019][1696554.719599] .tg_load_contrib : 0 [Wed Feb 6 09:40:54 2019][1696554.724394] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:54 2019][1696554.729186] .tg->runnable_avg : 0 [Wed Feb 6 09:40:54 2019][1696554.733978] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:54 2019][1696554.738772] .throttled : 0 [Wed Feb 6 09:40:54 2019][1696554.743565] .throttle_count : 0 [Wed Feb 6 09:40:54 2019][1696554.748356] .avg->runnable_avg_sum : 109 [Wed Feb 6 09:40:54 2019][1696554.753324] .avg->runnable_avg_period : 47704 [Wed Feb 6 09:40:54 2019][1696554.758465] [Wed Feb 6 09:40:54 2019][1696554.760136] rt_rq[32]:/ [Wed Feb 6 09:40:54 2019][1696554.762761] .rt_nr_running : 0 [Wed Feb 6 09:40:54 2019][1696554.767554] .rt_throttled : 0 [Wed Feb 6 09:40:54 2019][1696554.772347] .rt_time : 0.000000 [Wed Feb 6 09:40:54 2019][1696554.777746] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:54 2019][1696554.783405] [Wed Feb 6 09:40:54 2019][1696554.785079] runnable tasks: [Wed Feb 6 09:40:54 2019][1696554.788050] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:54 2019][1696554.798998] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:54 2019][1696554.809877] watchdog/32 171 -11.986895 424104 0 0.000000 197471.402177 0.000000 0 / [Wed Feb 6 09:40:54 2019][1696554.821209] migration/32 172 0.000000 38435 0 0.000000 440282.221738 0.000000 0 / [Wed Feb 6 09:40:54 2019][1696554.832547] ksoftirqd/32 173 26790102.265969 8203036 120 0.000000 86417.105850 0.000000 0 / [Wed Feb 6 09:40:54 2019][1696554.843884] kworker/32:0H 175 8474.306153 9 100 0.000000 0.025365 0.000000 0 / [Wed Feb 6 09:40:54 2019][1696554.855241] ext4-rsv-conver 7802 1847.674584 2 100 0.000000 0.008426 0.000000 0 / [Wed Feb 6 09:40:54 2019][1696554.866583] kworker/32:1H 17123 26790205.600523 9405307 100 0.000000 43340.380453 0.000000 0 / [Wed Feb 6 09:40:54 2019][1696554.877918] rpciod 17477 9512.956274 2 100 0.000000 0.011852 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696554.889254] xprtiod 17478 9524.964950 2 100 0.000000 0.009198 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696554.900593] mlx4 17701 9837.047920 2 100 0.000000 0.011932 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696554.911927] mlx4_ib 17716 9879.193971 2 100 0.000000 0.010560 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696554.923262] mlx4_ib_mcg 17717 9891.202514 2 100 0.000000 0.008966 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696554.934598] ipoib_flush 17808 10079.130299 2 100 0.000000 0.011802 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696554.945936] mlx5i 17809 10091.138866 2 100 0.000000 0.009217 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696554.957273] ipoib_wq 17813 10103.148388 2 100 0.000000 0.010400 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696554.968608] rdma_cm 18185 10237.824165 2 100 0.000000 0.011281 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696554.979964] kiblnd_sd_00_03 5878 26790256.696531 4155040228 120 0.000000 27443361.047934 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696554.991399] ldiskfs-rsv-con 6011 2229899.523991 2 100 0.000000 0.009669 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.002731] ldlm_cn00_002 6031 26774923.523598 13424746 120 0.000000 498649.511266 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.014068] mgs_params_noti 6044 2229976.075603 2 118 0.000000 0.015970 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.025404] ldiskfs-rsv-con 6152 2230295.519715 2 100 0.000000 0.012033 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.036741] mdt_attr00_000 6187 2230575.180853 2 120 0.000000 0.063501 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.048074] mdt_attr00_001 6188 2230587.213740 2 120 0.000000 0.032893 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.059411] mdt_out00_000 6195 26028816.932340 37114707 120 0.000000 872728.839265 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.070748] mdt_seqs_0000 6203 2230767.443067 2 120 0.000000 0.051248 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.082084] mdt_seqs_0001 6204 2230779.460203 2 120 0.000000 0.017142 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.093421] mdt_io00_002 6211 26032088.916715 5862866 120 0.000000 226372.609109 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.104759] qmt_reba_fir-QM 6232 2231046.314572 2 120 0.000000 0.024046 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.116095] osp-syn-9-0 6268 26779553.135728 774126 120 0.000000 20696.373759 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.127437] osp_up0-2 6432 2232029.235987 2 120 0.000000 0.020480 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.138785] osp-syn-30-2 6530 26790105.954343 434905 120 0.000000 12229.245161 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.150119] dist_txn-2 6596 22143025.902544 11 120 0.000000 0.381534 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.161458] mdt00_006 6690 26790108.480978 206492306 120 0.000000 7782458.817365 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.172794] ldlm_cn00_003 6762 26774924.012298 12723665 120 0.000000 496066.921642 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.184166] mdt_io00_026 20198 26032131.159137 2399313 120 0.000000 100116.417263 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.195523] mdt00_036 15228 26716695.005136 17488755 120 0.000000 4840295.051822 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.206867] mdt00_047 15406 26774341.206482 5501328 120 0.000000 997737.937712 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.218198] mdt00_050 15409 26046935.532406 7587357 120 0.000000 769507.732266 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.229538] mdt_io00_034 15465 26032088.859335 332696 120 0.000000 63622.611464 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.240883] kworker/32:2 26624 26790231.711094 8955 120 0.000000 7269.514749 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.252218] mdt_io00_044 27681 26032088.861570 251 120 0.000000 8.638471 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.263558] mdt_io00_058 27729 25506472.010071 187 120 0.000000 7.447062 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.274897] mdt_io00_061 27733 26032088.896757 380 120 0.000000 15.596441 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.286247] ll_mgs_0025 28993 26067715.705815 2914 120 0.000000 165.667001 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.297590] kworker/32:1 29908 26716616.497392 14 120 0.000000 0.108958 0.000000 0 / [Wed Feb 6 09:40:55 2019][1696555.308948] [Wed Feb 6 09:40:55 2019][1696555.310628] cpu#33, 1996.199 MHz [Wed Feb 6 09:40:55 2019][1696555.314045] .nr_running : 0 [Wed Feb 6 09:40:55 2019][1696555.318845] .load : 0 [Wed Feb 6 09:40:55 2019][1696555.323647] .nr_switches : 2960807817 [Wed Feb 6 09:40:55 2019][1696555.329230] .nr_load_updates : 364944689 [Wed Feb 6 09:40:55 2019][1696555.334722] .nr_uninterruptible : 156102 [Wed Feb 6 09:40:55 2019][1696555.339960] .next_balance : 5991.179225 [Wed Feb 6 09:40:55 2019][1696555.345624] .curr->pid : 0 [Wed Feb 6 09:40:55 2019][1696555.350427] .clock : 1696555349.859976 [Wed Feb 6 09:40:55 2019][1696555.356614] .cpu_load[0] : 0 [Wed Feb 6 09:40:55 2019][1696555.361413] .cpu_load[1] : 0 [Wed Feb 6 09:40:55 2019][1696555.366211] .cpu_load[2] : 0 [Wed Feb 6 09:40:55 2019][1696555.371011] .cpu_load[3] : 0 [Wed Feb 6 09:40:55 2019][1696555.375813] .cpu_load[4] : 0 [Wed Feb 6 09:40:55 2019][1696555.380617] .avg_idle : 9479095 [Wed Feb 6 09:40:55 2019][1696555.385938] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:55 2019][1696555.391257] [Wed Feb 6 09:40:55 2019][1696555.392932] cfs_rq[33]:/ [Wed Feb 6 09:40:55 2019][1696555.395644] .exec_clock : 0.000000 [Wed Feb 6 09:40:55 2019][1696555.401043] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:55 2019][1696555.406443] .min_vruntime : 20768870.804143 [Wed Feb 6 09:40:55 2019][1696555.412447] .max_vruntime : 0.000001 [Wed Feb 6 09:40:55 2019][1696555.417847] .spread : 0.000000 [Wed Feb 6 09:40:55 2019][1696555.423248] .spread0 : -7356764.583383 [Wed Feb 6 09:40:55 2019][1696555.429253] .nr_spread_over : 0 [Wed Feb 6 09:40:55 2019][1696555.434046] .nr_running : 0 [Wed Feb 6 09:40:55 2019][1696555.438838] .load : 0 [Wed Feb 6 09:40:55 2019][1696555.443632] .runnable_load_avg : 0 [Wed Feb 6 09:40:55 2019][1696555.448424] .blocked_load_avg : 0 [Wed Feb 6 09:40:55 2019][1696555.453218] .tg_load_avg : 0 [Wed Feb 6 09:40:55 2019][1696555.458010] .tg_load_contrib : 0 [Wed Feb 6 09:40:55 2019][1696555.462802] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:55 2019][1696555.467597] .tg->runnable_avg : 0 [Wed Feb 6 09:40:55 2019][1696555.472388] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:55 2019][1696555.477182] .throttled : 0 [Wed Feb 6 09:40:55 2019][1696555.481975] .throttle_count : 0 [Wed Feb 6 09:40:55 2019][1696555.486766] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:40:55 2019][1696555.491559] .avg->runnable_avg_period : 47019 [Wed Feb 6 09:40:55 2019][1696555.496703] [Wed Feb 6 09:40:55 2019][1696555.498371] rt_rq[33]:/ [Wed Feb 6 09:40:55 2019][1696555.500998] .rt_nr_running : 0 [Wed Feb 6 09:40:55 2019][1696555.505792] .rt_throttled : 0 [Wed Feb 6 09:40:55 2019][1696555.510583] .rt_time : 0.000000 [Wed Feb 6 09:40:55 2019][1696555.515984] .rt_runtime : 974.501132 [Wed Feb 6 09:40:55 2019][1696555.521554] [Wed Feb 6 09:40:55 2019][1696555.523228] runnable tasks: [Wed Feb 6 09:40:55 2019][1696555.526202] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:55 2019][1696555.537147] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:55 2019][1696555.548028] watchdog/33 176 -11.986083 424096 0 0.000000 154124.310199 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.559361] migration/33 177 0.000000 37019 0 0.000000 421436.175978 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.570698] ksoftirqd/33 178 20768724.887836 9216278 120 0.000000 87613.509438 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.582033] kworker/33:0H 180 34732.035215 10 100 0.000000 0.038573 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.593380] netns 256 47.367919 2 100 0.000000 0.010591 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.604714] kintegrityd 259 83.394073 2 100 0.000000 0.008606 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.616051] bioset 260 95.402490 2 100 0.000000 0.009008 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.627386] bioset 261 107.410265 2 100 0.000000 0.008356 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.638723] bioset 262 119.418359 2 100 0.000000 0.008676 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.650060] kblockd 263 131.426685 2 100 0.000000 0.008887 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.661396] md 264 143.434921 2 100 0.000000 0.008917 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.672731] watchdogd 266 167.451454 2 100 0.000000 0.008797 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.684068] irq/27-AMD-Vi 269 0.000000 2 49 0.000000 0.018114 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.695406] kswapd1 277 18637421.670141 7 120 0.000000 0.594362 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.706742] ksmd 280 335.718207 2 125 0.000000 0.011121 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.718076] crypto 282 359.814604 2 100 0.000000 0.009909 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.729413] kthrotld 290 456.029050 2 100 0.000000 0.009428 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.740750] kmpath_rdacd 295 516.078273 2 100 0.000000 0.018655 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.752087] kaluad 296 528.096059 2 100 0.000000 0.018496 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.763424] ipv6_addrconf 302 564.126883 2 100 0.000000 0.012554 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.774757] deferwq 315 624.271507 2 100 0.000000 0.013196 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.786097] kworker/33:1 321 20768858.796959 723330 120 0.000000 18187.123966 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.797436] mlx5_page_alloc 7394 3046.343860 2 100 0.000000 0.010891 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.808769] kworker/u263:0 7441 3188.966824 4 100 0.000000 0.015299 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.820105] mlx5_fc 7445 3162.585253 2 100 0.000000 0.014287 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.831441] ib-comp-wq 7466 3200.966796 2 100 0.000000 0.010039 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.842777] ib_nl_sa_wq 7469 3224.983970 2 100 0.000000 0.009688 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.854113] mlx5_ib_sigerr_ 7477 3236.993454 2 100 0.000000 0.010480 0.000000 1 / [Wed Feb 6 09:40:55 2019][1696555.865448] to_fifo 7483 3261.013195 2 100 0.000000 0.010089 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696555.876785] mkey_cache 7486 3273.022685 2 100 0.000000 0.010319 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696555.888129] kvm-irqfd-clean 13585 5916.136356 2 100 0.000000 0.028704 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696555.899474] dsm_sa_eventmgr 18952 20768836.529367 274498 120 0.000000 11136.983216 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696555.910819] nfsv4.1-svc 19229 16453.819499 2 120 0.000000 0.011041 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696555.922158] kworker/33:1H 27717 20768724.869684 6008819 100 0.000000 22914.128807 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696555.933498] clio 5829 2131213.203917 2 100 0.000000 0.010309 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696555.944850] ldlm_cn01_001 6033 20760492.593107 37670871 120 0.000000 989359.580171 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696555.956188] ldiskfs-rsv-con 6153 2132080.506375 2 100 0.000000 0.011533 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696555.967520] mdt_rdpg01_001 6182 20768855.857173 16457661 120 0.000000 714248.749339 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696555.978866] osp-syn-6-0 6310 20768756.384434 790277 120 0.000000 21064.016421 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696555.990204] osp-syn-12-2 6438 20768497.761101 442247 120 0.000000 12302.889412 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696556.001537] osp-syn-14-2 6450 20768770.087104 443236 120 0.000000 12931.282748 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696556.012875] osp-syn-25-2 6468 20768769.136502 442250 120 0.000000 12553.648992 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696556.024214] osp-pre-35-2 6527 20768857.455714 235005 120 0.000000 3455.575486 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696556.035558] mdt01_008 6952 20670870.154330 116791024 120 0.000000 4866229.999362 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696556.046895] ldlm_cn01_007 8616 20760502.533027 33727970 120 0.000000 935781.534240 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696556.058230] mdt_io01_018 8956 20768725.008865 8385478 120 0.000000 313030.342412 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696556.069568] mdt01_025 9005 20768858.791338 111471976 120 0.000000 4596099.605483 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696556.080902] ldlm_cn01_009 9152 20764981.463165 36245893 120 0.000000 950075.640204 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696556.092259] mdt_rdpg01_010 24682 20768860.837195 10932543 120 0.000000 442263.670787 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696556.103645] mdt_io01_103 27745 20768846.529958 43106 120 0.000000 1510.483159 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696556.114992] ll_mgs_0017 28985 20634541.566830 103 120 0.000000 2206.679629 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696556.126323] ll_mgs_0021 28989 20634540.634978 191 120 0.000000 1931.224809 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696556.137664] kworker/33:2 30273 20766693.528623 22 120 0.000000 0.171065 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696556.148997] l_getidentity 30463 20768752.184314 2 120 0.000000 0.909919 0.000000 1 / [Wed Feb 6 09:40:56 2019][1696556.160349] [Wed Feb 6 09:40:56 2019][1696556.162028] cpu#34, 1996.199 MHz [Wed Feb 6 09:40:56 2019][1696556.165447] .nr_running : 0 [Wed Feb 6 09:40:56 2019][1696556.170246] .load : 1024 [Wed Feb 6 09:40:56 2019][1696556.175310] .nr_switches : 788299663 [Wed Feb 6 09:40:56 2019][1696556.180801] .nr_load_updates : 299457474 [Wed Feb 6 09:40:56 2019][1696556.186298] .nr_uninterruptible : 109587 [Wed Feb 6 09:40:56 2019][1696556.191531] .next_balance : 5991.180074 [Wed Feb 6 09:40:56 2019][1696556.197197] .curr->pid : 0 [Wed Feb 6 09:40:56 2019][1696556.202001] .clock : 1696556201.880841 [Wed Feb 6 09:40:56 2019][1696556.208187] .cpu_load[0] : 0 [Wed Feb 6 09:40:56 2019][1696556.212993] .cpu_load[1] : 0 [Wed Feb 6 09:40:56 2019][1696556.217788] .cpu_load[2] : 0 [Wed Feb 6 09:40:56 2019][1696556.222587] .cpu_load[3] : 0 [Wed Feb 6 09:40:56 2019][1696556.227386] .cpu_load[4] : 0 [Wed Feb 6 09:40:56 2019][1696556.232188] .avg_idle : 3162554 [Wed Feb 6 09:40:56 2019][1696556.237506] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:56 2019][1696556.242819] [Wed Feb 6 09:40:56 2019][1696556.244486] cfs_rq[34]:/ [Wed Feb 6 09:40:56 2019][1696556.247198] .exec_clock : 0.000000 [Wed Feb 6 09:40:56 2019][1696556.252601] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:56 2019][1696556.257999] .min_vruntime : 16903088.693217 [Wed Feb 6 09:40:56 2019][1696556.264006] .max_vruntime : 0.000001 [Wed Feb 6 09:40:56 2019][1696556.269403] .spread : 0.000000 [Wed Feb 6 09:40:56 2019][1696556.274802] .spread0 : -11222549.482426 [Wed Feb 6 09:40:56 2019][1696556.280897] .nr_spread_over : 0 [Wed Feb 6 09:40:56 2019][1696556.285688] .nr_running : 0 [Wed Feb 6 09:40:56 2019][1696556.290482] .load : 0 [Wed Feb 6 09:40:56 2019][1696556.295276] .runnable_load_avg : 0 [Wed Feb 6 09:40:56 2019][1696556.300068] .blocked_load_avg : 11 [Wed Feb 6 09:40:56 2019][1696556.304948] .tg_load_avg : 0 [Wed Feb 6 09:40:56 2019][1696556.309738] .tg_load_contrib : 0 [Wed Feb 6 09:40:56 2019][1696556.314532] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:56 2019][1696556.319324] .tg->runnable_avg : 0 [Wed Feb 6 09:40:56 2019][1696556.324118] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:56 2019][1696556.328910] .throttled : 0 [Wed Feb 6 09:40:56 2019][1696556.333703] .throttle_count : 0 [Wed Feb 6 09:40:56 2019][1696556.338495] .avg->runnable_avg_sum : 657 [Wed Feb 6 09:40:56 2019][1696556.343463] .avg->runnable_avg_period : 48094 [Wed Feb 6 09:40:56 2019][1696556.348605] [Wed Feb 6 09:40:56 2019][1696556.350276] rt_rq[34]:/ [Wed Feb 6 09:40:56 2019][1696556.352901] .rt_nr_running : 0 [Wed Feb 6 09:40:56 2019][1696556.357694] .rt_throttled : 0 [Wed Feb 6 09:40:56 2019][1696556.362486] .rt_time : 0.006833 [Wed Feb 6 09:40:56 2019][1696556.367885] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:56 2019][1696556.373544] [Wed Feb 6 09:40:56 2019][1696556.375217] runnable tasks: [Wed Feb 6 09:40:56 2019][1696556.378190] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:56 2019][1696556.389137] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:56 2019][1696556.400019] watchdog/34 181 -11.986805 424073 0 0.000000 103850.323286 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.411349] migration/34 182 0.000000 16917 0 0.000000 583468.086567 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.422686] ksoftirqd/34 183 16903071.203866 6387278 120 0.000000 78893.201575 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.434023] kworker/34:0H 185 24661.660338 10 100 0.000000 0.039174 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.445373] kpsmoused 299 35.001511 2 100 0.000000 0.019027 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.456709] ib_mcast 7468 1881.300840 2 100 0.000000 0.009920 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.468054] gssproxy 17573 9645.363732 1 120 0.000000 0.018174 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.479388] dsm_sa_datamgrd 18866 12385.540102 1 120 0.000000 0.034676 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.490722] dsm_sa_datamgrd 18870 12421.571141 1 120 0.000000 0.009367 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.502060] writer#4 19097 16903035.984193 18620238 120 0.000000 194997.015091 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.513393] nfsiod 19163 13932.399112 2 100 0.000000 0.011080 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.524732] kworker/34:1H 27791 16902932.140568 4232710 100 0.000000 15438.860095 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.536071] kworker/34:2 95825 16903062.422882 643929 120 0.000000 9741.255069 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.547418] ptlrpcd_02_01 5921 16903071.125777 13214921 120 0.000000 183868.365020 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.558773] mdt_io02_001 6216 16652407.482575 4856740 120 0.000000 187938.265643 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.570116] osp-syn-41-0 6294 16902388.450268 779868 120 0.000000 20927.026753 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.581448] osp-syn-40-0 6306 16902805.928184 780292 120 0.000000 21286.221779 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.592783] osp-pre-6-0 6311 16903063.767025 247827 120 0.000000 5889.349524 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.604133] osp-pre-36-2 6523 16903071.215698 235053 120 0.000000 3481.816350 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.615475] mdt_io02_005 6868 16652407.515807 4153128 120 0.000000 165185.633023 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.626829] ldlm_cn02_010 9990 16845095.346543 24350570 120 0.000000 741445.501919 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.638194] mdt02_031 15183 16903016.349960 12844766 120 0.000000 709440.397757 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.649524] mdt02_032 15184 16664670.164210 1107479 120 0.000000 1748481.486092 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.660874] mdt02_049 15511 16903093.537777 9227877 120 0.000000 1046549.378454 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.672233] mdt_io02_069 27756 16655055.508625 178 120 0.000000 1826.508407 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.683580] ll_mgs_0020 28988 16662341.973912 195 120 0.000000 28.984530 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.694920] kworker/34:0 30225 16845236.278306 10 120 0.000000 0.069274 0.000000 2 / [Wed Feb 6 09:40:56 2019][1696556.706279] [Wed Feb 6 09:40:56 2019][1696556.707955] cpu#35, 1996.199 MHz [Wed Feb 6 09:40:56 2019][1696556.711371] .nr_running : 0 [Wed Feb 6 09:40:56 2019][1696556.716173] .load : 0 [Wed Feb 6 09:40:56 2019][1696556.720970] .nr_switches : 474691369 [Wed Feb 6 09:40:56 2019][1696556.726460] .nr_load_updates : 162498655 [Wed Feb 6 09:40:56 2019][1696556.731956] .nr_uninterruptible : 24621 [Wed Feb 6 09:40:56 2019][1696556.737104] .next_balance : 5991.180622 [Wed Feb 6 09:40:56 2019][1696556.742771] .curr->pid : 0 [Wed Feb 6 09:40:56 2019][1696556.747573] .clock : 1696556746.894159 [Wed Feb 6 09:40:56 2019][1696556.753760] .cpu_load[0] : 0 [Wed Feb 6 09:40:56 2019][1696556.758562] .cpu_load[1] : 0 [Wed Feb 6 09:40:56 2019][1696556.763364] .cpu_load[2] : 0 [Wed Feb 6 09:40:56 2019][1696556.768162] .cpu_load[3] : 0 [Wed Feb 6 09:40:56 2019][1696556.772959] .cpu_load[4] : 0 [Wed Feb 6 09:40:56 2019][1696556.777759] .avg_idle : 8757083 [Wed Feb 6 09:40:56 2019][1696556.783078] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:56 2019][1696556.788398] [Wed Feb 6 09:40:56 2019][1696556.790069] cfs_rq[35]:/ [Wed Feb 6 09:40:56 2019][1696556.792780] .exec_clock : 0.000000 [Wed Feb 6 09:40:56 2019][1696556.798179] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:56 2019][1696556.803580] .min_vruntime : 12194756.476341 [Wed Feb 6 09:40:56 2019][1696556.809584] .max_vruntime : 0.000001 [Wed Feb 6 09:40:56 2019][1696556.814986] .spread : 0.000000 [Wed Feb 6 09:40:56 2019][1696556.820385] .spread0 : -15930887.185756 [Wed Feb 6 09:40:56 2019][1696556.826477] .nr_spread_over : 0 [Wed Feb 6 09:40:56 2019][1696556.831270] .nr_running : 0 [Wed Feb 6 09:40:56 2019][1696556.836062] .load : 0 [Wed Feb 6 09:40:56 2019][1696556.840854] .runnable_load_avg : 0 [Wed Feb 6 09:40:56 2019][1696556.845648] .blocked_load_avg : 0 [Wed Feb 6 09:40:56 2019][1696556.850442] .tg_load_avg : 0 [Wed Feb 6 09:40:56 2019][1696556.855234] .tg_load_contrib : 0 [Wed Feb 6 09:40:56 2019][1696556.860027] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:56 2019][1696556.864818] .tg->runnable_avg : 0 [Wed Feb 6 09:40:56 2019][1696556.869612] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:57 2019][1696556.874405] .throttled : 0 [Wed Feb 6 09:40:57 2019][1696556.879199] .throttle_count : 0 [Wed Feb 6 09:40:57 2019][1696556.883989] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:40:57 2019][1696556.888783] .avg->runnable_avg_period : 47287 [Wed Feb 6 09:40:57 2019][1696556.893927] [Wed Feb 6 09:40:57 2019][1696556.895595] rt_rq[35]:/ [Wed Feb 6 09:40:57 2019][1696556.898221] .rt_nr_running : 0 [Wed Feb 6 09:40:57 2019][1696556.903015] .rt_throttled : 0 [Wed Feb 6 09:40:57 2019][1696556.907806] .rt_time : 0.000000 [Wed Feb 6 09:40:57 2019][1696556.913207] .rt_runtime : 979.166667 [Wed Feb 6 09:40:57 2019][1696556.918780] [Wed Feb 6 09:40:57 2019][1696556.920450] runnable tasks: [Wed Feb 6 09:40:57 2019][1696556.923425] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:57 2019][1696556.934370] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:57 2019][1696556.945254] watchdog/35 186 -11.984330 424090 0 0.000000 162177.121878 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696556.956591] migration/35 187 0.000000 32787 0 0.000000 414102.380959 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696556.967929] ksoftirqd/35 188 12134660.228566 1263172 120 0.000000 18805.768996 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696556.979265] kworker/35:0H 190 2699.027895 9 100 0.000000 0.028843 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696556.990614] scsi_eh_1 2380 1221.395768 2 120 0.000000 0.020639 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.001952] kworker/35:1H 7526 12130562.018586 1053209 100 0.000000 3339.351949 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.013291] ccp-2-q2 10541 3652.252181 2 120 0.000000 0.010680 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.024628] ccp-4-q4 10689 3662.125534 2 120 0.000000 0.007554 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.035973] polkitd 17530 12540.083017 1 120 0.000000 0.040367 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.047308] JS GC Helper 17531 12552.208119 3 120 0.000000 0.125108 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.058646] gssproxy 17563 12622.428935 3 120 0.000000 3.352562 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.069989] kworker/35:2 19805 12194730.531420 322749 120 0.000000 8640.106365 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.081349] ptlrpcd_03_05 5937 12194744.534822 13118826 120 0.000000 196924.152421 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.092681] ptlrpcd_03_08 5940 12194744.625304 12633551 120 0.000000 201140.953720 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.104040] osp-pre-15-0 6319 12194744.488745 247668 120 0.000000 7089.509431 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.115389] osp-pre-24-2 6503 12194744.487012 235040 120 0.000000 3312.487253 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.126730] mdt03_005 6758 12194723.701157 36239577 120 0.000000 1819320.799396 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.138068] mdt03_006 6759 12194744.481441 31826035 120 0.000000 1360562.257341 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.149417] mdt03_010 7135 12194711.706184 33694348 120 0.000000 1506534.806868 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.160782] mdt_io03_013 22733 11638358.910282 1894557 120 0.000000 67103.637364 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.172154] mdt03_017 17053 12194711.705773 9544991 120 0.000000 844854.755842 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.183494] mdt03_022 17207 12194723.641373 7050980 120 0.000000 308989.114038 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.194828] mdt_io03_020 17208 11638358.935831 149369 120 0.000000 7576.032735 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.206169] ldlm_cn03_018 18063 11954050.277543 2520937 120 0.000000 237852.813225 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.217503] ldlm_cn03_022 26176 12134242.134803 95945 120 0.000000 127894.048103 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.228838] mdt_io03_023 27457 12194712.779572 24292 120 0.000000 3213.886339 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.240172] mdt_io03_024 27458 11567947.970688 80 120 0.000000 3.070003 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.251511] mdt_io03_027 27461 11638358.909561 13911 120 0.000000 2797.368753 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.262862] ldlm_cn03_023 28510 12107935.133595 49263 120 0.000000 17742.725148 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.274200] ldlm_cn03_025 28518 12194721.134319 61524 120 0.000000 17096.004135 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.285546] kworker/35:0 29665 11950358.881947 18 120 0.000000 0.125149 0.000000 3 / [Wed Feb 6 09:40:57 2019][1696557.296904] [Wed Feb 6 09:40:57 2019][1696557.298585] cpu#36, 1996.199 MHz [Wed Feb 6 09:40:57 2019][1696557.302003] .nr_running : 0 [Wed Feb 6 09:40:57 2019][1696557.306805] .load : 0 [Wed Feb 6 09:40:57 2019][1696557.311606] .nr_switches : 4116042348 [Wed Feb 6 09:40:57 2019][1696557.317187] .nr_load_updates : 473884751 [Wed Feb 6 09:40:57 2019][1696557.322681] .nr_uninterruptible : -484547 [Wed Feb 6 09:40:57 2019][1696557.328005] .next_balance : 5991.181210 [Wed Feb 6 09:40:57 2019][1696557.333672] .curr->pid : 0 [Wed Feb 6 09:40:57 2019][1696557.338474] .clock : 1696557337.906278 [Wed Feb 6 09:40:57 2019][1696557.344661] .cpu_load[0] : 0 [Wed Feb 6 09:40:57 2019][1696557.349459] .cpu_load[1] : 0 [Wed Feb 6 09:40:57 2019][1696557.354255] .cpu_load[2] : 0 [Wed Feb 6 09:40:57 2019][1696557.359053] .cpu_load[3] : 0 [Wed Feb 6 09:40:57 2019][1696557.363845] .cpu_load[4] : 0 [Wed Feb 6 09:40:57 2019][1696557.368642] .avg_idle : 9798993 [Wed Feb 6 09:40:57 2019][1696557.373963] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:57 2019][1696557.379284] [Wed Feb 6 09:40:57 2019][1696557.380958] cfs_rq[36]:/ [Wed Feb 6 09:40:57 2019][1696557.383671] .exec_clock : 0.000000 [Wed Feb 6 09:40:57 2019][1696557.389071] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:57 2019][1696557.394470] .min_vruntime : 28757569.015976 [Wed Feb 6 09:40:57 2019][1696557.400475] .max_vruntime : 0.000001 [Wed Feb 6 09:40:57 2019][1696557.405876] .spread : 0.000000 [Wed Feb 6 09:40:57 2019][1696557.411274] .spread0 : 631921.469038 [Wed Feb 6 09:40:57 2019][1696557.417108] .nr_spread_over : 0 [Wed Feb 6 09:40:57 2019][1696557.421902] .nr_running : 0 [Wed Feb 6 09:40:57 2019][1696557.426694] .load : 0 [Wed Feb 6 09:40:57 2019][1696557.431485] .runnable_load_avg : 0 [Wed Feb 6 09:40:57 2019][1696557.436279] .blocked_load_avg : 0 [Wed Feb 6 09:40:57 2019][1696557.441071] .tg_load_avg : 0 [Wed Feb 6 09:40:57 2019][1696557.445863] .tg_load_contrib : 0 [Wed Feb 6 09:40:57 2019][1696557.450658] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:57 2019][1696557.455451] .tg->runnable_avg : 0 [Wed Feb 6 09:40:57 2019][1696557.460242] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:57 2019][1696557.465036] .throttled : 0 [Wed Feb 6 09:40:57 2019][1696557.469828] .throttle_count : 0 [Wed Feb 6 09:40:57 2019][1696557.474620] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:40:57 2019][1696557.479413] .avg->runnable_avg_period : 46939 [Wed Feb 6 09:40:57 2019][1696557.484556] [Wed Feb 6 09:40:57 2019][1696557.486227] rt_rq[36]:/ [Wed Feb 6 09:40:57 2019][1696557.488851] .rt_nr_running : 0 [Wed Feb 6 09:40:57 2019][1696557.493644] .rt_throttled : 0 [Wed Feb 6 09:40:57 2019][1696557.498437] .rt_time : 0.000000 [Wed Feb 6 09:40:57 2019][1696557.503839] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:57 2019][1696557.509496] [Wed Feb 6 09:40:57 2019][1696557.511169] runnable tasks: [Wed Feb 6 09:40:57 2019][1696557.514141] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:57 2019][1696557.525088] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:57 2019][1696557.535972] watchdog/36 191 -11.987907 424097 0 0.000000 174807.176149 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.547309] migration/36 192 0.000000 27299 0 0.000000 426795.380388 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.558647] ksoftirqd/36 193 28757557.022248 9312646 120 0.000000 92483.076593 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.569982] kworker/36:0H 195 19793.586740 10 100 0.000000 0.036358 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.581342] ccp-6-q3 10855 4469.238969 2 120 0.000000 0.080442 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.592681] sssd_nss 17601 28757482.352081 447058 120 0.000000 17345.370417 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.604022] slapd 19480 17393.070215 7 120 0.000000 8.698442 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.615354] kworker/36:1H 20169 28757557.016403 9509901 100 0.000000 42643.693674 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.626705] ptlrpcd_00_03 5899 28757557.071681 6055790 120 0.000000 87762.696904 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.638045] ptlrpcd_00_11 5907 28757557.079235 6058207 120 0.000000 88473.976041 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.649414] osp-pre-17-2 6471 28757557.036215 234965 120 0.000000 9702.333226 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.660760] mdt00_005 6616 28757549.295674 223298066 120 0.000000 7428815.172101 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.672102] mdt_rdpg00_003 6787 28757566.066473 72769206 120 0.000000 1690846.346036 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.683441] mdt00_008 6806 27702678.894869 225162793 120 0.000000 8005541.195033 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.694783] mdt_out00_003 7092 27692273.036432 52515675 120 0.000000 1154932.765093 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.706121] mdt_io00_015 8975 28757513.981608 6255715 120 0.000000 237576.077609 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.717458] mdt00_016 8985 28755983.198965 197426634 120 0.000000 7050726.362814 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.728796] mdt00_019 8999 27702115.478065 229022937 120 0.000000 9044711.363656 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.740143] ldlm_cn00_017 14897 28751189.010845 11455182 120 0.000000 603335.291735 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.751476] ldlm_cn00_018 14898 28756294.801363 10232658 120 0.000000 498080.334425 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.762818] ldlm_cn00_021 19975 28756296.633763 11447873 120 0.000000 513761.137667 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.774155] ldlm_cn00_030 21242 28751178.061552 12813321 120 0.000000 508457.493744 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.785501] mdt00_026 32929 28755983.198823 123226912 120 0.000000 4163348.296438 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.796844] kworker/36:1 125245 28757557.103283 97894 120 0.000000 7248.279666 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.808267] mdt00_039 15261 28757549.094181 8254448 120 0.000000 603871.734533 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.819650] kworker/36:0 29485 28742742.951776 4 120 0.000000 0.035096 0.000000 0 / [Wed Feb 6 09:40:57 2019][1696557.831009] [Wed Feb 6 09:40:57 2019][1696557.832691] cpu#37, 1996.199 MHz [Wed Feb 6 09:40:57 2019][1696557.836108] .nr_running : 0 [Wed Feb 6 09:40:57 2019][1696557.840911] .load : 0 [Wed Feb 6 09:40:57 2019][1696557.845711] .nr_switches : 3271376987 [Wed Feb 6 09:40:57 2019][1696557.851291] .nr_load_updates : 376217220 [Wed Feb 6 09:40:57 2019][1696557.856783] .nr_uninterruptible : 101074 [Wed Feb 6 09:40:57 2019][1696557.862011] .next_balance : 5991.181746 [Wed Feb 6 09:40:57 2019][1696557.867680] .curr->pid : 0 [Wed Feb 6 09:40:57 2019][1696557.872483] .clock : 1696557871.919660 [Wed Feb 6 09:40:57 2019][1696557.878668] .cpu_load[0] : 0 [Wed Feb 6 09:40:58 2019][1696557.883470] .cpu_load[1] : 0 [Wed Feb 6 09:40:58 2019][1696557.888272] .cpu_load[2] : 0 [Wed Feb 6 09:40:58 2019][1696557.893074] .cpu_load[3] : 0 [Wed Feb 6 09:40:58 2019][1696557.897874] .cpu_load[4] : 0 [Wed Feb 6 09:40:58 2019][1696557.902676] .avg_idle : 3600269 [Wed Feb 6 09:40:58 2019][1696557.907998] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:58 2019][1696557.913318] [Wed Feb 6 09:40:58 2019][1696557.914995] cfs_rq[37]:/ [Wed Feb 6 09:40:58 2019][1696557.917706] .exec_clock : 0.000000 [Wed Feb 6 09:40:58 2019][1696557.923107] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:58 2019][1696557.928506] .min_vruntime : 20343128.742366 [Wed Feb 6 09:40:58 2019][1696557.934512] .max_vruntime : 0.000001 [Wed Feb 6 09:40:58 2019][1696557.939910] .spread : 0.000000 [Wed Feb 6 09:40:58 2019][1696557.945311] .spread0 : -7782521.696992 [Wed Feb 6 09:40:58 2019][1696557.951317] .nr_spread_over : 0 [Wed Feb 6 09:40:58 2019][1696557.956110] .nr_running : 0 [Wed Feb 6 09:40:58 2019][1696557.960902] .load : 0 [Wed Feb 6 09:40:58 2019][1696557.965694] .runnable_load_avg : 0 [Wed Feb 6 09:40:58 2019][1696557.970489] .blocked_load_avg : 0 [Wed Feb 6 09:40:58 2019][1696557.975281] .tg_load_avg : 0 [Wed Feb 6 09:40:58 2019][1696557.980073] .tg_load_contrib : 0 [Wed Feb 6 09:40:58 2019][1696557.984866] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:58 2019][1696557.989660] .tg->runnable_avg : 0 [Wed Feb 6 09:40:58 2019][1696557.994454] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:58 2019][1696557.999245] .throttled : 0 [Wed Feb 6 09:40:58 2019][1696558.004037] .throttle_count : 0 [Wed Feb 6 09:40:58 2019][1696558.008832] .avg->runnable_avg_sum : 66 [Wed Feb 6 09:40:58 2019][1696558.013709] .avg->runnable_avg_period : 47449 [Wed Feb 6 09:40:58 2019][1696558.018851] [Wed Feb 6 09:40:58 2019][1696558.020523] rt_rq[37]:/ [Wed Feb 6 09:40:58 2019][1696558.023148] .rt_nr_running : 0 [Wed Feb 6 09:40:58 2019][1696558.027942] .rt_throttled : 0 [Wed Feb 6 09:40:58 2019][1696558.032734] .rt_time : 0.000000 [Wed Feb 6 09:40:58 2019][1696558.038134] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:58 2019][1696558.043793] [Wed Feb 6 09:40:58 2019][1696558.045466] runnable tasks: [Wed Feb 6 09:40:58 2019][1696558.048439] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:58 2019][1696558.059385] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:58 2019][1696558.070245] systemd 1 20343103.801991 791985 120 0.000000 133450.835752 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.081605] watchdog/37 196 -11.986985 424087 0 0.000000 150623.598002 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.092942] migration/37 197 0.000000 56742 0 0.000000 455647.682686 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.104277] ksoftirqd/37 198 20343112.163133 7126278 120 0.000000 69063.697817 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.115615] kworker/37:0H 200 14838.457106 10 100 0.000000 0.041990 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.126965] kworker/37:1 7277 20343112.127257 711756 120 0.000000 21467.400666 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.138296] scsi_tmf_2 7340 2226.425597 2 100 0.000000 0.010952 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.149648] gssproxy 17572 12322.552063 2 120 0.000000 0.036069 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.160987] gssproxy 17574 12334.559602 1 120 0.000000 0.007545 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.172323] gssproxy 17575 12346.566870 1 120 0.000000 0.007274 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.183662] kworker/37:1H 18736 20343112.054340 4702056 100 0.000000 19825.404435 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.194997] agetty 19332 23335.064194 9 120 0.000000 7.197649 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.206336] cfs_rh_00 5813 19805144.091526 1206 120 0.000000 1620.586720 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.217675] kiblnd_sd_01_01 5880 20343129.239149 2884991353 120 0.000000 21169151.782404 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.229107] ptlrpcd_01_04 5912 20343112.263133 6964728 120 0.000000 104712.844483 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.240445] ptlrpcd_01_11 5919 20343112.314472 7021943 120 0.000000 101527.550385 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.251788] ldlm_cn01_002 6034 19696909.098377 28832332 120 0.000000 739139.656050 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.263135] mdt_io01_002 6214 19331023.172478 1487757 120 0.000000 66546.377114 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.274473] osp-pre-43-0 6273 20343112.077983 247845 120 0.000000 7012.339752 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.285810] osp-pre-33-0 6301 20343112.069015 247571 120 0.000000 4199.126994 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.297152] osp-syn-43-2 6462 20342958.988001 440998 120 0.000000 12533.401016 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.308514] mdt01_015 8949 19700749.319777 93889140 120 0.000000 3641236.511934 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.319858] mdt01_033 9017 20342856.549047 108744998 120 0.000000 4225569.880542 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.331196] mdt_out01_005 9053 20343117.050130 27563319 120 0.000000 551176.611120 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.342532] mdt_rdpg01_006 9146 20343112.067502 16015269 120 0.000000 687434.655937 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.353882] mdt_io01_039 20860 20343116.172893 8642670 120 0.000000 326758.841593 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.365214] mdt_io01_045 20866 19331027.517463 513968 120 0.000000 31244.326885 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.376558] ldlm_cn01_027 28328 19761373.329288 10035429 120 0.000000 410672.158324 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.387906] mdt01_057 15248 20343113.659735 1823378 120 0.000000 363523.703774 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.399242] mdt01_073 15389 20343120.507054 7945071 120 0.000000 660864.340066 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.410576] mdt_io01_059 15426 19331023.387438 290324 120 0.000000 11405.314477 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.421911] mdt_io01_060 15427 19591426.827924 50184 120 0.000000 1806.993106 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.433248] mdt_io01_062 15430 19591303.286584 82942 120 0.000000 3133.463410 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.444583] mdt_io01_064 15432 20342850.544244 15183 120 0.000000 570.876094 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.455922] mdt_io01_071 15439 19331452.776787 476 120 0.000000 18.228478 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.467257] mdt_io01_072 15440 19331027.858927 93997 120 0.000000 3639.896587 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.478631] ll_mgs_0005 28956 19623093.040833 467 120 0.000000 1092.166915 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.489972] kworker/37:2 30074 19700737.309945 4 120 0.000000 0.032441 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.501311] l_getidentity 30457 20342730.591449 3 120 0.000000 0.890914 0.000000 1 / [Wed Feb 6 09:40:58 2019][1696558.512663] [Wed Feb 6 09:40:58 2019][1696558.514341] cpu#38, 1996.199 MHz [Wed Feb 6 09:40:58 2019][1696558.517750] .nr_running : 0 [Wed Feb 6 09:40:58 2019][1696558.522553] .load : 0 [Wed Feb 6 09:40:58 2019][1696558.527354] .nr_switches : 680900389 [Wed Feb 6 09:40:58 2019][1696558.532849] .nr_load_updates : 259530198 [Wed Feb 6 09:40:58 2019][1696558.538342] .nr_uninterruptible : 125118 [Wed Feb 6 09:40:58 2019][1696558.543573] .next_balance : 5991.182425 [Wed Feb 6 09:40:58 2019][1696558.549235] .curr->pid : 0 [Wed Feb 6 09:40:58 2019][1696558.554038] .clock : 1696558553.935732 [Wed Feb 6 09:40:58 2019][1696558.560226] .cpu_load[0] : 0 [Wed Feb 6 09:40:58 2019][1696558.565024] .cpu_load[1] : 0 [Wed Feb 6 09:40:58 2019][1696558.569829] .cpu_load[2] : 0 [Wed Feb 6 09:40:58 2019][1696558.574626] .cpu_load[3] : 0 [Wed Feb 6 09:40:58 2019][1696558.579424] .cpu_load[4] : 0 [Wed Feb 6 09:40:58 2019][1696558.584225] .avg_idle : 10000000 [Wed Feb 6 09:40:58 2019][1696558.589630] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:58 2019][1696558.594951] [Wed Feb 6 09:40:58 2019][1696558.596620] cfs_rq[38]:/ [Wed Feb 6 09:40:58 2019][1696558.599330] .exec_clock : 0.000000 [Wed Feb 6 09:40:58 2019][1696558.604732] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:58 2019][1696558.610130] .min_vruntime : 15911613.640760 [Wed Feb 6 09:40:58 2019][1696558.616135] .max_vruntime : 0.000001 [Wed Feb 6 09:40:58 2019][1696558.621536] .spread : 0.000000 [Wed Feb 6 09:40:58 2019][1696558.626937] .spread0 : -12214036.896235 [Wed Feb 6 09:40:58 2019][1696558.633028] .nr_spread_over : 0 [Wed Feb 6 09:40:58 2019][1696558.637821] .nr_running : 0 [Wed Feb 6 09:40:58 2019][1696558.642615] .load : 0 [Wed Feb 6 09:40:58 2019][1696558.647407] .runnable_load_avg : 0 [Wed Feb 6 09:40:58 2019][1696558.652200] .blocked_load_avg : 0 [Wed Feb 6 09:40:58 2019][1696558.656993] .tg_load_avg : 0 [Wed Feb 6 09:40:58 2019][1696558.661785] .tg_load_contrib : 0 [Wed Feb 6 09:40:58 2019][1696558.666577] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:58 2019][1696558.671370] .tg->runnable_avg : 0 [Wed Feb 6 09:40:58 2019][1696558.676163] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:58 2019][1696558.680956] .throttled : 0 [Wed Feb 6 09:40:58 2019][1696558.685749] .throttle_count : 0 [Wed Feb 6 09:40:58 2019][1696558.690541] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:40:58 2019][1696558.695335] .avg->runnable_avg_period : 46526 [Wed Feb 6 09:40:58 2019][1696558.700477] [Wed Feb 6 09:40:58 2019][1696558.702148] rt_rq[38]:/ [Wed Feb 6 09:40:58 2019][1696558.704773] .rt_nr_running : 0 [Wed Feb 6 09:40:58 2019][1696558.709565] .rt_throttled : 0 [Wed Feb 6 09:40:58 2019][1696558.714359] .rt_time : 0.000000 [Wed Feb 6 09:40:58 2019][1696558.719758] .rt_runtime : 1000.000000 [Wed Feb 6 09:40:58 2019][1696558.725418] [Wed Feb 6 09:40:58 2019][1696558.727089] runnable tasks: [Wed Feb 6 09:40:58 2019][1696558.730063] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:58 2019][1696558.741008] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:58 2019][1696558.751894] watchdog/38 201 -11.984901 424085 0 0.000000 118656.898186 0.000000 2 / [Wed Feb 6 09:40:58 2019][1696558.763231] migration/38 202 0.000000 12471 0 0.000000 566861.749337 0.000000 2 / [Wed Feb 6 09:40:58 2019][1696558.774566] ksoftirqd/38 203 15911400.675739 7790943 120 0.000000 75000.074287 0.000000 2 / [Wed Feb 6 09:40:58 2019][1696558.785903] kworker/38:0H 205 12891.570933 10 100 0.000000 0.032303 0.000000 2 / [Wed Feb 6 09:40:58 2019][1696558.797260] ccp-6-q4 10861 3406.962738 2 120 0.000000 0.021752 0.000000 2 / [Wed Feb 6 09:40:58 2019][1696558.808602] kworker/38:1H 18229 15911400.499655 4729592 100 0.000000 17692.380444 0.000000 2 / [Wed Feb 6 09:40:58 2019][1696558.819954] ptlrpc_hr02_001 5849 15911601.656690 83759847 120 0.000000 777279.836892 0.000000 2 / [Wed Feb 6 09:40:58 2019][1696558.831339] osp-syn-44-2 6476 15911581.777738 436867 120 0.000000 12560.806009 0.000000 2 / [Wed Feb 6 09:40:58 2019][1696558.842686] mdt02_009 6827 15024266.706452 83144577 120 0.000000 3466131.319500 0.000000 2 / [Wed Feb 6 09:40:58 2019][1696558.854020] mdt_io02_008 7099 14659621.248101 5431308 120 0.000000 219541.328142 0.000000 2 / [Wed Feb 6 09:40:58 2019][1696558.865360] mdt_io02_015 8979 15016955.701634 5531009 120 0.000000 228193.160809 0.000000 2 / [Wed Feb 6 09:40:58 2019][1696558.876699] mdt02_015 8983 15911564.199377 83784304 120 0.000000 3500815.012348 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696558.888036] mdt02_020 9030 15911564.279459 94039930 120 0.000000 4308321.680603 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696558.899370] mdt_rdpg02_005 9148 15911613.640760 14419332 120 0.000000 652215.814049 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696558.910728] ldlm_cn02_016 23319 15910527.045423 16987703 120 0.000000 534703.533062 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696558.922061] mdt_rdpg02_007 25566 15009218.034242 15005938 120 0.000000 641404.019548 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696558.933402] mdt_rdpg02_009 50797 15911610.030214 1829627 120 0.000000 81544.050010 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696558.944733] mdt02_027 53422 15911564.204828 28945039 120 0.000000 1529975.028445 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696558.956088] mdt_io02_033 15451 14659612.783903 141 120 0.000000 4.976244 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696558.967424] mdt_io02_036 15454 15016943.858400 389328 120 0.000000 14646.770874 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696558.978758] mdt_io02_039 15457 15016943.858030 421049 120 0.000000 15629.226758 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696558.990099] kworker/38:1 16288 15911601.664465 69119 120 0.000000 13447.134183 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696559.001451] mdt_io02_055 27726 15018434.701647 136 120 0.000000 1479.834391 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696559.012786] mdt_io02_057 27735 15016943.860315 621 120 0.000000 2736.689206 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696559.024124] mdt_io02_067 27752 15111739.955803 2556 120 0.000000 1845.841224 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696559.035458] mdt_io02_068 27753 14659612.643653 95 120 0.000000 4.547837 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696559.046799] ldlm_cn02_025 28509 15859338.335286 44073 120 0.000000 19090.721292 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696559.058144] kworker/38:0 30176 15911045.145180 10 120 0.000000 0.055725 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696559.069491] kworker/38:2 30385 15911065.587917 4 120 0.000000 0.023855 0.000000 2 / [Wed Feb 6 09:40:59 2019][1696559.080854] [Wed Feb 6 09:40:59 2019][1696559.082535] cpu#39, 1996.199 MHz [Wed Feb 6 09:40:59 2019][1696559.085955] .nr_running : 0 [Wed Feb 6 09:40:59 2019][1696559.090752] .load : 0 [Wed Feb 6 09:40:59 2019][1696559.095550] .nr_switches : 421294316 [Wed Feb 6 09:40:59 2019][1696559.101038] .nr_load_updates : 163803713 [Wed Feb 6 09:40:59 2019][1696559.106536] .nr_uninterruptible : 31049 [Wed Feb 6 09:40:59 2019][1696559.111682] .next_balance : 5991.182996 [Wed Feb 6 09:40:59 2019][1696559.117349] .curr->pid : 0 [Wed Feb 6 09:40:59 2019][1696559.122151] .clock : 1696559121.950484 [Wed Feb 6 09:40:59 2019][1696559.128342] .cpu_load[0] : 0 [Wed Feb 6 09:40:59 2019][1696559.133139] .cpu_load[1] : 0 [Wed Feb 6 09:40:59 2019][1696559.137934] .cpu_load[2] : 0 [Wed Feb 6 09:40:59 2019][1696559.142734] .cpu_load[3] : 0 [Wed Feb 6 09:40:59 2019][1696559.147538] .cpu_load[4] : 0 [Wed Feb 6 09:40:59 2019][1696559.152336] .avg_idle : 2882173 [Wed Feb 6 09:40:59 2019][1696559.157647] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:40:59 2019][1696559.162961] [Wed Feb 6 09:40:59 2019][1696559.164630] cfs_rq[39]:/ [Wed Feb 6 09:40:59 2019][1696559.167343] .exec_clock : 0.000000 [Wed Feb 6 09:40:59 2019][1696559.172741] .MIN_vruntime : 0.000001 [Wed Feb 6 09:40:59 2019][1696559.178140] .min_vruntime : 12353576.709294 [Wed Feb 6 09:40:59 2019][1696559.184146] .max_vruntime : 0.000001 [Wed Feb 6 09:40:59 2019][1696559.189546] .spread : 0.000000 [Wed Feb 6 09:40:59 2019][1696559.194946] .spread0 : -15772073.827701 [Wed Feb 6 09:40:59 2019][1696559.201038] .nr_spread_over : 0 [Wed Feb 6 09:40:59 2019][1696559.205832] .nr_running : 0 [Wed Feb 6 09:40:59 2019][1696559.210625] .load : 0 [Wed Feb 6 09:40:59 2019][1696559.215418] .runnable_load_avg : 0 [Wed Feb 6 09:40:59 2019][1696559.220211] .blocked_load_avg : 4 [Wed Feb 6 09:40:59 2019][1696559.225004] .tg_load_avg : 0 [Wed Feb 6 09:40:59 2019][1696559.229795] .tg_load_contrib : 0 [Wed Feb 6 09:40:59 2019][1696559.234588] .tg_runnable_contrib : 0 [Wed Feb 6 09:40:59 2019][1696559.239382] .tg->runnable_avg : 0 [Wed Feb 6 09:40:59 2019][1696559.244174] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:40:59 2019][1696559.248966] .throttled : 0 [Wed Feb 6 09:40:59 2019][1696559.253759] .throttle_count : 0 [Wed Feb 6 09:40:59 2019][1696559.258553] .avg->runnable_avg_sum : 366 [Wed Feb 6 09:40:59 2019][1696559.263519] .avg->runnable_avg_period : 47488 [Wed Feb 6 09:40:59 2019][1696559.268661] [Wed Feb 6 09:40:59 2019][1696559.270331] rt_rq[39]:/ [Wed Feb 6 09:40:59 2019][1696559.272956] .rt_nr_running : 0 [Wed Feb 6 09:40:59 2019][1696559.277749] .rt_throttled : 0 [Wed Feb 6 09:40:59 2019][1696559.282543] .rt_time : 0.000000 [Wed Feb 6 09:40:59 2019][1696559.287943] .rt_runtime : 979.166667 [Wed Feb 6 09:40:59 2019][1696559.293514] [Wed Feb 6 09:40:59 2019][1696559.295186] runnable tasks: [Wed Feb 6 09:40:59 2019][1696559.298159] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:40:59 2019][1696559.309105] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:40:59 2019][1696559.319990] watchdog/39 206 -11.983258 424095 0 0.000000 104264.415272 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.331327] migration/39 207 0.000000 34737 0 0.000000 410388.270969 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.342665] ksoftirqd/39 208 12352905.571865 1457956 120 0.000000 19880.448210 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.354000] kworker/39:0H 210 30161.995987 10 100 0.000000 0.039177 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.365349] kworker/39:1 7276 12353564.563606 293831 120 0.000000 6710.791759 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.376693] kdmwork-253:0 17010 12353549.332955 378612 120 0.000000 6196.839574 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.388043] kworker/39:1H 29212 12353549.347963 1173139 100 0.000000 3500.407853 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.399388] ptlrpc_hr03_003 5858 12353473.171425 20354875 120 0.000000 178954.465112 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.410725] ptlrpc_hr03_004 5859 12353473.176636 20349228 120 0.000000 183483.266205 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.422067] ptlrpcd_rcv 5895 12310253.370049 970 120 0.000000 603.452127 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.433406] ptlrpcd_03_06 5938 12353564.697241 12631415 120 0.000000 199342.129158 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.444748] ldlm_bl_01 6041 12353473.154534 33568895 120 0.000000 754800.367869 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.456090] mdt_io03_001 6219 12105375.170368 2414659 120 0.000000 86171.541309 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.467425] osp_up1-0 6244 12353413.968769 365 120 0.000000 10.772649 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.478760] osp-pre-13-0 6251 12353540.763941 247826 120 0.000000 3851.604953 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.490099] osp-syn-31-0 6264 12353397.466455 788664 120 0.000000 20739.636911 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.501433] osp-pre-9-0 6269 12353564.009884 247819 120 0.000000 5875.393848 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.512770] osp-pre-44-0 6287 12353540.038232 247636 120 0.000000 6266.913408 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.524108] osp-pre-38-0 6317 12353542.940510 247614 120 0.000000 4912.328617 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.535442] osp-pre-3-0 6327 12353551.892241 247824 120 0.000000 3715.802878 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.546782] osp-pre-3-2 6437 12353531.091161 205446 120 0.000000 3329.827197 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.558114] osp-pre-39-2 6443 12353533.809472 234958 120 0.000000 5497.871847 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.569451] osp-pre-10-2 6447 12353564.669087 234834 120 0.000000 3305.022461 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.580794] osp-pre-37-2 6521 12353556.122796 235087 120 0.000000 5469.666646 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.592132] osp-pre-34-2 6533 12353539.885602 234991 120 0.000000 4681.307435 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.603473] ldlm_cn03_003 6763 12309182.552453 8093526 120 0.000000 380824.664968 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.614809] mdt03_007 6814 12102896.003701 30809444 120 0.000000 1679814.363158 0.000000 3 / [Wed Feb 6 09:40:59 2019][1696559.626143] mdt_io03_005 6869 12105375.138257 2025226 120 0.000000 69569.166519 0.000000[1696559.633967] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [migration/3:26] [Wed Feb 6 09:40:59 2019][1696559.633969] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure dm_multipath pcspkr ipmi_si ipmi_devintf dm_mod ccp k10temp ipmi_msghandler sg i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper ahci mlx5_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops mlxfw(OE) libahci devlink ttm crct10dif_pclmul crct10dif_common mlx_compat(OE) tg3 drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [Wed Feb 6 09:40:59 2019][1696559.634037] CPU: 3 PID: 26 Comm: migration/3 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Wed Feb 6 09:40:59 2019][1696559.634038] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [Wed Feb 6 09:40:59 2019][1696559.634040] task: ffff914aa9f7b0c0 ti: ffff914aa9f90000 task.ti: ffff914aa9f90000 [Wed Feb 6 09:40:59 2019][1696559.634041] RIP: 0010:[] [] multi_cpu_stop+0x4f/0x110 [Wed Feb 6 09:40:59 2019][1696559.634046] RSP: 0000:ffff914aa9f93d98 EFLAGS: 00000246 [Wed Feb 6 09:40:59 2019][1696559.634047] RAX: 0000000000000001 RBX: ffff914aa95e7c00 RCX: dead000000000200 [Wed Feb 6 09:40:59 2019][1696559.634048] RDX: ffff915a3f415ff0 RSI: 0000000000000282 RDI: ffff9139eef23ab0 [Wed Feb 6 09:40:59 2019][1696559.634048] RBP: ffff914aa9f93dc0 R08: ffff9139eef23a50 R09: 0000000000000001 [Wed Feb 6 09:40:59 2019][1696559.634049] R10: 000000000000b449 R11: 0000000000000001 R12: ffff914aa9f93d20 [Wed Feb 6 09:40:59 2019][1696559.634050] R13: ffff9144c9204100 R14: ffffffffadcd2ea0 R15: ffff914aa9f93d00 [Wed Feb 6 09:40:59 2019][1696559.634051] FS: 00007f78f341b880(0000) GS:ffff915a3f400000(0000) knlGS:0000000000000000 [Wed Feb 6 09:40:59 2019][1696559.634052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Wed Feb 6 09:40:59 2019][1696559.634053] CR2: 00007f78e33efe98 CR3: 0000004033a9a000 CR4: 00000000003407e0 [Wed Feb 6 09:40:59 2019][1696559.634054] Call Trace: [Wed Feb 6 09:40:59 2019][1696559.634056] [] ? cpu_stop_should_run+0x50/0x50 [Wed Feb 6 09:40:59 2019][1696559.634057] [] cpu_stopper_thread+0x99/0x150 [Wed Feb 6 09:40:59 2019][1696559.634060] [] ? __schedule+0x3ff/0x890 [Wed Feb 6 09:41:00 2019][1696559.634063] [] ? queued_spin_lock_slowpath+0xb/0xf [Wed Feb 6 09:41:00 2019][1696559.634065] [] smpboot_thread_fn+0x144/0x1a0 [Wed Feb 6 09:41:00 2019][1696559.634067] [] ? lg_double_unlock+0x40/0x40 [Wed Feb 6 09:41:00 2019][1696559.634068] [] kthread+0xd1/0xe0 [Wed Feb 6 09:41:00 2019][1696559.634070] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:00 2019][1696559.634072] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:41:00 2019][1696559.634073] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:00 2019][1696559.634073] Code: 89 c5 48 8b 47 18 48 85 c0 0f 84 b3 00 00 00 0f a3 18 19 db 85 db 41 0f 95 c6 45 31 ff 31 c0 0f 1f 44 00 00 f3 90 41 8b 5c 24 20 <39> c3 74 5d 83 fb 02 74 68 83 fb 03 75 05 45 84 f6 75 6e f0 41 [Wed Feb 6 09:41:00 2019] [Wed Feb 6 09:41:00 2019][1696559.935237] 3 / [Wed Feb 6 09:41:00 2019][1696559.937534] mdt03_013 16071 12353502.844461 35474731 120 0.000000 1429616.224910 0.000000 3 / [Wed Feb 6 09:41:00 2019][1696559.948873] mdt_io03_012 19930 12105375.141823 2033035 120 0.000000 69856.415093 0.000000 3 / [Wed Feb 6 09:41:00 2019][1696559.960250] mdt_io03_021 17209 12310515.657814 158844 120 0.000000 7956.652645 0.000000 3 / [Wed Feb 6 09:41:00 2019][1696559.971585] mdt03_025 17220 12102987.345030 9092803 120 0.000000 361640.018749 0.000000 3 / [Wed Feb 6 09:41:00 2019][1696559.982926] mdt03_032 24899 12353579.325207 541822 120 0.000000 185426.743629 0.000000 3 / [Wed Feb 6 09:41:00 2019][1696559.994259] ldlm_cn03_021 26053 12310359.603987 26682 120 0.000000 187509.943943 0.000000 3 / [Wed Feb 6 09:41:00 2019][1696560.005615] ldlm_cn03_029 28538 12353567.312754 88348 120 0.000000 9080.646565 0.000000 3 / [Wed Feb 6 09:41:00 2019][1696560.016951] kworker/39:0 28897 12079823.596516 4 120 0.000000 0.031510 0.000000 3 / [Wed Feb 6 09:41:00 2019][1696560.028315] [Wed Feb 6 09:41:00 2019][1696560.029991] cpu#40, 1996.199 MHz [Wed Feb 6 09:41:00 2019][1696560.033405] .nr_running : 0 [Wed Feb 6 09:41:00 2019][1696560.038209] .load : 0 [Wed Feb 6 09:41:00 2019][1696560.043010] .nr_switches : 4328537803 [Wed Feb 6 09:41:00 2019][1696560.048592] .nr_load_updates : 442437214 [Wed Feb 6 09:41:00 2019][1696560.054085] .nr_uninterruptible : -664456 [Wed Feb 6 09:41:00 2019][1696560.059406] .next_balance : 5991.183944 [Wed Feb 6 09:41:00 2019][1696560.065076] .curr->pid : 0 [Wed Feb 6 09:41:00 2019][1696560.069874] .clock : 1696560068.978110 [Wed Feb 6 09:41:00 2019][1696560.076066] .cpu_load[0] : 0 [Wed Feb 6 09:41:00 2019][1696560.080862] .cpu_load[1] : 0 [Wed Feb 6 09:41:00 2019][1696560.085654] .cpu_load[2] : 0 [Wed Feb 6 09:41:00 2019][1696560.090454] .cpu_load[3] : 0 [Wed Feb 6 09:41:00 2019][1696560.095249] .cpu_load[4] : 0 [Wed Feb 6 09:41:00 2019][1696560.100043] .avg_idle : 4674925 [Wed Feb 6 09:41:00 2019][1696560.105362] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:00 2019][1696560.110681] [Wed Feb 6 09:41:00 2019][1696560.112353] cfs_rq[40]:/ [Wed Feb 6 09:41:00 2019][1696560.115067] .exec_clock : 0.000000 [Wed Feb 6 09:41:00 2019][1696560.120466] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:00 2019][1696560.125865] .min_vruntime : 24781337.252460 [Wed Feb 6 09:41:00 2019][1696560.131870] .max_vruntime : 0.000001 [Wed Feb 6 09:41:00 2019][1696560.137269] .spread : 0.000000 [Wed Feb 6 09:41:00 2019][1696560.142669] .spread0 : -3344313.284535 [Wed Feb 6 09:41:00 2019][1696560.148675] .nr_spread_over : 0 [Wed Feb 6 09:41:00 2019][1696560.153469] .nr_running : 0 [Wed Feb 6 09:41:00 2019][1696560.158262] .load : 0 [Wed Feb 6 09:41:00 2019][1696560.163056] .runnable_load_avg : 0 [Wed Feb 6 09:41:00 2019][1696560.167847] .blocked_load_avg : 0 [Wed Feb 6 09:41:00 2019][1696560.172641] .tg_load_avg : 0 [Wed Feb 6 09:41:00 2019][1696560.177432] .tg_load_contrib : 0 [Wed Feb 6 09:41:00 2019][1696560.182225] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:00 2019][1696560.187018] .tg->runnable_avg : 0 [Wed Feb 6 09:41:00 2019][1696560.191810] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:00 2019][1696560.196603] .throttled : 0 [Wed Feb 6 09:41:00 2019][1696560.201397] .throttle_count : 0 [Wed Feb 6 09:41:00 2019][1696560.206189] .avg->runnable_avg_sum : 63 [Wed Feb 6 09:41:00 2019][1696560.211068] .avg->runnable_avg_period : 46351 [Wed Feb 6 09:41:00 2019][1696560.216215] [Wed Feb 6 09:41:00 2019][1696560.217891] rt_rq[40]:/ [Wed Feb 6 09:41:00 2019][1696560.220516] .rt_nr_running : 0 [Wed Feb 6 09:41:00 2019][1696560.225308] .rt_throttled : 0 [Wed Feb 6 09:41:00 2019][1696560.230102] .rt_time : 0.005671 [Wed Feb 6 09:41:00 2019][1696560.235502] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:00 2019][1696560.241159] [Wed Feb 6 09:41:00 2019][1696560.242832] runnable tasks: [Wed Feb 6 09:41:00 2019][1696560.245805] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:00 2019][1696560.256753] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:00 2019][1696560.267638] watchdog/40 211 -11.987747 424089 0 0.000000 133530.198641 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.278974] migration/40 212 0.000000 3322 0 0.000000 558371.510622 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.290311] ksoftirqd/40 213 24772944.833156 8890346 120 0.000000 92901.073336 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.301647] kworker/40:0H 215 4077.662548 9 100 0.000000 0.037231 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.313007] kworker/40:1H 16783 24772944.791423 10355169 100 0.000000 48182.637815 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.324370] kiblnd_sd_00_02 5877 24781337.706433 4154949431 120 0.000000 27448508.078846 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.335798] ptlrpcd_00_01 5897 24781309.330909 6071543 120 0.000000 89715.930195 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.347141] ldlm_cb00_000 6019 24528045.968548 5820719 120 0.000000 118320.101890 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.358480] mgs_fir_notify 6062 41698.450586 96 118 0.000000 1.589802 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.369818] mdt_io00_001 6210 23558769.066413 5026425 120 0.000000 224237.613776 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.381180] osp-pre-3-2 6517 24781321.170356 234812 120 0.000000 6600.770307 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.392535] mdt_io00_010 7100 23973020.973288 4919197 120 0.000000 197890.011807 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.403882] mdt_io00_020 9003 23973020.997685 6691949 120 0.000000 259935.891091 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.415225] ldlm_cn00_011 12237 24643900.893925 16361482 120 0.000000 648409.694902 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.426563] ldlm_cn00_012 14696 24643900.893584 13035022 120 0.000000 660578.583231 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.437906] mdt_io00_025 20197 23973020.981744 6476159 120 0.000000 252774.570053 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.449261] kworker/40:1 61854 24781319.754914 272311 120 0.000000 9389.843814 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.460614] mdt_io00_031 15462 23973020.982185 130725 120 0.000000 5177.748966 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.471951] mdt_io00_035 15466 23973020.979020 345898 120 0.000000 14012.813302 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.483301] mdt_io00_039 27673 23973020.989590 370 120 0.000000 15.171936 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.494632] mdt_io00_040 27674 24135489.713420 2408 120 0.000000 436.710990 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.505976] mdt_io00_060 27731 23973021.110910 146 120 0.000000 5.403064 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.517323] kworker/40:2 28621 23843991.792406 19 120 0.000000 0.157569 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.528672] l_getidentity 30462 24781228.299228 2 120 0.000000 0.867729 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.540002] l_getidentity 30467 24781268.864465 2 120 0.000000 0.901694 0.000000 0 / [Wed Feb 6 09:41:00 2019][1696560.551355] [Wed Feb 6 09:41:00 2019][1696560.553036] cpu#41, 1996.199 MHz [Wed Feb 6 09:41:00 2019][1696560.556455] .nr_running : 0 [Wed Feb 6 09:41:00 2019][1696560.561255] .load : 0 [Wed Feb 6 09:41:00 2019][1696560.566057] .nr_switches : 2973859047 [Wed Feb 6 09:41:00 2019][1696560.571633] .nr_load_updates : 364161503 [Wed Feb 6 09:41:00 2019][1696560.577124] .nr_uninterruptible : 154142 [Wed Feb 6 09:41:00 2019][1696560.582358] .next_balance : 5991.184465 [Wed Feb 6 09:41:00 2019][1696560.588026] .curr->pid : 0 [Wed Feb 6 09:41:00 2019][1696560.592829] .clock : 1696560591.989102 [Wed Feb 6 09:41:00 2019][1696560.599017] .cpu_load[0] : 0 [Wed Feb 6 09:41:00 2019][1696560.603819] .cpu_load[1] : 0 [Wed Feb 6 09:41:00 2019][1696560.608620] .cpu_load[2] : 0 [Wed Feb 6 09:41:00 2019][1696560.613420] .cpu_load[3] : 0 [Wed Feb 6 09:41:00 2019][1696560.618220] .cpu_load[4] : 0 [Wed Feb 6 09:41:00 2019][1696560.623025] .avg_idle : 3679079 [Wed Feb 6 09:41:00 2019][1696560.628345] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:00 2019][1696560.633665] [Wed Feb 6 09:41:00 2019][1696560.635339] cfs_rq[41]:/ [Wed Feb 6 09:41:00 2019][1696560.638052] .exec_clock : 0.000000 [Wed Feb 6 09:41:00 2019][1696560.643451] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:00 2019][1696560.648850] .min_vruntime : 17535983.008565 [Wed Feb 6 09:41:00 2019][1696560.654856] .max_vruntime : 0.000001 [Wed Feb 6 09:41:00 2019][1696560.660257] .spread : 0.000000 [Wed Feb 6 09:41:00 2019][1696560.665657] .spread0 : -10589667.528430 [Wed Feb 6 09:41:00 2019][1696560.671748] .nr_spread_over : 0 [Wed Feb 6 09:41:00 2019][1696560.676542] .nr_running : 0 [Wed Feb 6 09:41:00 2019][1696560.681335] .load : 0 [Wed Feb 6 09:41:00 2019][1696560.686128] .runnable_load_avg : 0 [Wed Feb 6 09:41:00 2019][1696560.690920] .blocked_load_avg : 0 [Wed Feb 6 09:41:00 2019][1696560.695714] .tg_load_avg : 0 [Wed Feb 6 09:41:00 2019][1696560.700506] .tg_load_contrib : 0 [Wed Feb 6 09:41:00 2019][1696560.705299] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:00 2019][1696560.710090] .tg->runnable_avg : 0 [Wed Feb 6 09:41:00 2019][1696560.714883] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:00 2019][1696560.719678] .throttled : 0 [Wed Feb 6 09:41:00 2019][1696560.724469] .throttle_count : 0 [Wed Feb 6 09:41:00 2019][1696560.729263] .avg->runnable_avg_sum : 6 [Wed Feb 6 09:41:00 2019][1696560.734055] .avg->runnable_avg_period : 46185 [Wed Feb 6 09:41:00 2019][1696560.739198] [Wed Feb 6 09:41:00 2019][1696560.740868] rt_rq[41]:/ [Wed Feb 6 09:41:00 2019][1696560.743493] .rt_nr_running : 0 [Wed Feb 6 09:41:00 2019][1696560.748286] .rt_throttled : 0 [Wed Feb 6 09:41:00 2019][1696560.753079] .rt_time : 0.000000 [Wed Feb 6 09:41:00 2019][1696560.758477] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:00 2019][1696560.764137] [Wed Feb 6 09:41:00 2019][1696560.765809] runnable tasks: [Wed Feb 6 09:41:00 2019][1696560.768784] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:00 2019][1696560.779729] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:00 2019][1696560.790615] watchdog/41 216 -11.986625 424070 0 0.000000 123313.627072 0.000000 1 / [Wed Feb 6 09:41:00 2019][1696560.801952] migration/41 217 0.000000 5969 0 0.000000 464760.583182 0.000000 1 / [Wed Feb 6 09:41:00 2019][1696560.813288] ksoftirqd/41 218 17535813.829736 9138361 120 0.000000 83803.072935 0.000000 1 / [Wed Feb 6 09:41:00 2019][1696560.824623] kworker/41:0H 220 6383.959562 10 100 0.000000 0.039666 0.000000 1 / [Wed Feb 6 09:41:00 2019][1696560.835966] edac-poller 265 17063534.067963 5 100 0.000000 0.029446 0.000000 1 / [Wed Feb 6 09:41:00 2019][1696560.847341] kworker/41:1H 30572 17535813.659452 5960114 100 0.000000 24381.784184 0.000000 1 / [Wed Feb 6 09:41:00 2019][1696560.858690] ptlrpcd_01_00 5908 17535971.081163 6970823 120 0.000000 105139.899385 0.000000 1 / [Wed Feb 6 09:41:00 2019][1696560.870029] lc_watchdogd 6020 17535644.371939 14154 120 0.000000 19463.434984 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696560.881389] osp-syn-3-0 6326 17535105.322654 770268 120 0.000000 20820.000807 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696560.892740] mdt01_003 6591 17535973.922037 97052457 120 0.000000 3728942.675732 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696560.904077] ldlm_cn01_005 6768 17510622.353347 34441729 120 0.000000 1004156.679082 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696560.915415] mdt_io01_009 6867 17395523.030422 4588644 120 0.000000 172446.994514 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696560.926769] mdt01_028 9010 17396394.019405 109573915 120 0.000000 4177120.933159 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696560.938107] mdt01_030 9014 17535835.898329 97597168 120 0.000000 4125203.195173 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696560.949477] kworker/41:2 65939 17535952.449580 122561 120 0.000000 5664.276589 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696560.960823] mdt01_061 15260 17535978.224883 10284411 120 0.000000 482281.096790 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696560.972164] mdt01_072 15388 17396394.019455 253267 120 0.000000 10117.365716 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696560.983500] mdt_io01_076 15446 17391784.796483 12638 120 0.000000 472.882170 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696560.994853] mdt_io01_086 27689 17389726.194292 25665 120 0.000000 990.849283 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696561.006188] mdt_io01_090 27696 17007240.988510 802 120 0.000000 28.457060 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696561.017544] kworker/41:1 29345 17435032.909974 2 120 0.000000 0.008246 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696561.028882] l_getidentity 30470 17535951.781401 2 120 0.000000 0.873159 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696561.040220] l_getidentity 30471 17535952.584246 2 120 0.000000 0.823204 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696561.051557] l_getidentity 30472 17535953.362004 2 120 0.000000 0.801133 0.000000 1 / [Wed Feb 6 09:41:01 2019][1696561.062909] [Wed Feb 6 09:41:01 2019][1696561.064589] cpu#42, 1996.199 MHz [Wed Feb 6 09:41:01 2019][1696561.068008] .nr_running : 0 [Wed Feb 6 09:41:01 2019][1696561.072809] .load : 0 [Wed Feb 6 09:41:01 2019][1696561.077611] .nr_switches : 1024094450 [Wed Feb 6 09:41:01 2019][1696561.083190] .nr_load_updates : 319881339 [Wed Feb 6 09:41:01 2019][1696561.088687] .nr_uninterruptible : 113207 [Wed Feb 6 09:41:01 2019][1696561.093918] .next_balance : 5991.184978 [Wed Feb 6 09:41:01 2019][1696561.099588] .curr->pid : 0 [Wed Feb 6 09:41:01 2019][1696561.104393] .clock : 1696561104.167958 [Wed Feb 6 09:41:01 2019][1696561.110580] .cpu_load[0] : 0 [Wed Feb 6 09:41:01 2019][1696561.115380] .cpu_load[1] : 0 [Wed Feb 6 09:41:01 2019][1696561.120178] .cpu_load[2] : 0 [Wed Feb 6 09:41:01 2019][1696561.124975] .cpu_load[3] : 0 [Wed Feb 6 09:41:01 2019][1696561.129773] .cpu_load[4] : 0 [Wed Feb 6 09:41:01 2019][1696561.134570] .avg_idle : 5477532 [Wed Feb 6 09:41:01 2019][1696561.139892] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:01 2019][1696561.145213] [Wed Feb 6 09:41:01 2019][1696561.146884] cfs_rq[42]:/ [Wed Feb 6 09:41:01 2019][1696561.149597] .exec_clock : 0.000000 [Wed Feb 6 09:41:01 2019][1696561.154996] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:01 2019][1696561.160398] .min_vruntime : 11679675.081838 [Wed Feb 6 09:41:01 2019][1696561.166401] .max_vruntime : 0.000001 [Wed Feb 6 09:41:01 2019][1696561.171803] .spread : 0.000000 [Wed Feb 6 09:41:01 2019][1696561.177203] .spread0 : -16445975.455157 [Wed Feb 6 09:41:01 2019][1696561.183294] .nr_spread_over : 0 [Wed Feb 6 09:41:01 2019][1696561.188088] .nr_running : 0 [Wed Feb 6 09:41:01 2019][1696561.192879] .load : 0 [Wed Feb 6 09:41:01 2019][1696561.197672] .runnable_load_avg : 0 [Wed Feb 6 09:41:01 2019][1696561.202465] .blocked_load_avg : 0 [Wed Feb 6 09:41:01 2019][1696561.207258] .tg_load_avg : 0 [Wed Feb 6 09:41:01 2019][1696561.212050] .tg_load_contrib : 0 [Wed Feb 6 09:41:01 2019][1696561.216845] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:01 2019][1696561.221635] .tg->runnable_avg : 0 [Wed Feb 6 09:41:01 2019][1696561.226429] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:01 2019][1696561.231221] .throttled : 0 [Wed Feb 6 09:41:01 2019][1696561.236015] .throttle_count : 0 [Wed Feb 6 09:41:01 2019][1696561.240807] .avg->runnable_avg_sum : 22 [Wed Feb 6 09:41:01 2019][1696561.245688] .avg->runnable_avg_period : 47386 [Wed Feb 6 09:41:01 2019][1696561.250828] [Wed Feb 6 09:41:01 2019][1696561.252500] rt_rq[42]:/ [Wed Feb 6 09:41:01 2019][1696561.255124] .rt_nr_running : 0 [Wed Feb 6 09:41:01 2019][1696561.259918] .rt_throttled : 0 [Wed Feb 6 09:41:01 2019][1696561.264710] .rt_time : 0.000000 [Wed Feb 6 09:41:01 2019][1696561.270112] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:01 2019][1696561.275770] [Wed Feb 6 09:41:01 2019][1696561.277443] runnable tasks: [Wed Feb 6 09:41:01 2019][1696561.280416] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:01 2019][1696561.291361] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:01 2019][1696561.302249] watchdog/42 221 -11.986555 424089 0 0.000000 126119.989263 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.313582] migration/42 222 0.000000 2103 0 0.000000 559725.357437 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.324918] ksoftirqd/42 223 11679632.718025 6686732 120 0.000000 65773.862365 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.336257] kworker/42:0H 225 4701.281756 10 100 0.000000 0.038263 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.347621] polkitd 17539 3792.326888 3 120 0.000000 0.095302 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.358961] writer#3 19096 11679657.507276 18705073 120 0.000000 192944.777639 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.370305] kworker/42:1H 30717 11679601.145190 4028977 100 0.000000 13870.363632 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.381654] ptlrpcd_02_06 5926 11679658.125348 13160360 120 0.000000 188409.531653 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.393002] mdt_rdpg02_001 6184 11128641.449322 13544605 120 0.000000 609757.532018 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.404347] osp-pre-35-0 6337 11679655.849751 247823 120 0.000000 4070.223315 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.415682] osp-syn-26-2 6452 11679539.912533 436888 120 0.000000 13597.118343 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.427018] osp-pre-32-2 6479 11679657.715783 235085 120 0.000000 3427.216534 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.438360] mdt02_006 6694 11134708.235317 81928097 120 0.000000 3741986.046167 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.449697] mdt02_007 6761 11679551.257903 84619680 120 0.000000 3480172.357332 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.461039] mdt02_008 6804 11679539.254907 91045834 120 0.000000 4380518.750884 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.472374] mdt_io02_006 7096 11129348.378615 5086049 120 0.000000 197612.524100 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.483706] mdt_io02_009 7101 11679509.977609 7307003 120 0.000000 292200.944673 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.495047] mdt_io02_014 8963 11129346.800075 5472320 120 0.000000 218134.211294 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.506380] mdt02_016 8984 11133219.827258 85518365 120 0.000000 4470784.401609 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.517717] mdt02_017 9006 11679675.254301 70259543 120 0.000000 3226850.300519 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.529053] ldlm_cn02_007 9032 11679651.666911 19778777 120 0.000000 635868.211285 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.540387] mdt_rdpg02_004 9147 11128641.448991 12756113 120 0.000000 577097.152502 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.551726] mdt_io02_021 9200 11129346.855070 6054987 120 0.000000 242254.898861 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.563064] kworker/42:2 12966 11679663.038717 561460 120 0.000000 17103.192183 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.574404] mdt_io02_025 20190 11679663.086166 5703843 120 0.000000 227827.210594 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.585742] mdt_io02_026 20859 11129338.389862 5950971 120 0.000000 235184.378259 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.597083] ldlm_cn02_012 23313 11588192.274672 12275903 120 0.000000 506087.405938 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.608436] mdt02_028 15178 11133571.060722 4678060 120 0.000000 167171.008751 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.619767] mdt02_034 15186 11678556.734087 5639680 120 0.000000 513073.654597 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.631108] mdt02_039 15267 11678556.734128 2926963 120 0.000000 387609.335253 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.642446] mdt_io02_027 15429 11129346.751654 367920 120 0.000000 15294.020138 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.653788] mdt_io02_030 15448 11129347.214523 254062 120 0.000000 9708.985141 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.665120] mdt_io02_031 15449 11129346.809173 268858 120 0.000000 10157.649367 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.676456] mdt_io02_032 15450 11129346.892441 251314 120 0.000000 13355.102272 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.687813] mdt_io02_043 27690 11129346.901369 202 120 0.000000 8.920090 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.699149] mdt_io02_046 27701 11129346.825493 302 120 0.000000 13.848419 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.710484] mdt_io02_049 27711 11129346.771672 645 120 0.000000 26.096358 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.721819] mdt_io02_051 27716 11129346.847045 3107 120 0.000000 194.595248 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.733156] mdt_io02_052 27719 11129346.857344 216 120 0.000000 8.269715 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.744496] mdt_io02_058 27737 11129346.819132 160 120 0.000000 8.018722 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.755829] mdt_io02_059 27738 11129346.921006 19469 120 0.000000 937.592721 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.767164] mdt_io02_061 27740 10218977.205645 129 120 0.000000 5.711852 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.778506] mdt_io02_075 27766 11129346.847996 333 120 0.000000 16.913505 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.789838] mdt_io02_077 27768 10218983.797126 29 120 0.000000 0.804227 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.801176] mdt_io02_080 27771 11129346.799686 534 120 0.000000 22.621640 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.812510] mdt_io02_081 27772 10218983.876915 21 120 0.000000 0.946630 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.823846] mdt_io02_088 27779 11129346.847676 22234 120 0.000000 791.225376 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.835183] mdt_io02_091 27782 11129346.835885 326 120 0.000000 14.328958 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.846519] ldlm_cn02_024 28508 11588367.981513 91764 120 0.000000 21697.367274 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.857869] kworker/42:1 30164 11587534.232974 4 120 0.000000 0.024989 0.000000 2 / [Wed Feb 6 09:41:01 2019][1696561.869221] [Wed Feb 6 09:41:01 2019][1696561.870898] cpu#43, 1996.199 MHz [Wed Feb 6 09:41:01 2019][1696561.874313] .nr_running : 0 [Wed Feb 6 09:41:01 2019][1696561.879116] .load : 0 [Wed Feb 6 09:41:02 2019][1696561.883913] .nr_switches : 477966646 [Wed Feb 6 09:41:02 2019][1696561.889403] .nr_load_updates : 160133848 [Wed Feb 6 09:41:02 2019][1696561.894899] .nr_uninterruptible : 24206 [Wed Feb 6 09:41:02 2019][1696561.900042] .next_balance : 5991.185784 [Wed Feb 6 09:41:02 2019][1696561.905706] .curr->pid : 0 [Wed Feb 6 09:41:02 2019][1696561.910507] .clock : 1696561910.021052 [Wed Feb 6 09:41:02 2019][1696561.916691] .cpu_load[0] : 0 [Wed Feb 6 09:41:02 2019][1696561.921483] .cpu_load[1] : 0 [Wed Feb 6 09:41:02 2019][1696561.926276] .cpu_load[2] : 0 [Wed Feb 6 09:41:02 2019][1696561.931069] .cpu_load[3] : 0 [Wed Feb 6 09:41:02 2019][1696561.935862] .cpu_load[4] : 0 [Wed Feb 6 09:41:02 2019][1696561.940664] .avg_idle : 10000000 [Wed Feb 6 09:41:02 2019][1696561.946065] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:02 2019][1696561.951385] [Wed Feb 6 09:41:02 2019][1696561.953063] cfs_rq[43]:/ [Wed Feb 6 09:41:02 2019][1696561.955773] .exec_clock : 0.000000 [Wed Feb 6 09:41:02 2019][1696561.961174] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:02 2019][1696561.966574] .min_vruntime : 11252321.729469 [Wed Feb 6 09:41:02 2019][1696561.972579] .max_vruntime : 0.000001 [Wed Feb 6 09:41:02 2019][1696561.977977] .spread : 0.000000 [Wed Feb 6 09:41:02 2019][1696561.983376] .spread0 : -16873328.885505 [Wed Feb 6 09:41:02 2019][1696561.989470] .nr_spread_over : 0 [Wed Feb 6 09:41:02 2019][1696561.994263] .nr_running : 0 [Wed Feb 6 09:41:02 2019][1696561.999055] .load : 0 [Wed Feb 6 09:41:02 2019][1696562.003849] .runnable_load_avg : 0 [Wed Feb 6 09:41:02 2019][1696562.008641] .blocked_load_avg : 0 [Wed Feb 6 09:41:02 2019][1696562.013435] .tg_load_avg : 0 [Wed Feb 6 09:41:02 2019][1696562.018227] .tg_load_contrib : 0 [Wed Feb 6 09:41:02 2019][1696562.023019] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:02 2019][1696562.027814] .tg->runnable_avg : 0 [Wed Feb 6 09:41:02 2019][1696562.032605] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:02 2019][1696562.037397] .throttled : 0 [Wed Feb 6 09:41:02 2019][1696562.042191] .throttle_count : 0 [Wed Feb 6 09:41:02 2019][1696562.046985] .avg->runnable_avg_sum : 8 [Wed Feb 6 09:41:02 2019][1696562.051776] .avg->runnable_avg_period : 47805 [Wed Feb 6 09:41:02 2019][1696562.056919] [Wed Feb 6 09:41:02 2019][1696562.058589] rt_rq[43]:/ [Wed Feb 6 09:41:02 2019][1696562.061216] .rt_nr_running : 0 [Wed Feb 6 09:41:02 2019][1696562.066009] .rt_throttled : 0 [Wed Feb 6 09:41:02 2019][1696562.070800] .rt_time : 0.000000 [Wed Feb 6 09:41:02 2019][1696562.076200] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:02 2019][1696562.081860] [Wed Feb 6 09:41:02 2019][1696562.083531] runnable tasks: [Wed Feb 6 09:41:02 2019][1696562.086505] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:02 2019][1696562.097451] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:02 2019][1696562.108339] watchdog/43 226 -11.987606 424085 0 0.000000 107226.422759 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.119673] migration/43 227 0.000000 3640 0 0.000000 539201.057055 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.131009] ksoftirqd/43 228 11252214.232669 1228539 120 0.000000 13751.738479 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.142345] kworker/43:0H 230 11234.770160 10 100 0.000000 0.035404 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.153694] kworker/43:1 7280 11252309.784926 310095 120 0.000000 4399.882415 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.165036] ccp-6-q0 10781 2858.446469 2 120 0.000000 0.011342 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.176371] ccp-7-q3 10876 2880.423668 2 120 0.000000 0.008617 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.187709] ccp-7-q4 10878 2886.429323 2 120 0.000000 0.005661 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.199044] ccp-8-q0 10883 2906.422603 2 120 0.000000 0.007014 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.210381] ccp-8-q1 10884 2914.428532 2 120 0.000000 0.005931 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.221717] ccp-8-q2 10885 2922.433940 2 120 0.000000 0.005410 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.233052] ccp-8-q3 10886 2930.439679 2 120 0.000000 0.005741 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.244389] ccp-8-q4 10887 2938.444837 2 120 0.000000 0.005160 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.255727] kmpath_handlerd 11777 3451.259717 2 100 0.000000 0.016350 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.267068] irqbalance 17549 11252315.574988 248919 120 0.000000 1315585.484310 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.278419] kworker/43:1H 49676 11252291.406594 1006781 100 0.000000 3004.077423 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.289762] kiblnd_sd_03_03 5890 11252321.915152 440055046 120 0.000000 3227471.358481 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.301116] mdt_rdpg03_001 6186 11252292.380440 4179674 120 0.000000 166519.093172 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.312456] osp-pre-3-0 6247 11252304.580476 205452 120 0.000000 2445.744921 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.323786] osp-pre-13-0 6251 11252307.812637 247827 120 0.000000 3851.614251 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.335126] osp-pre-4-0 6297 11252306.165281 247814 120 0.000000 4378.918625 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.346460] osp-pre-24-0 6313 11252307.707495 247838 120 0.000000 7632.713065 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.357796] osp-pre-3-0 6327 11252309.948275 247825 120 0.000000 3715.814961 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.369134] osp-pre-34-0 6343 11252307.742801 247843 120 0.000000 3577.832750 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.380470] osp-pre-39-2 6443 11252306.891106 234959 120 0.000000 5497.881775 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.391806] osp-pre-11-2 6449 11252304.035735 234863 120 0.000000 7187.642346 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.403141] osp-pre-14-2 6451 11252306.547674 235169 120 0.000000 6493.643123 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.414480] osp-syn-31-2 6454 11252231.129265 443376 120 0.000000 12367.403930 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.425813] osp-pre-9-2 6459 11252305.245254 234781 120 0.000000 5229.538149 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.437154] osp-pre-29-2 6511 11252306.044562 235001 120 0.000000 3346.453955 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.448487] osp-syn-37-2 6520 11252227.994339 441094 120 0.000000 14263.174657 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.459848] mdt_io03_007 9000 10368287.969715 2257774 120 0.000000 81025.549032 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.471200] mdt_rdpg03_003 19894 10134960.983491 5355850 120 0.000000 219131.939911 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.482547] ldlm_cn03_009 23310 11237096.561946 10168888 120 0.000000 399298.097225 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.493884] ldlm_cn03_010 23311 11252304.073228 11563430 120 0.000000 495455.552069 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.505228] mdt03_014 47983 11252291.509202 21850342 120 0.000000 1346542.875932 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.516590] mdt03_015 17050 10633417.299865 9473776 120 0.000000 3870528.193957 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.527930] mdt03_023 17218 11252226.716470 5348056 120 0.000000 516725.891647 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.539263] mdt03_026 17221 11252307.365570 9458386 120 0.000000 988405.341282 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.550630] ll_mgs_0018 28986 10164656.511366 403 120 0.000000 18.623267 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.561967] kworker/43:0 29622 10633324.948583 22 120 0.000000 0.142912 0.000000 3 / [Wed Feb 6 09:41:02 2019][1696562.573321] [Wed Feb 6 09:41:02 2019][1696562.575003] cpu#44, 1996.199 MHz [Wed Feb 6 09:41:02 2019][1696562.578419] .nr_running : 1 [Wed Feb 6 09:41:02 2019][1696562.583221] .load : 1024 [Wed Feb 6 09:41:02 2019][1696562.588285] .nr_switches : 4128870180 [Wed Feb 6 09:41:02 2019][1696562.593865] .nr_load_updates : 473811064 [Wed Feb 6 09:41:02 2019][1696562.599359] .nr_uninterruptible : -539012 [Wed Feb 6 09:41:02 2019][1696562.604682] .next_balance : 5991.186530 [Wed Feb 6 09:41:02 2019][1696562.610345] .curr->pid : 6615 [Wed Feb 6 09:41:02 2019][1696562.615408] .clock : 1696562615.041023 [Wed Feb 6 09:41:02 2019][1696562.621595] .cpu_load[0] : 1024 [Wed Feb 6 09:41:02 2019][1696562.626647] .cpu_load[1] : 1024 [Wed Feb 6 09:41:02 2019][1696562.631705] .cpu_load[2] : 1024 [Wed Feb 6 09:41:02 2019][1696562.636761] .cpu_load[3] : 1024 [Wed Feb 6 09:41:02 2019][1696562.641817] .cpu_load[4] : 1024 [Wed Feb 6 09:41:02 2019][1696562.646877] .avg_idle : 10000000 [Wed Feb 6 09:41:02 2019][1696562.652287] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:02 2019][1696562.657608] [Wed Feb 6 09:41:02 2019][1696562.659284] cfs_rq[44]:/ [Wed Feb 6 09:41:02 2019][1696562.661997] .exec_clock : 0.000000 [Wed Feb 6 09:41:02 2019][1696562.667396] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:02 2019][1696562.672794] .min_vruntime : 30745701.570964 [Wed Feb 6 09:41:02 2019][1696562.678801] .max_vruntime : 0.000001 [Wed Feb 6 09:41:02 2019][1696562.684200] .spread : 0.000000 [Wed Feb 6 09:41:02 2019][1696562.689599] .spread0 : 2620050.266277 [Wed Feb 6 09:41:02 2019][1696562.695519] .nr_spread_over : 0 [Wed Feb 6 09:41:02 2019][1696562.700311] .nr_running : 1 [Wed Feb 6 09:41:02 2019][1696562.705104] .load : 1024 [Wed Feb 6 09:41:02 2019][1696562.710158] .runnable_load_avg : 1023 [Wed Feb 6 09:41:02 2019][1696562.715211] .blocked_load_avg : 0 [Wed Feb 6 09:41:02 2019][1696562.720002] .tg_load_avg : 0 [Wed Feb 6 09:41:02 2019][1696562.724795] .tg_load_contrib : 0 [Wed Feb 6 09:41:02 2019][1696562.729588] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:02 2019][1696562.734380] .tg->runnable_avg : 0 [Wed Feb 6 09:41:02 2019][1696562.739174] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:02 2019][1696562.743966] .throttled : 0 [Wed Feb 6 09:41:02 2019][1696562.748760] .throttle_count : 0 [Wed Feb 6 09:41:02 2019][1696562.753551] .avg->runnable_avg_sum : 46486 [Wed Feb 6 09:41:02 2019][1696562.758693] .avg->runnable_avg_period : 46317 [Wed Feb 6 09:41:02 2019][1696562.763836] [Wed Feb 6 09:41:02 2019][1696562.765514] rt_rq[44]:/ [Wed Feb 6 09:41:02 2019][1696562.768139] .rt_nr_running : 0 [Wed Feb 6 09:41:02 2019][1696562.772930] .rt_throttled : 0 [Wed Feb 6 09:41:02 2019][1696562.777724] .rt_time : 0.000000 [Wed Feb 6 09:41:02 2019][1696562.783123] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:02 2019][1696562.788781] [Wed Feb 6 09:41:02 2019][1696562.790454] runnable tasks: [Wed Feb 6 09:41:02 2019][1696562.793428] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:02 2019][1696562.804373] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:02 2019][1696562.815262] watchdog/44 231 -11.987557 424070 0 0.000000 105680.400951 0.000000 0 / [Wed Feb 6 09:41:02 2019][1696562.826597] migration/44 232 0.000000 2218 0 0.000000 574763.862806 0.000000 0 / [Wed Feb 6 09:41:02 2019][1696562.837932] ksoftirqd/44 233 30455458.147731 9626548 120 0.000000 95982.240736 0.000000 0 / [Wed Feb 6 09:41:02 2019][1696562.849269] kworker/44:0H 235 4994.512381 10 100 0.000000 0.029836 0.000000 0 / [Wed Feb 6 09:41:02 2019][1696562.860607] kdevtmpfs 255 46428.467560 333 120 0.000000 5.201514 0.000000 0 / [Wed Feb 6 09:41:03 2019][1696562.871972] kworker/44:1H 19463 30514551.793590 9607293 100 0.000000 42492.697150 0.000000 0 / [Wed Feb 6 09:41:03 2019][1696562.883366] R mdt00_004 6615 30745918.576744 197419725 120 0.000000 32538880.774323 0.000000 0 / [Wed Feb 6 09:41:03 2019][1696562.894708] mdt_io00_005 6805 26490356.739868 6671178 120 0.000000 258788.582543 0.000000 0 / [Wed Feb 6 09:41:03 2019][1696562.906057] mdt_rdpg00_004 8959 26488984.776532 99805429 120 0.000000 2192474.417147 0.000000 0 / [Wed Feb 6 09:41:03 2019][1696562.917400] ldlm_cb00_003 9162 21397709.291574 4635020 120 0.000000 95527.188381 0.000000 0 / [Wed Feb 6 09:41:03 2019][1696562.928755] mdt_rdpg00_007 23301 26501420.952574 72893584 120 0.000000 1720596.140171 0.000000 0 / [Wed Feb 6 09:41:03 2019][1696562.940099] kworker/44:2 70988 25978853.575239 2262 120 0.000000 23.430749 0.000000 0 / [Wed Feb 6 09:41:03 2019][1696562.951433] kworker/44:0 71199 30624008.813445 209451 120 0.000000 10515.089677 0.000000 0 / [Wed Feb 6 09:41:03 2019][1696562.962850] [Wed Feb 6 09:41:03 2019][1696562.964529] cpu#45, 1996.199 MHz [Wed Feb 6 09:41:03 2019][1696562.967944] .nr_running : 0 [Wed Feb 6 09:41:03 2019][1696562.972747] .load : 0 [Wed Feb 6 09:41:03 2019][1696562.977548] .nr_switches : 3287318762 [Wed Feb 6 09:41:03 2019][1696562.983130] .nr_load_updates : 375083039 [Wed Feb 6 09:41:03 2019][1696562.988623] .nr_uninterruptible : 101892 [Wed Feb 6 09:41:03 2019][1696562.993857] .next_balance : 5991.186878 [Wed Feb 6 09:41:03 2019][1696562.999524] .curr->pid : 0 [Wed Feb 6 09:41:03 2019][1696563.004327] .clock : 1696563004.048000 [Wed Feb 6 09:41:03 2019][1696563.010513] .cpu_load[0] : 0 [Wed Feb 6 09:41:03 2019][1696563.015306] .cpu_load[1] : 0 [Wed Feb 6 09:41:03 2019][1696563.020107] .cpu_load[2] : 0 [Wed Feb 6 09:41:03 2019][1696563.024910] .cpu_load[3] : 0 [Wed Feb 6 09:41:03 2019][1696563.029709] .cpu_load[4] : 0 [Wed Feb 6 09:41:03 2019][1696563.034508] .avg_idle : 10000000 [Wed Feb 6 09:41:03 2019][1696563.039914] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:03 2019][1696563.045235] [Wed Feb 6 09:41:03 2019][1696563.046909] cfs_rq[45]:/ [Wed Feb 6 09:41:03 2019][1696563.049622] .exec_clock : 0.000000 [Wed Feb 6 09:41:03 2019][1696563.055020] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:03 2019][1696563.060420] .min_vruntime : 19100047.196184 [Wed Feb 6 09:41:03 2019][1696563.066426] .max_vruntime : 0.000001 [Wed Feb 6 09:41:03 2019][1696563.071826] .spread : 0.000000 [Wed Feb 6 09:41:03 2019][1696563.077225] .spread0 : -9025604.480390 [Wed Feb 6 09:41:03 2019][1696563.083231] .nr_spread_over : 0 [Wed Feb 6 09:41:03 2019][1696563.088025] .nr_running : 0 [Wed Feb 6 09:41:03 2019][1696563.092816] .load : 0 [Wed Feb 6 09:41:03 2019][1696563.097611] .runnable_load_avg : 0 [Wed Feb 6 09:41:03 2019][1696563.102404] .blocked_load_avg : 0 [Wed Feb 6 09:41:03 2019][1696563.107196] .tg_load_avg : 0 [Wed Feb 6 09:41:03 2019][1696563.111988] .tg_load_contrib : 0 [Wed Feb 6 09:41:03 2019][1696563.116782] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:03 2019][1696563.121573] .tg->runnable_avg : 0 [Wed Feb 6 09:41:03 2019][1696563.126366] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:03 2019][1696563.131161] .throttled : 0 [Wed Feb 6 09:41:03 2019][1696563.135952] .throttle_count : 0 [Wed Feb 6 09:41:03 2019][1696563.140746] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:41:03 2019][1696563.145538] .avg->runnable_avg_period : 47132 [Wed Feb 6 09:41:03 2019][1696563.150681] [Wed Feb 6 09:41:03 2019][1696563.152349] rt_rq[45]:/ [Wed Feb 6 09:41:03 2019][1696563.154975] .rt_nr_running : 0 [Wed Feb 6 09:41:03 2019][1696563.159768] .rt_throttled : 0 [Wed Feb 6 09:41:03 2019][1696563.164561] .rt_time : 0.000000 [Wed Feb 6 09:41:03 2019][1696563.169961] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:03 2019][1696563.175620] [Wed Feb 6 09:41:03 2019][1696563.177294] runnable tasks: [Wed Feb 6 09:41:03 2019][1696563.180267] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:03 2019][1696563.191211] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:03 2019][1696563.202102] watchdog/45 236 -11.986324 424089 0 0.000000 167950.970388 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.213435] migration/45 237 0.000000 10935 0 0.000000 451469.585305 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.224771] ksoftirqd/45 238 19099987.373684 7108231 120 0.000000 66404.140323 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.236106] kworker/45:0H 240 9580.250767 10 100 0.000000 0.037572 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.247450] ata_sff 4059 892.912781 2 100 0.000000 0.018105 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.258805] gssproxy 17576 5035.832395 1 120 0.000000 0.010029 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.270152] kworker/45:1H 54720 19099986.790084 4690786 100 0.000000 19739.205284 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.281490] ptlrpc_hr01_004 5846 19100035.209529 121178233 120 0.000000 1150647.233540 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.292845] ll_mgs_0000 6046 16587092.400359 1665990 120 0.000000 40731.963270 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.304188] osp-syn-13-0 6250 19099120.505986 782419 120 0.000000 20693.892441 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.315538] osp-syn-4-2 6486 19099986.860010 444364 120 0.000000 14456.498374 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.326888] mdt_io01_005 6785 16276739.984715 6256976 120 0.000000 227043.186932 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.338220] mdt_io01_006 6788 16276620.910384 6869051 120 0.000000 262436.200215 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.349562] mdt_rdpg01_003 7112 19100037.857064 11367324 120 0.000000 497159.637824 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.360905] mdt_io01_015 8953 16276618.178130 7404074 120 0.000000 270172.420977 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.372237] mdt_io01_017 8955 16276618.237720 1840302 120 0.000000 70181.650079 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.383579] mdt01_022 8997 16578346.966552 88596287 120 0.000000 3434694.508234 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.394915] mdt_rdpg01_008 9150 19100046.579351 13054113 120 0.000000 569971.393157 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.406248] mdt_io01_023 9189 16276620.977663 5175700 120 0.000000 192059.365820 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.417582] mdt_io01_028 9839 19099986.819463 5339365 120 0.000000 205237.187097 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.428924] mdt_io01_035 15117 16577317.458098 5967250 120 0.000000 216048.095419 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.440256] kworker/45:2 18925 19100035.222143 544580 120 0.000000 14686.252815 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.451593] ldlm_cn01_011 19972 19060764.074866 17720465 120 0.000000 608999.673476 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.462931] mdt_io01_051 20872 16276620.678263 105148 120 0.000000 16144.458569 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.474262] mdt_io01_052 20873 16576642.716322 6433532 120 0.000000 242059.907134 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.485600] mdt_io01_056 20877 16276620.228631 6285230 120 0.000000 235171.021512 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.496942] ldlm_cn01_021 27821 19099141.474339 18767993 120 0.000000 518060.321233 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.508275] ldlm_cn01_028 28329 19100035.216023 14205634 120 0.000000 374011.832199 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.519616] kworker/45:1 131022 13800669.585667 2 120 0.000000 0.010790 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.531047] mdt01_059 15258 19100035.201294 1482180 120 0.000000 206796.917686 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.542392] mdt_io01_065 15433 16276620.424146 574 120 0.000000 21.693396 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.553731] mdt_io01_066 15434 16276742.999492 4436 120 0.000000 175.166771 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.565067] mdt_io01_067 15435 16276746.787858 818 120 0.000000 30.441679 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.576404] mdt_io01_070 15438 16276620.165370 476 120 0.000000 17.709093 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.587765] mdt_io01_095 27718 19100035.202807 9446 120 0.000000 1964.883635 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.599100] mdt_io01_096 27720 16276747.261918 824 120 0.000000 36.250881 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.610438] mdt_io01_097 27721 16577359.547068 20671 120 0.000000 682.010087 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.621774] mdt_io01_098 27723 16276742.036454 793 120 0.000000 32.723364 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.633113] mdt_io01_100 27727 16577317.418192 611 120 0.000000 23.504805 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.644451] mdt_io01_110 27761 16276739.976230 604 120 0.000000 28.287743 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.655806] crond 30469 19100047.196184 5 120 0.000000 3.058012 0.000000 1 / [Wed Feb 6 09:41:03 2019][1696563.667154] [Wed Feb 6 09:41:03 2019][1696563.668833] cpu#46, 1996.199 MHz [Wed Feb 6 09:41:03 2019][1696563.672251] .nr_running : 0 [Wed Feb 6 09:41:03 2019][1696563.677051] .load : 0 [Wed Feb 6 09:41:03 2019][1696563.681854] .nr_switches : 727418526 [Wed Feb 6 09:41:03 2019][1696563.687348] .nr_load_updates : 254916634 [Wed Feb 6 09:41:03 2019][1696563.692843] .nr_uninterruptible : 126473 [Wed Feb 6 09:41:03 2019][1696563.698081] .next_balance : 5991.187582 [Wed Feb 6 09:41:03 2019][1696563.703752] .curr->pid : 0 [Wed Feb 6 09:41:03 2019][1696563.708555] .clock : 1696563708.065896 [Wed Feb 6 09:41:03 2019][1696563.714741] .cpu_load[0] : 0 [Wed Feb 6 09:41:03 2019][1696563.719537] .cpu_load[1] : 0 [Wed Feb 6 09:41:03 2019][1696563.724335] .cpu_load[2] : 0 [Wed Feb 6 09:41:03 2019][1696563.729138] .cpu_load[3] : 0 [Wed Feb 6 09:41:03 2019][1696563.733939] .cpu_load[4] : 0 [Wed Feb 6 09:41:03 2019][1696563.738740] .avg_idle : 9799537 [Wed Feb 6 09:41:03 2019][1696563.744061] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:03 2019][1696563.749382] [Wed Feb 6 09:41:03 2019][1696563.751060] cfs_rq[46]:/ [Wed Feb 6 09:41:03 2019][1696563.753771] .exec_clock : 0.000000 [Wed Feb 6 09:41:03 2019][1696563.759172] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:03 2019][1696563.764569] .min_vruntime : 10327465.313200 [Wed Feb 6 09:41:03 2019][1696563.770577] .max_vruntime : 0.000001 [Wed Feb 6 09:41:03 2019][1696563.775976] .spread : 0.000000 [Wed Feb 6 09:41:03 2019][1696563.781374] .spread0 : -17798186.363374 [Wed Feb 6 09:41:03 2019][1696563.787467] .nr_spread_over : 0 [Wed Feb 6 09:41:03 2019][1696563.792261] .nr_running : 0 [Wed Feb 6 09:41:03 2019][1696563.797054] .load : 0 [Wed Feb 6 09:41:03 2019][1696563.801847] .runnable_load_avg : 0 [Wed Feb 6 09:41:03 2019][1696563.806638] .blocked_load_avg : 0 [Wed Feb 6 09:41:03 2019][1696563.811433] .tg_load_avg : 0 [Wed Feb 6 09:41:03 2019][1696563.816224] .tg_load_contrib : 0 [Wed Feb 6 09:41:03 2019][1696563.821017] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:03 2019][1696563.825811] .tg->runnable_avg : 0 [Wed Feb 6 09:41:03 2019][1696563.830603] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:03 2019][1696563.835396] .throttled : 0 [Wed Feb 6 09:41:03 2019][1696563.840189] .throttle_count : 0 [Wed Feb 6 09:41:03 2019][1696563.844983] .avg->runnable_avg_sum : 210 [Wed Feb 6 09:41:03 2019][1696563.849947] .avg->runnable_avg_period : 47078 [Wed Feb 6 09:41:03 2019][1696563.855090] [Wed Feb 6 09:41:03 2019][1696563.856760] rt_rq[46]:/ [Wed Feb 6 09:41:03 2019][1696563.859386] .rt_nr_running : 0 [Wed Feb 6 09:41:03 2019][1696563.864179] .rt_throttled : 0 [Wed Feb 6 09:41:03 2019][1696563.868970] .rt_time : 0.000000 [Wed Feb 6 09:41:04 2019][1696563.874371] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:04 2019][1696563.880029] [Wed Feb 6 09:41:04 2019][1696563.881704] runnable tasks: [Wed Feb 6 09:41:04 2019][1696563.884674] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:04 2019][1696563.895622] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:04 2019][1696563.906513] watchdog/46 241 -11.987235 424088 0 0.000000 77814.595265 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696563.917843] migration/46 242 0.000000 2719 0 0.000000 606922.555546 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696563.929180] ksoftirqd/46 243 10327346.931440 8342872 120 0.000000 81560.078530 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696563.940516] kworker/46:0H 245 8606.028986 10 100 0.000000 0.032814 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696563.951858] scsi_tmf_1 2384 770.660378 2 100 0.000000 0.013606 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696563.963190] kworker/46:1 7282 10327452.936074 605623 120 0.000000 10720.196014 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696563.974548] reader#4 19102 5461.006425 1 120 0.000000 0.018806 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696563.985884] kworker/46:1H 57542 10327346.736756 5138496 100 0.000000 20115.909588 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696563.997220] ptlrpc_hr02_002 5850 10327446.654729 83753771 120 0.000000 775657.555613 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.008552] ptlrpc_hr02_004 5853 10327409.258007 83791791 120 0.000000 773393.237183 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.019888] ptlrpc_hr02_005 5854 10327430.044968 83782642 120 0.000000 771353.694890 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.031236] ptlrpcd_02_04 5924 10327449.668819 13154469 120 0.000000 187763.080930 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.042571] ptlrpcd_02_10 5930 10327451.637469 13228204 120 0.000000 190720.978652 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.053912] ldlm_cn02_000 6035 10327358.985004 27496447 120 0.000000 877008.507914 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.065258] mdt_rdpg02_000 6183 10327463.322828 6811984 120 0.000000 324084.035848 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.076611] osp-syn-26-0 6262 10326982.645702 775156 120 0.000000 20908.558266 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.087954] osp-pre-45-0 6309 10327449.918792 247859 120 0.000000 4641.912865 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.099288] osp-syn-34-0 6342 10327409.246737 775874 120 0.000000 21070.877646 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.110625] osp-pre-31-2 6455 10327449.134996 235005 120 0.000000 3514.980937 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.121965] osp-pre-38-2 6507 10327449.626787 234946 120 0.000000 6394.251486 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.133310] mdt02_004 6687 10327367.977797 86889353 120 0.000000 4089916.437931 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.144671] mdt_io02_018 9031 9557087.654234 7463174 120 0.000000 302996.158720 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.156010] ldlm_cb02_003 9046 10326881.794394 9083121 120 0.000000 155832.236993 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.167349] mdt02_022 9184 10327367.970401 95792585 120 0.000000 4161160.390685 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.178686] mdt_io02_023 11807 10082861.986741 4373452 120 0.000000 179659.853376 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.190022] mdt_out02_004 15751 9976975.691758 8626129 120 0.000000 203624.872705 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.201370] ldlm_cn02_015 23318 10314297.242478 12872619 120 0.000000 466189.487060 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.212710] ldlm_cn02_017 23320 10327447.072775 15803763 120 0.000000 502621.604291 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.224049] ldlm_cn02_022 28933 10327148.291019 8228477 120 0.000000 406975.818288 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.235382] mdt02_025 48636 10327464.096448 34980092 120 0.000000 1613404.124979 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.246719] mdt02_026 48676 10327453.846795 29422465 120 0.000000 1811179.595128 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.258060] mdt02_029 15181 10327458.904692 9527914 120 0.000000 11337408.039680 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.269412] mdt02_048 15510 10327368.001302 12339614 120 0.000000 988898.454012 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.280763] mdt_io02_044 27699 10327367.984420 13784 120 0.000000 2471.526449 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.292103] mdt_io02_065 27747 9972273.244399 3045 120 0.000000 1828.606060 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.303439] mdt_io02_086 27777 10082861.986500 187 120 0.000000 7.220119 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.314772] ldlm_cn02_026 28520 10314290.971782 9056 120 0.000000 16300.462684 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.326106] kworker/u264:0 28537 9855487.370624 125 100 0.000000 2339.226498 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.337445] ldlm_cn02_029 28945 10314291.238212 2874 120 0.000000 4996.680301 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.348779] ldlm_cn02_030 28946 10327447.058809 8760 120 0.000000 6546.156119 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.360115] kworker/46:0 28975 9976975.671871 13 120 0.000000 0.070917 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.371453] ll_mgs_0022 28990 10327466.118009 50889 120 0.000000 1148.434004 0.000000 2 / [Wed Feb 6 09:41:04 2019][1696564.382816] [Wed Feb 6 09:41:04 2019][1696564.384494] cpu#47, 1996.199 MHz [Wed Feb 6 09:41:04 2019][1696564.387910] .nr_running : 0 [Wed Feb 6 09:41:04 2019][1696564.392710] .load : 0 [Wed Feb 6 09:41:04 2019][1696564.397513] .nr_switches : 423768473 [Wed Feb 6 09:41:04 2019][1696564.403009] .nr_load_updates : 164820054 [Wed Feb 6 09:41:04 2019][1696564.408501] .nr_uninterruptible : 29386 [Wed Feb 6 09:41:04 2019][1696564.413650] .next_balance : 5991.188298 [Wed Feb 6 09:41:04 2019][1696564.419320] .curr->pid : 0 [Wed Feb 6 09:41:04 2019][1696564.424124] .clock : 1696564424.083969 [Wed Feb 6 09:41:04 2019][1696564.430305] .cpu_load[0] : 0 [Wed Feb 6 09:41:04 2019][1696564.435113] .cpu_load[1] : 0 [Wed Feb 6 09:41:04 2019][1696564.439911] .cpu_load[2] : 0 [Wed Feb 6 09:41:04 2019][1696564.444714] .cpu_load[3] : 0 [Wed Feb 6 09:41:04 2019][1696564.449514] .cpu_load[4] : 0 [Wed Feb 6 09:41:04 2019][1696564.454316] .avg_idle : 8761938 [Wed Feb 6 09:41:04 2019][1696564.459635] .max_idle_balance_cost : 5000000 [Wed Feb 6 09:41:04 2019][1696564.464958] [Wed Feb 6 09:41:04 2019][1696564.466631] cfs_rq[47]:/ [Wed Feb 6 09:41:04 2019][1696564.469345] .exec_clock : 0.000000 [Wed Feb 6 09:41:04 2019][1696564.474743] .MIN_vruntime : 0.000001 [Wed Feb 6 09:41:04 2019][1696564.480145] .min_vruntime : 9395632.129801 [Wed Feb 6 09:41:04 2019][1696564.486063] .max_vruntime : 0.000001 [Wed Feb 6 09:41:04 2019][1696564.491463] .spread : 0.000000 [Wed Feb 6 09:41:04 2019][1696564.496862] .spread0 : -18730019.546773 [Wed Feb 6 09:41:04 2019][1696564.502954] .nr_spread_over : 0 [Wed Feb 6 09:41:04 2019][1696564.507747] .nr_running : 0 [Wed Feb 6 09:41:04 2019][1696564.512540] .load : 0 [Wed Feb 6 09:41:04 2019][1696564.517334] .runnable_load_avg : 0 [Wed Feb 6 09:41:04 2019][1696564.522125] .blocked_load_avg : 0 [Wed Feb 6 09:41:04 2019][1696564.526918] .tg_load_avg : 0 [Wed Feb 6 09:41:04 2019][1696564.531710] .tg_load_contrib : 0 [Wed Feb 6 09:41:04 2019][1696564.536504] .tg_runnable_contrib : 0 [Wed Feb 6 09:41:04 2019][1696564.541298] .tg->runnable_avg : 0 [Wed Feb 6 09:41:04 2019][1696564.546090] .tg->cfs_bandwidth.timer_active: 0 [Wed Feb 6 09:41:04 2019][1696564.550882] .throttled : 0 [Wed Feb 6 09:41:04 2019][1696564.555674] .throttle_count : 0 [Wed Feb 6 09:41:04 2019][1696564.560468] .avg->runnable_avg_sum : 0 [Wed Feb 6 09:41:04 2019][1696564.565260] .avg->runnable_avg_period : 47257 [Wed Feb 6 09:41:04 2019][1696564.570406] [Wed Feb 6 09:41:04 2019][1696564.572081] rt_rq[47]:/ [Wed Feb 6 09:41:04 2019][1696564.574706] .rt_nr_running : 0 [Wed Feb 6 09:41:04 2019][1696564.579499] .rt_throttled : 0 [Wed Feb 6 09:41:04 2019][1696564.584294] .rt_time : 0.000000 [Wed Feb 6 09:41:04 2019][1696564.589692] .rt_runtime : 1000.000000 [Wed Feb 6 09:41:04 2019][1696564.595352] [Wed Feb 6 09:41:04 2019][1696564.597023] runnable tasks: [Wed Feb 6 09:41:04 2019][1696564.599997] task PID tree-key switches prio wait-time sum-exec sum-sleep [Wed Feb 6 09:41:04 2019][1696564.610945] ---------------------------------------------------------------------------------------------------------- [Wed Feb 6 09:41:04 2019][1696564.621832] watchdog/47 246 -11.985712 424086 0 0.000000 190055.554794 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.633166] migration/47 247 0.000000 3446 0 0.000000 411159.864594 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.644502] ksoftirqd/47 248 9140455.999884 1445125 120 0.000000 16684.871522 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.655839] kworker/47:0H 250 9330.596450 10 100 0.000000 0.037822 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.667188] ccp-1-q2 10459 2459.578382 2 120 0.000000 0.009318 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.678524] multipathd 13033 0.000000 48 0 0.000000 12.264158 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.689871] kworker/47:1H 27748 7574200.352055 1432310 100 0.000000 4824.430875 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.701229] ptlrpcd_03_01 5933 9395620.318869 13125267 120 0.000000 199677.273422 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.712578] mdt_out03_001 6202 7535581.147892 157617 120 0.000000 4937.091174 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.723921] osp-pre-2-0 6271 9395620.147735 247634 120 0.000000 6380.632789 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.735263] osp-pre-11-2 6449 9395620.148446 234864 120 0.000000 7187.660991 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.746603] osp-pre-20-2 6493 9395620.147916 235081 120 0.000000 4554.709115 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.757935] osp-pre-6-2 6501 9395620.159478 234895 120 0.000000 3353.083079 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.769272] osp-pre-34-2 6533 9395620.157013 234992 120 0.000000 4681.334647 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.780613] ldlm_cn03_004 6770 9395620.854334 8422022 120 0.000000 317764.396540 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.791955] mdt03_008 6845 9395632.129801 35487186 120 0.000000 1565116.342705 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.803333] kworker/47:2 42267 9395608.236038 327135 120 0.000000 6644.445555 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.814698] mdt_io03_022 17257 7570416.994367 227246 120 0.000000 9439.261130 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.826030] mdt_out03_003 17417 9395620.084534 183652 120 0.000000 4888.830200 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.837390] kworker/47:0 28887 7486782.234620 17 120 0.000000 2.254712 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.848741] l_getidentity 30461 9395620.204048 2 120 0.000000 0.893288 0.000000 3 / [Wed Feb 6 09:41:04 2019][1696564.860091] [Wed Feb 6 09:41:04 2019][1696564.862098] INFO: rcu_sched self-detected stall on CPU [Wed Feb 6 09:41:04 2019][1696564.862126] INFO: rcu_sched detected stalls on CPUs/tasks: { 14} (detected by 6, t=0 jiffies, g=134912029, c=134912028, q=358366) [Wed Feb 6 09:41:05 2019][1696564.862127] Task dump for CPU 0: [Wed Feb 6 09:41:05 2019][1696564.862131] swapper/0 R running task 0 0 0 0x00000000 [Wed Feb 6 09:41:05 2019][1696564.862132] Call Trace: [Wed Feb 6 09:41:05 2019][1696564.862143] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:05 2019][1696564.862149] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:05 2019][1696564.862152] [] ? native_safe_halt+0x6/0x10 [Wed Feb 6 09:41:05 2019][1696564.862155] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862160] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862164] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:05 2019][1696564.862167] [] ? rest_init+0x77/0x80 [Wed Feb 6 09:41:05 2019][1696564.862172] [] ? start_kernel+0x44b/0x46c [Wed Feb 6 09:41:05 2019][1696564.862174] [] ? repair_env_string+0x5c/0x5c [Wed Feb 6 09:41:05 2019][1696564.862177] [] ? early_idt_handler_array+0x120/0x120 [Wed Feb 6 09:41:05 2019][1696564.862179] [] ? x86_64_start_reservations+0x24/0x26 [Wed Feb 6 09:41:05 2019][1696564.862181] [] ? x86_64_start_kernel+0x154/0x177 [Wed Feb 6 09:41:05 2019][1696564.862185] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:05 2019][1696564.862185] Task dump for CPU 1: [Wed Feb 6 09:41:05 2019][1696564.862189] swapper/1 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:05 2019][1696564.862189] Call Trace: [Wed Feb 6 09:41:05 2019][1696564.862192] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:05 2019][1696564.862194] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:05 2019][1696564.862197] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862199] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862202] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:05 2019][1696564.862206] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:05 2019][1696564.862208] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:05 2019][1696564.862209] Task dump for CPU 2: [Wed Feb 6 09:41:05 2019][1696564.862212] ldlm_bl_25 R running task 0 15988 2 0x00000080 [Wed Feb 6 09:41:05 2019][1696564.862212] Call Trace: [Wed Feb 6 09:41:05 2019][1696564.862274] [] ? ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:41:05 2019][1696564.862310] [] ? ldlm_cancel_locks_for_export_cb+0x3f/0x50 [ptlrpc] [Wed Feb 6 09:41:05 2019][1696564.862327] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [Wed Feb 6 09:41:05 2019][1696564.862360] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:41:05 2019][1696564.862395] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:41:05 2019][1696564.862407] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [Wed Feb 6 09:41:05 2019][1696564.862441] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [Wed Feb 6 09:41:05 2019][1696564.862481] [] ? ldlm_bl_thread_main+0x4c8/0x700 [ptlrpc] [Wed Feb 6 09:41:05 2019][1696564.862486] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:41:05 2019][1696564.862525] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:41:05 2019][1696564.862528] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:41:05 2019][1696564.862531] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:05 2019][1696564.862535] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:41:05 2019][1696564.862537] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:05 2019][1696564.862538] Task dump for CPU 3: [Wed Feb 6 09:41:05 2019][1696564.862541] swapper/3 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:05 2019][1696564.862541] Call Trace: [Wed Feb 6 09:41:05 2019][1696564.862544] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:05 2019][1696564.862546] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:05 2019][1696564.862548] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862550] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862553] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:05 2019][1696564.862555] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:05 2019][1696564.862558] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:05 2019][1696564.862559] Task dump for CPU 4: [Wed Feb 6 09:41:05 2019][1696564.862562] swapper/4 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:05 2019][1696564.862562] Call Trace: [Wed Feb 6 09:41:05 2019][1696564.862565] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:05 2019][1696564.862567] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:05 2019][1696564.862570] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862573] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862575] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:05 2019][1696564.862578] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:05 2019][1696564.862580] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:05 2019][1696564.862581] Task dump for CPU 5: [Wed Feb 6 09:41:05 2019][1696564.862583] swapper/5 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:05 2019][1696564.862584] Call Trace: [Wed Feb 6 09:41:05 2019][1696564.862588] [] ? tick_nohz_stop_sched_tick+0x7c/0x380 [Wed Feb 6 09:41:05 2019][1696564.862590] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:05 2019][1696564.862593] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862595] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862597] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:05 2019][1696564.862600] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:05 2019][1696564.862603] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:05 2019][1696564.862604] Task dump for CPU 6: [Wed Feb 6 09:41:05 2019][1696564.862606] swapper/6 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:05 2019][1696564.862606] Call Trace: [Wed Feb 6 09:41:05 2019][1696564.862611] [] sched_show_task+0xa8/0x110 [Wed Feb 6 09:41:05 2019][1696564.862613] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 09:41:05 2019][1696564.862615] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 09:41:05 2019][1696564.862618] [] rcu_check_callbacks+0x6f2/0x730 [Wed Feb 6 09:41:05 2019][1696564.862621] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 09:41:05 2019][1696564.862624] [] update_process_times+0x46/0x80 [Wed Feb 6 09:41:05 2019][1696564.862627] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 09:41:05 2019][1696564.862629] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 09:41:05 2019][1696564.862633] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 09:41:05 2019][1696564.862637] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 09:41:05 2019][1696564.862641] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 09:41:05 2019][1696564.862644] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 09:41:05 2019][1696564.862648] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 09:41:05 2019][1696564.862652] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:05 2019][1696564.862654] [] ? native_safe_halt+0x6/0x10 [Wed Feb 6 09:41:05 2019][1696564.862657] [] default_idle+0x1e/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862659] [] arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862662] [] cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:05 2019][1696564.862664] [] start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:05 2019][1696564.862666] [] start_cpu+0x5/0x14 [Wed Feb 6 09:41:05 2019][1696564.862668] Task dump for CPU 7: [Wed Feb 6 09:41:05 2019][1696564.862670] swapper/7 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:05 2019][1696564.862671] Call Trace: [Wed Feb 6 09:41:05 2019][1696564.862674] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:05 2019][1696564.862676] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:05 2019][1696564.862679] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862681] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862683] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:05 2019][1696564.862685] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:05 2019][1696564.862687] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:05 2019][1696564.862688] Task dump for CPU 8: [Wed Feb 6 09:41:05 2019][1696564.862691] systemd-journal R running task 0 7874 1 0x00000000 [Wed Feb 6 09:41:05 2019][1696564.862691] Call Trace: [Wed Feb 6 09:41:05 2019][1696564.862696] [] ? __vma_link_rb+0xb8/0xe0 [Wed Feb 6 09:41:05 2019][1696564.862703] [] ? string.isra.7+0x3b/0xf0 [Wed Feb 6 09:41:05 2019][1696564.862706] [] ? __do_page_fault+0x228/0x500 [Wed Feb 6 09:41:05 2019][1696564.862709] [] ? free_pages_and_swap_cache+0xad/0xd0 [Wed Feb 6 09:41:05 2019][1696564.862713] [] ? tlb_flush_mmu.part.76+0x8c/0xe0 [Wed Feb 6 09:41:05 2019][1696564.862717] [] ? filename_lookup+0x2b/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862719] [] ? putname+0x3d/0x60 [Wed Feb 6 09:41:05 2019][1696564.862722] [] ? do_filp_open+0x4d/0xb0 [Wed Feb 6 09:41:05 2019][1696564.862726] [] ? __alloc_fd+0x47/0x170 [Wed Feb 6 09:41:05 2019][1696564.862728] [] ? putname+0x3d/0x60 [Wed Feb 6 09:41:05 2019][1696564.862733] [] ? do_sys_open+0x1ab/0x240 [Wed Feb 6 09:41:05 2019][1696564.862736] [] ? SyS_open+0x1e/0x20 [Wed Feb 6 09:41:05 2019][1696564.862739] [] ? page_fault+0x28/0x30 [Wed Feb 6 09:41:05 2019][1696564.862741] Task dump for CPU 9: [Wed Feb 6 09:41:05 2019][1696564.862743] swapper/9 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:05 2019][1696564.862743] Call Trace: [Wed Feb 6 09:41:05 2019][1696564.862746] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:05 2019][1696564.862748] [] ? native_safe_halt+0x6/0x10 [Wed Feb 6 09:41:05 2019][1696564.862751] [] ? update_ts_time_stats+0x58/0x90 [Wed Feb 6 09:41:05 2019][1696564.862754] [] ? schedule_preempt_disabled+0x29/0x70 [Wed Feb 6 09:41:05 2019][1696564.862757] [] ? cpu_startup_entry+0x18a/0x1e0 [Wed Feb 6 09:41:05 2019][1696564.862760] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:05 2019][1696564.862763] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:05 2019][1696564.862764] Task dump for CPU 10: [Wed Feb 6 09:41:05 2019][1696564.862766] swapper/10 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:05 2019][1696564.862767] Call Trace: [Wed Feb 6 09:41:05 2019][1696564.862770] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:05 2019][1696564.862772] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:05 2019][1696564.862775] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862777] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862779] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:05 2019][1696564.862783] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:05 2019][1696564.862785] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:05 2019][1696564.862786] Task dump for CPU 11: [Wed Feb 6 09:41:05 2019][1696564.862788] slapd R running task 0 84636 1 0x00000080 [Wed Feb 6 09:41:05 2019][1696564.862789] Call Trace: [Wed Feb 6 09:41:05 2019][1696564.862792] [] ? do_page_mkwrite+0x8a/0xe0 [Wed Feb 6 09:41:05 2019][1696564.862795] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:41:05 2019][1696564.862799] [] ? unlock_page+0x2b/0x30 [Wed Feb 6 09:41:05 2019][1696564.862800] [] ? do_wp_page+0x65e/0x720 [Wed Feb 6 09:41:05 2019][1696564.862802] [] ? update_time+0x89/0xd0 [Wed Feb 6 09:41:05 2019][1696564.862805] [] ? __sb_start_write+0x58/0x110 [Wed Feb 6 09:41:05 2019][1696564.862807] [] ? file_update_time+0xaa/0xf0 [Wed Feb 6 09:41:05 2019][1696564.862809] [] ? handle_pte_fault+0x37d/0xd10 [Wed Feb 6 09:41:05 2019][1696564.862811] [] ? handle_mm_fault+0x39d/0x9b0 [Wed Feb 6 09:41:05 2019][1696564.862813] [] ? __do_page_fault+0x228/0x500 [Wed Feb 6 09:41:05 2019][1696564.862815] [] ? do_page_fault+0x35/0x90 [Wed Feb 6 09:41:05 2019][1696564.862819] [] ? page_fault+0x28/0x30 [Wed Feb 6 09:41:05 2019][1696564.862820] Task dump for CPU 12: [Wed Feb 6 09:41:05 2019][1696564.862823] swapper/12 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:05 2019][1696564.862824] Call Trace: [Wed Feb 6 09:41:05 2019][1696564.862827] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:05 2019][1696564.862829] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:05 2019][1696564.862832] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862834] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862836] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:05 2019][1696564.862839] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:05 2019][1696564.862841] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:05 2019][1696564.862842] Task dump for CPU 13: [Wed Feb 6 09:41:05 2019][1696564.862844] swapper/13 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:05 2019][1696564.862844] Call Trace: [Wed Feb 6 09:41:05 2019][1696564.862847] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:05 2019][1696564.862849] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:05 2019][1696564.862851] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:05 2019][1696564.862854] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.862856] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.862858] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.862860] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.862861] Task dump for CPU 14: [Wed Feb 6 09:41:06 2019][1696564.862863] swapper/14 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:06 2019][1696564.862864] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.862866] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:06 2019][1696564.862869] [] ? native_safe_halt+0x6/0x10 [Wed Feb 6 09:41:06 2019][1696564.862872] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:06 2019][1696564.862874] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.862876] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.862880] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.862882] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.862883] Task dump for CPU 15: [Wed Feb 6 09:41:06 2019][1696564.862886] swapper/15 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:06 2019][1696564.862886] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.862890] [] ? tick_nohz_stop_sched_tick+0x7c/0x380 [Wed Feb 6 09:41:06 2019][1696564.862892] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:06 2019][1696564.862896] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:06 2019][1696564.862898] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.862901] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.862905] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.862907] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.862908] Task dump for CPU 16: [Wed Feb 6 09:41:06 2019][1696564.862911] swapper/16 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:06 2019][1696564.862911] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.862914] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:06 2019][1696564.862916] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:06 2019][1696564.862919] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:06 2019][1696564.862921] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.862924] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.862926] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.862928] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.862929] Task dump for CPU 17: [Wed Feb 6 09:41:06 2019][1696564.862931] swapper/17 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:06 2019][1696564.862932] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.862935] [] ? tick_nohz_stop_sched_tick+0x7c/0x380 [Wed Feb 6 09:41:06 2019][1696564.862937] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:06 2019][1696564.862940] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:06 2019][1696564.862942] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.862944] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.862948] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.862950] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.862952] Task dump for CPU 18: [Wed Feb 6 09:41:06 2019][1696564.862954] swapper/18 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:06 2019][1696564.862954] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.862957] [] ? tick_nohz_stop_sched_tick+0x7c/0x380 [Wed Feb 6 09:41:06 2019][1696564.862960] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:06 2019][1696564.862963] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:06 2019][1696564.862965] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.862967] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.862970] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.862971] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.862972] Task dump for CPU 19: [Wed Feb 6 09:41:06 2019][1696564.862975] swapper/19 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:06 2019][1696564.862975] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.862979] [] ? tick_nohz_stop_sched_tick+0x7c/0x380 [Wed Feb 6 09:41:06 2019][1696564.862982] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:06 2019][1696564.862985] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:06 2019][1696564.862987] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.862989] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.862993] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.862994] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.862995] Task dump for CPU 20: [Wed Feb 6 09:41:06 2019][1696564.862998] swapper/20 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:06 2019][1696564.862998] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.863002] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:06 2019][1696564.863004] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:06 2019][1696564.863007] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863010] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863012] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.863015] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.863017] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.863018] Task dump for CPU 21: [Wed Feb 6 09:41:06 2019][1696564.863021] ldlm_bl_03 R running task 0 7089 2 0x00000080 [Wed Feb 6 09:41:06 2019][1696564.863021] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.863056] [] ? ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:41:06 2019][1696564.863089] [] ? ldlm_cancel_locks_for_export_cb+0x3f/0x50 [ptlrpc] [Wed Feb 6 09:41:06 2019][1696564.863100] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [Wed Feb 6 09:41:06 2019][1696564.863133] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:41:06 2019][1696564.863167] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:41:06 2019][1696564.863178] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [Wed Feb 6 09:41:06 2019][1696564.863212] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [Wed Feb 6 09:41:06 2019][1696564.863248] [] ? ldlm_bl_thread_main+0x4c8/0x700 [ptlrpc] [Wed Feb 6 09:41:06 2019][1696564.863252] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:41:06 2019][1696564.863288] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:41:06 2019][1696564.863291] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:41:06 2019][1696564.863294] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:06 2019][1696564.863298] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:41:06 2019][1696564.863300] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:06 2019][1696564.863302] Task dump for CPU 22: [Wed Feb 6 09:41:06 2019][1696564.863304] swapper/22 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:06 2019][1696564.863304] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.863307] [] ? tick_nohz_stop_sched_tick+0x7c/0x380 [Wed Feb 6 09:41:06 2019][1696564.863309] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:06 2019][1696564.863313] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863315] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863317] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.863320] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.863322] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.863323] Task dump for CPU 23: [Wed Feb 6 09:41:06 2019][1696564.863325] swapper/23 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:06 2019][1696564.863326] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.863329] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:06 2019][1696564.863331] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:06 2019][1696564.863334] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863336] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863339] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.863341] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.863344] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.863345] Task dump for CPU 24: [Wed Feb 6 09:41:06 2019][1696564.863347] swapper/24 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:06 2019][1696564.863347] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.863350] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:06 2019][1696564.863354] [] ? native_safe_halt+0x6/0x10 [Wed Feb 6 09:41:06 2019][1696564.863356] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863358] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863361] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.863364] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.863366] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.863367] Task dump for CPU 25: [Wed Feb 6 09:41:06 2019][1696564.863369] swapper/25 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:06 2019][1696564.863369] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.863372] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:06 2019][1696564.863375] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:06 2019][1696564.863378] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863380] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863382] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.863385] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.863387] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.863388] Task dump for CPU 26: [Wed Feb 6 09:41:06 2019][1696564.863391] swapper/26 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:06 2019][1696564.863391] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.863394] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:06 2019][1696564.863396] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:06 2019][1696564.863399] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863401] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863404] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.863407] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.863409] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.863411] Task dump for CPU 27: [Wed Feb 6 09:41:06 2019][1696564.863413] swapper/27 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:06 2019][1696564.863413] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.863416] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:06 2019][1696564.863419] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:06 2019][1696564.863422] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863425] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863427] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.863430] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.863432] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.863433] Task dump for CPU 28: [Wed Feb 6 09:41:06 2019][1696564.863436] swapper/28 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:06 2019][1696564.863436] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.863439] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:06 2019][1696564.863441] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:06 2019][1696564.863444] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863446] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863448] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.863451] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.863453] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.863454] Task dump for CPU 29: [Wed Feb 6 09:41:06 2019][1696564.863456] swapper/29 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:06 2019][1696564.863457] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.863460] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:06 2019][1696564.863462] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:06 2019][1696564.863464] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863467] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:06 2019][1696564.863469] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:06 2019][1696564.863472] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:06 2019][1696564.863474] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:06 2019][1696564.863475] Task dump for CPU 30: [Wed Feb 6 09:41:06 2019][1696564.863478] ldlm_bl_16 R running task 0 15973 2 0x00000080 [Wed Feb 6 09:41:06 2019][1696564.863478] Call Trace: [Wed Feb 6 09:41:06 2019][1696564.863514] [] ? ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.863547] [] ? ldlm_cancel_locks_for_export_cb+0x3f/0x50 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.863559] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [Wed Feb 6 09:41:07 2019][1696564.863593] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.863627] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.863638] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [Wed Feb 6 09:41:07 2019][1696564.863672] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.863708] [] ? ldlm_bl_thread_main+0x4c8/0x700 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.863712] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:41:07 2019][1696564.863749] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.863751] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:41:07 2019][1696564.863754] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:07 2019][1696564.863757] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:41:07 2019][1696564.863760] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:07 2019][1696564.863761] Task dump for CPU 31: [Wed Feb 6 09:41:07 2019][1696564.863764] ldlm_bl_11 R running task 0 15968 2 0x00000080 [Wed Feb 6 09:41:07 2019][1696564.863765] Call Trace: [Wed Feb 6 09:41:07 2019][1696564.863799] [] ? ldlm_cancel_lock_for_export.isra.26+0x1c2/0x390 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.863832] [] ? ldlm_cancel_locks_for_export_cb+0x3f/0x50 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.863844] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [Wed Feb 6 09:41:07 2019][1696564.863878] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.863911] [] ? ldlm_cancel_lock_for_export.isra.26+0x390/0x390 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.863922] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [Wed Feb 6 09:41:07 2019][1696564.863956] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.863993] [] ? ldlm_bl_thread_main+0x4c8/0x700 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.863995] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:41:07 2019][1696564.864032] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.864034] [] ? kthread+0xd1/0xe0 [Wed Feb 6 09:41:07 2019][1696564.864037] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:07 2019][1696564.864041] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:41:07 2019][1696564.864043] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:07 2019][1696564.864045] Task dump for CPU 32: [Wed Feb 6 09:41:07 2019][1696564.864048] swapper/32 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:07 2019][1696564.864049] Call Trace: [Wed Feb 6 09:41:07 2019][1696564.864052] [] ? tick_nohz_stop_sched_tick+0x7c/0x380 [Wed Feb 6 09:41:07 2019][1696564.864054] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:07 2019][1696564.864058] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864061] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864063] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:07 2019][1696564.864065] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:07 2019][1696564.864067] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:07 2019][1696564.864068] Task dump for CPU 33: [Wed Feb 6 09:41:07 2019][1696564.864070] swapper/33 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:07 2019][1696564.864071] Call Trace: [Wed Feb 6 09:41:07 2019][1696564.864074] [] ? tick_nohz_stop_sched_tick+0x7c/0x380 [Wed Feb 6 09:41:07 2019][1696564.864076] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:07 2019][1696564.864079] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864082] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864084] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:07 2019][1696564.864087] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:07 2019][1696564.864089] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:07 2019][1696564.864090] Task dump for CPU 34: [Wed Feb 6 09:41:07 2019][1696564.864092] swapper/34 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:07 2019][1696564.864093] Call Trace: [Wed Feb 6 09:41:07 2019][1696564.864096] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:07 2019][1696564.864099] [] ? native_safe_halt+0x6/0x10 [Wed Feb 6 09:41:07 2019][1696564.864101] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864103] [] ? rcu_idle_enter+0x22/0x30 [Wed Feb 6 09:41:07 2019][1696564.864105] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864108] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:07 2019][1696564.864112] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:07 2019][1696564.864114] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:07 2019][1696564.864115] Task dump for CPU 35: [Wed Feb 6 09:41:07 2019][1696564.864117] swapper/35 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:07 2019][1696564.864118] Call Trace: [Wed Feb 6 09:41:07 2019][1696564.864121] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:07 2019][1696564.864123] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:07 2019][1696564.864126] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864129] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864132] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:07 2019][1696564.864135] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:07 2019][1696564.864137] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:07 2019][1696564.864138] Task dump for CPU 36: [Wed Feb 6 09:41:07 2019][1696564.864140] swapper/36 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:07 2019][1696564.864140] Call Trace: [Wed Feb 6 09:41:07 2019][1696564.864143] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:07 2019][1696564.864145] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:07 2019][1696564.864147] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864150] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864152] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:07 2019][1696564.864154] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:07 2019][1696564.864156] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:07 2019][1696564.864156] Task dump for CPU 37: [Wed Feb 6 09:41:07 2019][1696564.864159] swapper/37 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:07 2019][1696564.864159] Call Trace: [Wed Feb 6 09:41:07 2019][1696564.864162] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:07 2019][1696564.864164] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:07 2019][1696564.864168] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864170] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864173] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:07 2019][1696564.864177] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:07 2019][1696564.864179] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:07 2019][1696564.864180] Task dump for CPU 38: [Wed Feb 6 09:41:07 2019][1696564.864181] swapper/38 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:07 2019][1696564.864182] Call Trace: [Wed Feb 6 09:41:07 2019][1696564.864184] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:07 2019][1696564.864187] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:07 2019][1696564.864190] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864192] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864194] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:07 2019][1696564.864197] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:07 2019][1696564.864199] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:07 2019][1696564.864200] Task dump for CPU 39: [Wed Feb 6 09:41:07 2019][1696564.864202] swapper/39 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:07 2019][1696564.864203] Call Trace: [Wed Feb 6 09:41:07 2019][1696564.864207] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:07 2019][1696564.864210] [] ? native_safe_halt+0x6/0x10 [Wed Feb 6 09:41:07 2019][1696564.864213] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864215] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864218] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:07 2019][1696564.864220] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:07 2019][1696564.864222] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:07 2019][1696564.864223] Task dump for CPU 40: [Wed Feb 6 09:41:07 2019][1696564.864225] swapper/40 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:07 2019][1696564.864226] Call Trace: [Wed Feb 6 09:41:07 2019][1696564.864229] [] ? tick_nohz_stop_sched_tick+0x7c/0x380 [Wed Feb 6 09:41:07 2019][1696564.864232] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:07 2019][1696564.864234] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864237] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864240] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:07 2019][1696564.864242] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:07 2019][1696564.864244] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:07 2019][1696564.864245] Task dump for CPU 41: [Wed Feb 6 09:41:07 2019][1696564.864247] swapper/41 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:07 2019][1696564.864248] Call Trace: [Wed Feb 6 09:41:07 2019][1696564.864250] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:07 2019][1696564.864253] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:07 2019][1696564.864256] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864258] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864261] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:07 2019][1696564.864264] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:07 2019][1696564.864266] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:07 2019][1696564.864267] Task dump for CPU 42: [Wed Feb 6 09:41:07 2019][1696564.864269] swapper/42 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:07 2019][1696564.864269] Call Trace: [Wed Feb 6 09:41:07 2019][1696564.864272] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:07 2019][1696564.864274] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:07 2019][1696564.864277] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864280] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864282] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:07 2019][1696564.864285] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:07 2019][1696564.864287] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:07 2019][1696564.864288] Task dump for CPU 43: [Wed Feb 6 09:41:07 2019][1696564.864290] swapper/43 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:07 2019][1696564.864290] Call Trace: [Wed Feb 6 09:41:07 2019][1696564.864293] [] ? tick_nohz_stop_sched_tick+0x7c/0x380 [Wed Feb 6 09:41:07 2019][1696564.864295] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:07 2019][1696564.864298] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864300] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:07 2019][1696564.864303] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:07 2019][1696564.864306] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:07 2019][1696564.864308] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:07 2019][1696564.864309] Task dump for CPU 44: [Wed Feb 6 09:41:07 2019][1696564.864312] mdt00_004 R running task 0 6615 2 0x00000080 [Wed Feb 6 09:41:07 2019][1696564.864312] Call Trace: [Wed Feb 6 09:41:07 2019][1696564.864354] [] ? ptlrpc_unregister_reply+0x3bc/0x790 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.864395] [] ? ptlrpc_expire_one_request+0xee/0x520 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.864433] [] ptlrpc_check_set.part.23+0x935/0x1df0 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.864455] [] ? lnet_md_unlink+0x125/0x3f0 [lnet] [Wed Feb 6 09:41:07 2019][1696564.864494] [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.864533] [] ptlrpc_set_wait+0x57d/0x8d0 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.864537] [] ? wake_up_state+0x20/0x20 [Wed Feb 6 09:41:07 2019][1696564.864571] [] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.864609] [] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.864637] [] mdt_do_glimpse+0x1e9/0x4c0 [mdt] [Wed Feb 6 09:41:07 2019][1696564.864675] [] ? ldlm_process_inodebits_lock+0x93/0x3d0 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.864711] [] ? ldlm_inodebits_compat_queue+0x440/0x440 [ptlrpc] [Wed Feb 6 09:41:07 2019][1696564.864730] [] mdt_glimpse_enqueue+0x3d3/0x4f0 [mdt] [Wed Feb 6 09:41:07 2019][1696564.864744] [] mdt_intent_glimpse+0x1f/0x30 [mdt] [Wed Feb 6 09:41:07 2019][1696564.864758] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:41:07 2019][1696564.864771] [] ? mdt_intent_brw+0x30/0x30 [mdt] [Wed Feb 6 09:41:07 2019][1696564.864805] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:41:08 2019][1696564.864816] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Wed Feb 6 09:41:08 2019][1696564.864828] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Wed Feb 6 09:41:08 2019][1696564.864866] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:41:08 2019][1696564.864907] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Wed Feb 6 09:41:08 2019][1696564.864965] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:41:08 2019][1696564.865015] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:41:08 2019][1696564.865059] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Wed Feb 6 09:41:08 2019][1696564.865069] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Wed Feb 6 09:41:08 2019][1696564.865111] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:41:08 2019][1696564.865154] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Wed Feb 6 09:41:08 2019][1696564.865156] [] ? default_wake_function+0x12/0x20 [Wed Feb 6 09:41:08 2019][1696564.865159] [] ? __wake_up_common+0x5b/0x90 [Wed Feb 6 09:41:08 2019][1696564.865199] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:41:08 2019][1696564.865241] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Wed Feb 6 09:41:08 2019][1696564.865244] [] kthread+0xd1/0xe0 [Wed Feb 6 09:41:08 2019][1696564.865247] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:08 2019][1696564.865250] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:41:08 2019][1696564.865253] [] ? insert_kthread_work+0x40/0x40 [Wed Feb 6 09:41:08 2019][1696564.865254] Task dump for CPU 45: [Wed Feb 6 09:41:08 2019][1696564.865257] swapper/45 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:08 2019][1696564.865257] Call Trace: [Wed Feb 6 09:41:08 2019][1696564.865261] [] ? tick_nohz_stop_sched_tick+0x7c/0x380 [Wed Feb 6 09:41:08 2019][1696564.865263] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:08 2019][1696564.865267] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:08 2019][1696564.865269] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:08 2019][1696564.865272] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:08 2019][1696564.865274] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:08 2019][1696564.865276] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:08 2019][1696564.865277] Task dump for CPU 46: [Wed Feb 6 09:41:08 2019][1696564.865280] sssd_nss R running task 0 17601 17535 0x00000080 [Wed Feb 6 09:41:08 2019][1696564.865281] Call Trace: [Wed Feb 6 09:41:08 2019][1696564.865286] [] ? physflat_send_IPI_mask+0xe/0x10 [Wed Feb 6 09:41:08 2019][1696564.865289] [] ? native_send_call_func_single_ipi+0x42/0x50 [Wed Feb 6 09:41:08 2019][1696564.865294] [] ? __slab_free+0x81/0x2f0 [Wed Feb 6 09:41:08 2019][1696564.865297] [] ? __slab_free+0x81/0x2f0 [Wed Feb 6 09:41:08 2019][1696564.865300] [] ? __wake_up_sync_key+0x4f/0x60 [Wed Feb 6 09:41:08 2019][1696564.865305] [] ? unix_write_space+0x4b/0x80 [Wed Feb 6 09:41:08 2019][1696564.865309] [] ? skb_release_data+0xf5/0x140 [Wed Feb 6 09:41:08 2019][1696564.865313] [] ? kmem_cache_free+0x1e2/0x200 [Wed Feb 6 09:41:08 2019][1696564.865317] [] ? kfree_skbmem+0x37/0x90 [Wed Feb 6 09:41:08 2019][1696564.865319] [] ? consume_skb+0x34/0x90 [Wed Feb 6 09:41:08 2019][1696564.865323] [] ? unix_stream_read_generic+0x6d7/0x8e0 [Wed Feb 6 09:41:08 2019][1696564.865327] [] ? __mem_cgroup_uncharge_common+0xcd/0x2f0 [Wed Feb 6 09:41:08 2019][1696564.865331] [] ? lru_cache_add+0xe/0x10 [Wed Feb 6 09:41:08 2019][1696564.865334] [] ? putback_lru_page+0x52/0xb0 [Wed Feb 6 09:41:08 2019][1696564.865339] [] ? should_numa_migrate_memory+0x5b/0x150 [Wed Feb 6 09:41:08 2019][1696564.865342] [] ? mpol_misplaced+0x18c/0x1e0 [Wed Feb 6 09:41:08 2019][1696564.865344] [] ? do_numa_page+0x1be/0x250 [Wed Feb 6 09:41:08 2019][1696564.865348] [] ? kmem_cache_alloc+0x1c2/0x1f0 [Wed Feb 6 09:41:08 2019][1696564.865352] [] ? locks_alloc_lock+0x1b/0x60 [Wed Feb 6 09:41:08 2019][1696564.865355] [] ? __posix_lock_file+0x21a/0x550 [Wed Feb 6 09:41:08 2019][1696564.865359] [] ? vfs_lock_file+0x53/0x60 [Wed Feb 6 09:41:08 2019][1696564.865363] [] ? do_lock_file_wait.part.31+0x54/0x100 [Wed Feb 6 09:41:08 2019][1696564.865366] [] ? get_empty_filp+0x5c/0x1a0 [Wed Feb 6 09:41:08 2019][1696564.865369] [] ? kmem_cache_alloc+0x1c2/0x1f0 [Wed Feb 6 09:41:08 2019][1696564.865372] [] ? locks_alloc_lock+0x1b/0x60 [Wed Feb 6 09:41:08 2019][1696564.865374] [] ? locks_free_lock+0x3c/0x50 [Wed Feb 6 09:41:08 2019][1696564.865377] [] ? fcntl_setlk+0x1f8/0x2d0 [Wed Feb 6 09:41:08 2019][1696564.865382] [] ? __audit_syscall_exit+0x1e6/0x280 [Wed Feb 6 09:41:08 2019][1696564.865385] [] ? sysret_audit+0x17/0x21 [Wed Feb 6 09:41:08 2019][1696564.865387] Task dump for CPU 47: [Wed Feb 6 09:41:08 2019][1696564.865389] swapper/47 R running task 0 0 1 0x00000000 [Wed Feb 6 09:41:08 2019][1696564.865390] Call Trace: [Wed Feb 6 09:41:08 2019][1696564.865392] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:08 2019][1696564.865395] [] ? rcu_eqs_enter_common.isra.30+0x3d/0xe0 [Wed Feb 6 09:41:08 2019][1696564.865397] [] ? default_idle+0x1e/0xc0 [Wed Feb 6 09:41:08 2019][1696564.865400] [] ? arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:08 2019][1696564.865402] [] ? cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:08 2019][1696564.865404] [] ? start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:08 2019][1696564.865406] [] ? start_cpu+0x5/0x14 [Wed Feb 6 09:41:08 2019][1696568.308211] { 14} (t=3448 jiffies g=134912029 c=134912028 q=395939) [Wed Feb 6 09:41:08 2019][1696568.314911] Task dump for CPU 14: [Wed Feb 6 09:41:08 2019][1696568.318405] swapper/14 R running task 0 0 1 0x00000008 [Wed Feb 6 09:41:08 2019][1696568.325676] Call Trace: [Wed Feb 6 09:41:08 2019][1696568.328303] [] sched_show_task+0xa8/0x110 [Wed Feb 6 09:41:08 2019][1696568.334778] [] dump_cpu_task+0x39/0x70 [Wed Feb 6 09:41:08 2019][1696568.340351] [] rcu_dump_cpu_stacks+0x90/0xd0 [Wed Feb 6 09:41:08 2019][1696568.346443] [] rcu_check_callbacks+0x442/0x730 [Wed Feb 6 09:41:08 2019][1696568.352712] [] ? do_IRQ+0x56/0xf0 [Wed Feb 6 09:41:08 2019][1696568.357850] [] ? tick_sched_do_timer+0x50/0x50 [Wed Feb 6 09:41:08 2019][1696568.364115] [] update_process_times+0x46/0x80 [Wed Feb 6 09:41:08 2019][1696568.370294] [] tick_sched_handle+0x30/0x70 [Wed Feb 6 09:41:08 2019][1696568.376214] [] tick_sched_timer+0x39/0x80 [Wed Feb 6 09:41:08 2019][1696568.382047] [] __hrtimer_run_queues+0xf3/0x270 [Wed Feb 6 09:41:08 2019][1696568.388312] [] hrtimer_interrupt+0xaf/0x1d0 [Wed Feb 6 09:41:08 2019][1696568.394319] [] local_apic_timer_interrupt+0x3b/0x60 [Wed Feb 6 09:41:08 2019][1696568.401017] [] smp_apic_timer_interrupt+0x43/0x60 [Wed Feb 6 09:41:08 2019][1696568.407546] [] apic_timer_interrupt+0x162/0x170 [Wed Feb 6 09:41:08 2019][1696568.413895] [] ? __cpuidle_text_start+0x8/0x8 [Wed Feb 6 09:41:08 2019][1696568.420708] [] ? native_safe_halt+0x6/0x10 [Wed Feb 6 09:41:08 2019][1696568.426629] [] default_idle+0x1e/0xc0 [Wed Feb 6 09:41:08 2019][1696568.432116] [] arch_cpu_idle+0x20/0xc0 [Wed Feb 6 09:41:08 2019][1696568.437689] [] cpu_startup_entry+0x14a/0x1e0 [Wed Feb 6 09:41:08 2019][1696568.443779] [] start_secondary+0x1f7/0x270 [Wed Feb 6 09:41:08 2019][1696568.449701] [] start_cpu+0x5/0x14